SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Agrawal, A., Sheldon, D. R., and Domke, J. (2020). Advances in black-box VI: Normalizing flows, importance weighting, and optimization. In Advances in Neural Information Processing Systems.
Appendices to "Discriminative Calibration"
The Appendices are organized as follows. In Section A, we provide several extra theory results for the main paper. In Section B, we discuss our experiment and implementations. In Section C, we provide proofs for the theory claims.
A Additional theory results
Theorem 7 (Closed form expression of the divergence). Consider the label generating process in Section 4. Let  $\pi(t, \phi)$  be the joint density of the label and features after mapping  $\Phi$ , i.e., the distribution of the  $(t, \phi)$  generated by the simulation process:
1. sample  $(\theta ,y)\sim p(\theta ,y)$
2. sample  $(\tilde{\theta}_1,\dots ,\tilde{\theta}_M)$  from  $q(\theta |y)$
3. generate  $((t_1,\phi_1),\ldots ,(t_L,\phi_L))) = \Phi ((y,\theta ,\tilde{\theta}_1,\ldots ,\tilde{\theta}_M));$
4. sample an index  $l$  from  $\operatorname{Uniform}(1, \ldots, L)$ ;
5. return  $(t_l,\phi_l)$
$$
\pi_ {k} (\phi) = \pi (\phi | t = k) = \int_ {y} \pi (y) \pi (\phi | y, t = k) d y
$$
to be the  $y$ -averaged law of  $\phi |t = k$ . Note that  $y$  has been averaged out.
Then classification divergence defended through (7) in Theorem 1 has a closed form expression, a Jensen-Shannon divergence in this projected  $\phi$  space.
$$
D ^ {\mathrm {o p t}} (p, q, \Phi) = \sum_ {k = 0} ^ {K - 1} w _ {k} \mathrm {K L} \left(\pi_ {k} (\cdot) | | \sum_ {j = 0} ^ {K - 1} w _ {j} \pi_ {j} (\cdot)\right). \tag {13}
$$
This Theorem 7 gives a connection between our divergence and the familiar Jensen-Shannon divergence, which is well known to be linked to IID sample classification error (note that the classification examples we generate are not IID for they share  $y$ ).
As a special case, when  $K = 2$ ,  $w_0 = w_1 = 1/2$ , we recover the symmetric two-group Jensen-Shannon divergence, or JSD. In general, between two densities  $\pi_1(x)$  and  $\pi_2(x)$ , the JSD is
$$
J e s e n S h a n n o n d i v e r g e n c e \left(\pi_ {1}, \pi_ {2}\right) = \frac {1}{2} \mathrm {K L} \left(\pi_ {1} \| \frac {\pi_ {1} + \pi_ {2}}{2}\right) + \frac {1}{2} \mathrm {K L} \left(\pi_ {2} \| \frac {\pi_ {1} + \pi_ {2}}{2}\right). \tag {14}
$$
As before, the standard notation [5] of conditional divergence is defined as taking expectations over conditional variables:
$$
\begin{array}{l} \operatorname {J S D} \left(\pi_ {1} (x | z), \pi_ {2} (x | z)\right) = \frac {1}{2} \operatorname {K L} \left(\pi_ {1} (x | z) \mid \mid \frac {\pi_ {1} (x | z) + \pi_ {2} (x | z)}{2}\right) \\ + \frac {1}{2} \operatorname {K L} \left(\pi_ {2} (x | z) \mid \mid \frac {\pi_ {1} (x | z) + \pi_ {2} (x | z)}{2}\right). \tag {15} \\ \end{array}
$$
Theorem 8 (weighting). The binary label mapping generates  $M$  Paris label-1 examples and one pair of label-0 examples per simulation. We can reweight the binary classifier by letting the ELPD be  $\mathbb{E}[\frac{C}{M + 1}\mathbb{1}(t = 1)\log p(t = 1|c(\phi)) + \frac{CM}{M + 1}\mathbb{1}(t = 0)\log p(t = 0|c(\phi))]$ , where  $C = \frac{(M + 1)^2}{2M}$  is a normalizing constant.
That is if we modify the training utility function or the validation data LPD to be:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\frac {C}{M + 1} \mathbb {1} (t _ {i} = 1) \log \Pr (t = 1 | c (\phi_ {i})) + \frac {C M}{M + 1} \mathbb {1} (t _ {i} = 0) \log \Pr (t = 0 | c (\phi_ {i}))\right).
$$
then the resulting binary classification divergence in Thm. 1, i.e.,
$$
w e i g h t e d E L P D + \log 2,
$$
is the conditional Jensen Shannon divergence (15).
A.1 SBC test and generative classifier
In Example 3, we stated the intuitive connection between the SBC rank test and using rank as the feature in a "generative" classifier (in contrast to our "discriminate" approach). Now we make such comparison more precise: Assuming the parameter  $\theta$  is one-dimensional. This is what SBC does: From  $p$  we obtain  $\theta, y$ , and for each  $\theta$  we compute the rank of it in the paired  $q$  samples who shared the same  $y$ :  $r = \sum_{m=1}^{M} \mathbb{1}(\theta \leq \tilde{\theta}_m)$ . Repeating this simulation many times, we obtain a sample of  $r$ , and SBC will test if such  $r$  is discrete-uniformly distributed. This test could be done by simulating a reference distribution that matches the null. We do so by generating the ranks in the  $q$  samples:  $\tilde{r}_n = \sum_{m \neq n} \mathbb{1}(\tilde{\theta}_n \leq \tilde{\theta}_m) + \mathbb{1}(\tilde{\theta}_n \leq \theta)$ . To be clear, there are other ways to generate reference samples, but all it matters is that the reference distribution matches the distribution of  $r$  under the null. The uniformity test in SBC now becomes to test if  $r$  and  $\tilde{r}$  have the same distribution. For example, we can do a two-sample Kolmogorov-Smirnov test or a permutation test.
In comparison, this is what naive Bayes would do: the naive Bayes classifier first estimates the two conditional distributions:  $\operatorname*{Pr}(r|t = 0)$  from the empirical distribution of the prior rank  $r$ , and  $\operatorname*{Pr}(r|t = 1)$  from the empirical distribution of  $\tilde{r}$ . Then to test if  $\operatorname*{Pr}(r|t = 0) = \operatorname*{Pr}(r|t = 1)$  becomes the same to testing if the empirical distribution of  $r$  is the same as the empirical distribution of  $r$ —the same task as in SBC. The point is that given any two-sample based test,
$$
\mathrm {S B C} + \text {s a m p l e - b a s e d t e s t}
$$
is operationally the same as
only use rank + naive Bayes classifier + density estimated by histogram + sample-based test.
B Implementations
B.1 A cheat sheet of useful pre-learned features
Table 1 summarizes how to use the statistical feature (whenever available) in both the binary and multiclass classifiers.
binary	multiclass
full feature	(θ,y)	(θ1, ..., θK,y)
MCMC	p(θ,y), r(θ,y)	p(θt,y) r(θt,y), 1 ≤ t ≤ K
VI	p(θ,y), q(θ|y)	p(θt,y) q(θt|y), 1 ≤ t ≤ K
likelihood-free	p(θ), q(θ|y)	p(θt), q(θt|y), 1 ≤ t ≤ K
Table 1: A cheat sheet of useful pre-learned features
B.2 Code and data
We share the python and Jax implementation of our binary and multiclass calibration code in https://github.com/yao-yl/DiscCalibration.
In the MLP training we include a standard  $L^2$  weight decay (i.e., training loss function = cross entropy loss + tuning weight *  $L^2$  penalization). We tune the weight of the decay term by a 5 fold cross-validation in the training set on a fixed grid  $\{0.1, 0.01, 0.001, 0.0001\}$ .
In the cosmology experiment in Section 5, we use one-hidden-layer MLP with 64 nodes to parameterize the classifier with the form (11), with additional pre-learned features such as  $\log q(\theta |y)$  added as linear features. The approximate log density  $\log q(\theta |y)$  is known (i.e., we can evaluate the for any  $\theta$  and  $y$ , or at least least up to a constant) in either the normalizing flow or the ensemble of the normalizing flows. One classification run with roughly one million examples took roughly two hour
model 1 (dy=38)	model 2 (dy=151)	model 3 (dy=1031)
1-flow	0.008 (0.003)	0.05(0.006)	0.0001 (0.0002)
5-ensemble	0.001 (0.002)	0.008(0.003)	0.001(0.001)
Figure 8: The estimated divergence of two simulation based inferences (either using one flow or an ensemble of five flows) in three cosmology models. These three models have the same parameter space and involve expensive simulations. Here we apply our classifier approach and estimate the Jensen-Shannon divergence (and standard deviation) using weighted binary classifiers from  $S = 2000$ ,  $M = 500$  simulations: that is  $2000 * 501 = 1$  million classification examples. In this table, we run this classifier by the black-box multilayer perceptron (MLP) on the full  $(\theta, y)$  vector, while the table we have in the main paper further adds  $\log q(\theta | y)$  as a pre-learned feature since it is known in the normalizing flows. In this black box estimate table, we would pass a t-test and not even detect miscalculation. Adding a statistically-meaningful feature greatly improves the classifier prediction, and tightens the bound of the divergence estimate.
cpu time on a local laptop. It would be more efficient to run the classification on GPU, thought the classification cost is overall negligible compared with the simulation step  $y|\theta$  which is pre-computed and stored.
In the closed-form Gaussian experiment in Section 5, we consider the easiest setting:  $\theta \in \mathbb{R}^d \sim \mathrm{MVN}(\mathbf{0},\mathrm{Id}_d)$  and a normal data model  $y|\theta \sim \mathrm{MVN}(\theta ,\mathrm{Id}_d)$ . Indeed, we have kept both the true distribution and the sampling corruption mean-field to make the calibration task easier for the traditional SBC rank-test. The power of the traditional SBC rank-test would be even worse if we add some interaction terms in the corrupted samples  $q(\theta |y)$ , while our method leans the joint space sampling quality by default. The exact posterior  $p(\theta |y)$  is known but we pretend we cannot evaluate it. We set  $d = 16$  to be comparable with the real data example. The right panel validates Thm. 3 fixed the simulation size  $S = 5000$  and vary  $M$ . We have tried other fixed  $M = 1000$  but it seems the resulting graph is very similar. The confidence band in Figure 3 and Figure 4 is computed using 1.96 times standard error from repeated experiments.
Table 8 shows the naive classifier: using the bandbox MLP without the linear features. The estimate is rather loose compared with the table we obtained in Section 5 which used statistically-meaningful features.
C Proofs of the theorems
We prove the theorems in the following order: Thm. 7, Thm. 1, Thm. 8, Thm. 6, Thm. 3, Thm. 5, Thm. 4, and finally Thm. 2.
C.1 The classification divergence 15
C.2 Reweighting (Theorem 8) leads to symmetric divergence 18
C.3 MCMC without thinning (Theorem 6) 19
C.4 Large sample limit and rate as  $M\to \infty$  (Theorem 3) 20
C.5 Valid hypothesis testing (Theorem 4) 22
C.6 Why does classifier help test? (Theorem 5) 23
C.7 The maximum discriminative generator (Theorem 2) 23
C.1 The classification divergence
To prove our Theorem 1, we first state an equivalent but a more general theorem. Note that we are allowing non-IID classification examples for the shared  $z$  variables in this theory.
Theorem 9 (The general relation between divergence and classification error). Define a simulation process  $\pi(z, k, x)$  through the following process, among the three random variables,  $z \in \mathbb{R}^n$ ,  $x \in \mathbb{R}^d$  can have any dimension, and  $k \in \{1, 2, \dots, K\}$  is an integer.
1. Sample  $z \sim \pi(z)$
2. Sample  $k \sim \operatorname{Categorical}(w)$ , where  $w$  is a simplex.
3. Sample  $x \sim \pi_k(x|z)$
4. Return  $(z, k, x)$
Define  $\mathrm{U}(\pi)$  to be the expected log probability or cross-entropy of an optimal classifier trained to predict  $k$  from  $(x, z)$ , i.e.
$$
\mathrm {U} (\pi) = \max  _ {c} \mathbb {E} _ {\pi} U (k, c (x, z)),
$$
where  $U(k, c) = \log c(k)$ . Then
$$
\mathrm {U} (\pi) - \sum_ {k = 1} ^ {K} w _ {k} \log w _ {k} = \sum_ {n = 1} ^ {K} w _ {k} \mathrm {K L} \left(\pi_ {n} (x | z) \| \pi (x | z)\right)
$$
where  $\pi (x|z) = \sum_{k = 1}^{K}w_{k}\pi_{k}(x|z)$  is the mixture over all groups.
We define  $D(\pi) = \mathrm{U}(\pi) - \sum_{k=1}^{K} w_k \log w_k$ . Then this  $D$  has the following properties:
1. (lower bound).  $D(\pi) \geq 0$ . It achieves zero if and only if all  $p_k$  are the same, i.e.,  $\operatorname{Pr}(X^k \in A) = \operatorname{Pr}(X^j \in A)$  for any measurable set  $A$ .
2. (upper bound).  $D(\pi) \leq \sum_{n=1}^{K} w_k \log w_k$ . This maximum is achieved if  $p_k$  are disjoint, i.e.,  $\operatorname{Pr}(X^k \in A) \operatorname{Pr}(X^j \in A) = 0$  for any measurable set  $A$ .
3. (alternative upper bound when the difference is small).  $D(\pi) \leq \max_{j,k} \mathrm{KL}(\pi_k(x|z), \pi_j(x|z))$ .
4. (reparametrization invariance). For any fixed bijective transformation  $x \mapsto g(x)$ , let  $p_k^g$  be the law of  $g(x^k)$ , then  $D(p_k^g, \ldots, p_k^g) = D(p_k, \ldots, p_k)$ .
5. (transformation decreases divergence). For any fixed transformation  $x \mapsto g(x)$ , let  $p_k^g$  be the law of  $g(x^k)$ , then  $D(p_k^g, \ldots, p_k^g) \leq D(p_k, \ldots, p_k)$ .
Proof. The (unknown) optimal conditional classifier is the Bayes classifier:
$$
\hat {\Pr} (t = k | x, z) = \frac {w _ {k} \pi_ {k} (x | z)}{\sum_ {j} w _ {k} \pi_ {j} (x | z)}.
$$
Let  $r(x|z)\sum_{j = 1}^{K}w_k\pi_j(x|z)$  be mixture density marginalized over the group index  $k$
3We emphasize that we are using the notation of conditional KL divergence:
KL  $(\pi_n(x|z)\parallel \pi (x|z))\coloneqq \int \left[p(z)\int \pi_k(x|z)\log \left(\frac{\pi_k(x|z)}{r(x|z)}\right)dx\right]dz.$
The expected log predictive density of the optimal classification is then
$$
\begin{array}{l} \mathrm {E L P D} (\hat {\mathrm {P r}} (t = k | x, z)) = \mathbb {E} _ {z} \int r (x | z) \sum_ {k = 1} ^ {K} \left[ \frac {w _ {k} \pi_ {k} (x | z)}{r (x | z)} \log \left(\frac {w _ {k} \pi_ {k} (x | z)}{r (x | z)}\right) \right] d x \\ = \mathbb {E} _ {z} \sum_ {k = 1} ^ {K} w _ {k} \int \pi_ {k} (x | z) \log \left(\frac {\pi_ {k} (x | z)}{r (x | z)} + \log w _ {k}\right) d x \\ = \mathbb {E} _ {z} \sum_ {k = 1} ^ {K} w _ {k} \log (w _ {k}) + \sum_ {n = 1} ^ {K} w _ {k} \mathbb {E} _ {z} \int \pi_ {k} (x | z) \log \left(\frac {\pi_ {k} (x | z)}{r (x | z)}\right) d x \\ = \sum_ {k = 1} ^ {K} w _ {k} \log (w _ {k}) + \sum_ {n = 1} ^ {K} w _ {k} \mathrm {K L} (\pi_ {k} (x | z), r (x | z)). \\ \end{array}
$$
When  $\mathrm{ELPD} - \sum_{k=1}^{K} w_k \log(w_k) = 0$ ,  $\sum_{k=1}^{K} w_k \pi_k(x|z) = 0$ , hence these  $K$  conditional densities  $\pi_k(x|z)$  are the same almost everywhere.
The reparametrization invariance is directly inherited from KL diverge.
The upper bound is the consequence of ELPD  $\leq 0$  for category classification and the lower bound is that KL divergence  $\geq 0$ .
Let's emphasize again that we are following the notation of conditional divergence in [5], such that for any two joint density  $p(\theta \mid y)$  and  $q(\theta, y)$ , the conditional KL divergence is  $\mathrm{KL}(p(\theta | y) \| q(\theta | y)) \coloneqq \mathbb{E}_{p(y)} \mathbb{E}_{p(y|\theta)} \log \frac{p(y|\theta)}{q(y|\theta)}$ , an expectation is taken over  $y$ .
The procedure in Theorem 9 can reflect many different generating processes. Let's recap the examples we show in Section 2.
In Example 1 of the main paper, the binary case,  $K = 2$ , and we generate  $M$  samples from  $q$ .  $\pi_1 = p(\theta | y)$ ,  $\pi_2 = q(\theta | y)$ ,  $w_1 = 1 / M$ ,  $w_2 = (M - 1) / M$ . Our simulation process in example 1 is equivalent to
1. Sample  $y \sim$  marginal  $p(y)$ ,
2. Sample  $k \sim \mathrm{Categorical}(w)$ , where  $w_{1} = 1 / M$ ,  $w_{2} = (M - 1) / M$ .
3. Sample  $x \sim \pi_k(x|z)$ , where  $\pi_1 = p(\theta | y)$ ,  $\pi_2 = q(\theta | y)$ .
4. Return  $(z,k,x)$
As a direct consequence of Theorem 9, the resulting divergence in the binary case is
$$
D _ {1} (p, q) = w _ {1} \operatorname {K L} (p (\theta | y) \| r (\theta | y)) + w _ {2} \operatorname {K L} (q (\theta | y) \| r (\theta | y)),
$$
where  $r = w_{1}p + w_{2}q$
In Example 2: Binary label with no  $y$ , we generate  $M$  samples from  $q$ .  $K = 2$ ,  $\pi_1 = p(\theta)$ ,  $\pi_2 = q(\theta)$ ,  $w_1 = 1 / (M + 1)$ ,  $w_2 = M / (M + 1)$ . Our simulation process in example 2 is equivalent to
1. Sample  $k \sim$  Categorical(w), where  ${w}_{1} = 1/M,{w}_{2} = \left( {M - 1}\right) /M$  .
2. Sample  $x\sim \pi_k(x)$  , where  $\pi_1 = p(\theta),\pi_2 = q(\theta)$
3. Return  $(k, x)$
From Theorem 9, the resulting divergence reads
$$
D _ {3} (p, q) = w _ {1} \mathrm {K L} (p (\theta) \| r (\theta)) + w _ {2} \mathrm {K L} (q (\theta) \| r (\theta))
$$
In the multivariate Example 4,  $K = M + 1$ , and the individual density is
$$
\pi_ {k} (\theta_ {1: K}) = p (\theta_ {k} | y) \prod_ {m \neq k} q (\theta_ {m} | y)
$$
$$
D _ {4} (p, q) = \mathrm {K L} \left(p (\theta_ {1} | y) \prod_ {m > 1} q (\theta_ {m} | y) \| \frac {1}{K} \sum_ {k = 1} ^ {K} p (\theta_ {k} | y) \prod_ {m \neq k} q (\theta_ {m} | y)\right)
$$
We give more results on this  $D_4$  in Section C.4.
The discriminative calibration is designed to detect and gauge the difference between  $p(\theta |y)$  and  $q(\theta |y)$ . From now on we use a long vector  $\theta_{1:M + 1}$  (the index starts by 1) to represent the simulation draw. Each simulation draw first generates one  $\theta_{1}$  from the prior, one data point  $y$  from the sampling distribution  $p(y|\theta_1)$ , and finally  $M$  draws from the approximate inference  $\theta_{1,\dots,M + 1}\sim q(\theta |y)$ .
The "label generating process"  $\Phi$  takes a simulated draw as its input and returns a vector of features  $\phi$  and labels  $t$ .
$$
\Phi : (y, \theta_ {1, \dots , M + 1}) \mapsto \left\{\left(\phi_ {1}, t _ {1}\right), \dots , \left(\phi_ {K}, t _ {K}\right) \right\}.
$$
The classifier will then see this feature and labels (the index  $k$  is disregarded) across all simulations.
We only ask that the map  $\Phi$  be within in set  $\mathbb{F}$  : satisfying that:
$$
\mathbb {F}: \quad \phi | y \text {i s i n d e p e n d e n t o f} t \text {i f} p (\theta | y) = q (\theta | y), \text {a . e .}
$$
Proof of Theorem 1.
Proof. The classifier sees features  $\phi$  and labels  $t$ . Given a  $t$ ,  $(\phi, y)$  are IID draws from  $p(y)\pi(\phi | y, t)$ . In case  $y$  is not observable by the classifier,  $\phi | t$  are IID draws from  $\int_y p(y)p(\phi | y, t)$ .
To use Theorem 9, let  $\mathbf{x}^k = \{(\phi_j,y)|t_j = k\}$  and  $w_{k} = \operatorname *{Pr}(t = k)$ . The divergence from Theorem 9: reads
$$
D ^ {\mathrm {o p t}} (p, q, \Phi) = \sum_ {n = 1} ^ {K} w _ {k} \mathrm {K L} \left(\pi_ {1} (\phi), \sum_ {k = 1} ^ {K} w _ {k} \pi_ {1} (\phi)\right),
$$
in which  $\pi_k(\phi) = p(\phi | t = k) = \int_y p(y) \pi(\phi | y, t = k) dy$  is the  $y$ -averaged  $\phi$  margin. This is essentially Theorem 7.
If  $p = q$ , from  $\mathbb{F}$ , the law of  $\phi, y$  is independent of  $t$  ( $y$  is always independent of  $t$  marginally), so that  $\pi_k(\phi) = \int_y p(y)\pi (\phi |y,t = k)dy$  is also independent of  $t$ , as it only depends on  $y$  and  $\phi$ , hence the divergence remains 0.
C.2 Reweighting (Theorem 8) leads to symmetric divergence
We now prove Theorem 8. The binary label mapping generates  $M$  pairs label-1 examples and one pair of label-0 examples per simulation. We can reweight the binary classifier by letting the ELPD be  $\mathbb{E}[\frac{C}{M + 1}\mathbb{1}(t = 1)\log p(t = 1|c(\phi)) + \frac{CM}{M + 1}\mathbb{1}(t = 0)\log p(t = 0|c(\phi))]$ , where  $C = \frac{(M + 1)^2}{2M}$  is a normalizing constant. We want to prove that after weighting, the classification divergence is symmetric Jensen shannon distance, as if we have balanced classification data.
Proof. After the label-reweighing, the optimal classifier is the balanced-class classifier
$$
c (t = 1 | \phi) = \frac {\operatorname * {P r} (\phi | t = 1)}{\operatorname * {P r} (\phi | t = 1) + \operatorname * {P r} (\phi | t = 0)}.
$$
The re-weighted ELPD in the population is
$$
\begin{array}{l} \mathrm {E L P D} = \mathbb {E} _ {\phi} [ \Pr (t = 0 | \phi) \frac {C M}{M + 1} \log c (t = 0 | \phi) + \Pr (t = 0 | \phi) \frac {C}{M + 1} \log c (t = 1 | \phi) ] \\ = \mathbb {E} _ {y} \left[ \frac {p (\theta | y) + M q (\theta | y)}{M + 1} \left(\frac {p (\theta | y)}{p (\theta | y) + M q (\theta | y)} \frac {C M}{M + 1} \log \frac {p (\theta | y)}{2 r (\theta | y)} + \frac {M q (\theta | y)}{p (\theta | y) + M q (\theta | y)} \frac {C}{M + 1} \log \frac {q (\theta | y)}{2 r (\theta | y)}\right) \right] \\ = \mathbb {E} _ {y} \left[ C \frac {2 M}{(M + 1) ^ {2}} \left(\frac {p (\theta | y)}{2} \log \frac {p (\theta | y)}{2 r (\theta | y)} + \frac {q (\theta | y)}{2} \log \frac {q (\theta | y)}{2 r (\theta | y)}\right). \right] \\ \end{array}
$$
With an appropriate normalizing constant  $C = \frac{(M + 1)^2}{2M}$ ,
$$
\mathrm {E L P D} = \mathbb {E} _ {y} \left[ \frac {1}{2} p (\theta | y) \log \frac {p (\theta | y)}{r (\theta | y)} + \frac {1}{2} q (\theta | y) \log \frac {p (\theta | y)}{r (\theta | y)} \right] - \log 2.
$$
That is, the reweighted ELPD  $+\log 2$  is the conditional Jensen Shannon divergence (15) between  $p(\theta |y)$  and  $q(\theta |y)$ .
C.3 MCMC without thinning (Theorem 6)
To diagnose MCMC, we sample  $(\theta ,y)\sim p(\theta ,y)$ , and sample (potentially autocorrelated) draws  $(\tilde{\theta}_1,\ldots ,\tilde{\theta}_M)$  from a MCMC sampler whose stationary distribution we believe is  $q(\theta |y)$  (i.e., marginally  $\tilde{\theta}_i$  is from  $q(\theta |y)$ ), and generate examples  $((t_1,\phi_1),\dots ,(t_{M + 1},\phi_{M + 1}))$ , from the multiclass permutation (see definition in Example 4), such that
$$
\phi = \left(\theta_ {0}, \theta_ {1}, \dots , \theta_ {M}\right).
$$
Then we run an exchangeable classifier parameterized by
$$
\Pr (t = k \mid \left(\theta_ {0}, \theta_ {1}, \dots , \theta_ {M}, y\right)) = \frac {\exp \left(g \left(\theta_ {k} , y\right)\right)}{\sum_ {k ^ {\prime} = 0} ^ {M} \exp \left(g \left(\theta_ {k ^ {\prime}} , y\right)\right)}, \tag {16}
$$
where  $g$  is any  $\Theta \times \mathcal{V}\to \mathbb{R}$  mapping to be learned.
Note that here  $\operatorname*{Pr}(t = k | (\theta_0, \theta_1, \dots, \theta_M, y))$  is the classifier model we restrict it to be, not the true population  $\pi$ . In general  $\pi(t = k | (\theta_0, \theta_1, \dots, \theta_M, y)) = \neq \operatorname*{Pr}(t = k | (\theta_0, \theta_1, \dots, \theta_M, y))$  Roughly speaking, the optimal classifier should be the Bayes classifier projected to the restricted space (16), and we need to proof that this projected restricted solution turns out to be the same as the IID case 10, which is not trivial in general.
Proof. Intuitively, this separable network design (16) avoids the interaction between  $\tilde{\theta}_m$  with  $\tilde{\theta}_{m'}$  and disallows the network to predict  $t$  based on the autocorrelation or clustering of  $\tilde{\theta}$ .
Because of the permutation design and we define  $q$  to be the marginal distribution of  $\tilde{\theta}$ , first, we know that
$$
\pi (\theta_ {k} | y, t = k) = p (\theta_ {k} | y), \pi (\theta_ {k} | y, t \neq k) = q (\theta_ {k} | y),
$$
in the MCMC population  $\pi$  (there is no need to address the joint for now).
From this we obtain the conditionals in the population
$$
\pi (t = k | \theta_ {k}, y) = \frac {p (\theta_ {k} | y)}{p (\theta_ {k} | y) + M q (\theta | y)}
$$
and further the ratios for any  $m, k$  within the index set  $\{0, 2, \ldots, M\}$  and  $m \neq k$ :
$$
\pi (t = k | \theta_ {k}, \theta_ {m}, y) = \frac {p (\theta_ {k} | y) q (\theta_ {k} | y)}{p (\theta_ {k} | y) q (\theta_ {m} | y) + q (\theta_ {k} | y) p (\theta_ {k} | y) + (M - 1) q _ {1 2} (\theta_ {k} , \theta_ {m})}.
$$
Here  $q_{12}(\theta_k, \theta_m)$  is the joint distribution of two out of  $M$  draws from the MCMC sampler  $q(\theta | y)$ , which would often not be the same as  $q(\theta_k | y) q(\theta_m | y)$  because of the autocorrelation of Markov chains. We do not need to specify the form of this joint. The key blessing is that when  $\theta_k$  is from  $p(\theta | y)$  and  $\theta_m$  is from  $q(\theta | y)$ , then they are independent (the numerator).
Next, from the line above we obtain the ratio estimate in the true population  $\pi$ :
$$
\frac {\pi \left(t = k \mid \theta_ {k} , \theta_ {m} , y\right)}{\pi \left(t = m \mid \theta_ {k} , \theta_ {m} , y\right)} = \frac {p \left(\theta_ {k} \mid y\right) q \left(\theta_ {k} \mid y\right)}{q \left(\theta_ {k} \mid y\right) p \left(\theta_ {k} \mid y\right)} = \frac {p \left(\theta_ {k} \mid y\right) / q \left(\theta_ {k} \mid y\right)}{p \left(\theta_ {m} \mid y\right) / q \left(\theta_ {m} \mid y\right)}. \tag {17}
$$
Now we project the restricted classifier (16). Intuitively, the separable classifier "almost" only depends on  $\theta_t$ , except for the normalizing constant. We can remove the dependence on the normalizing constant by specifying the ratio