SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) 100 observations (generated via sbibm package)
Figure 6: Accuracy / correlation of  $\ell$ -C2ST(-NF) w.r.t. the oracle C2ST. We show scatter plots for all sibbm examples on (a) 10 and (b) 100 different reference observations. Each point corresponds to one observation and represents the MSE test statistic obtained for the oracle C2ST (x-axis) vs. our  $\ell$ -C2ST(-NF) method (y-axis). The diagonal represents the case where  $\ell$ -C2ST(-NF) = C2ST. The closer points are to the diagonal, the more accurate  $\ell$ -C2ST is w.r.t. C2ST.
\(N_{train}\)
\(10^{2}\)	\(10^{3}\)	\(10^{4}\)	\(10^{5}\)
SLCP	\(10^{-4}/0.12\)	\(10^{-3}/0.03\)	0.31 / 0.82	0.21 / 0.40
Two Moons	\(10^{-27}/10^{-9}\)	\(10^{-3}/0.19\)	\(10^{-16}/10^{-11}\)	\(0.052/10^{-5}\)
Gaussian Mixture	\(10^{-8}/10^{-12}\)	\(10^{-7}/0.01\)	0.006 / 0.35	\(10^{-14}/0.006\)
Gauss. Linear Unif.	\(10^{-13}/10^{-12}\)	0.07 / 10\(^{-9}\)	0.42 / 0.002	0.68 / 0.87
Bernoulli GLM	\(10^{-8}/10^{-5}\)	\(10^{-10}/10^{-4}\)	0.67 / 0.18	0.39 / 0.31
Bernoulli GLM Raw	0.03 / 0.37	\(10^{-8}/10^{-4}\)	\(10^{-4}/0.04\)	0.92 / 0.06
Table 3: P-values of the Pearson test of non-correlation between the  $\ell$ -C2ST/ -NF and the oracle C2ST MSE test statistic. Obtained for 100 observations (as plotted in Figure 6b) using scipy.stats.pearsonr. Blue values indicate the cases for which the Pearson test rejects the null hypothesis of non-correlation with  $95\%$  confidence (i.e. there is significance evidence for correlation).
F.3 Benefit of the -NF version
The results of all benchmark experiments indicate that the -NF version of  $\ell$ -C2ST works better when the (true) posterior distribution of the model is "more complicated" than a Gaussian distribution (see Table 2 at the beginning of Appendix F). This is for example the case for the Two Moons and SLCP tasks: the posterior distributions are globally multi-modal and locally structured. We observe in Column 3 of Figure 2 in the main paper that the -NF version requires less samples (i.e. lower  $N_{\mathrm{cal}}$ ) to reach maximum power/TPR. This is also the case for the additional Bernoulli GLM task (see Column 3 of Figure 5 in Appendix F.1). In contrast, for Gaussian Mixture and Gaussian Linear Uniform, where the posterior is Gaussian, the normal  $\ell$ -C2ST is as powerful or even better than its -NF counterpart.
Finally, we refer the reader to the analysis in Section F.1 to point out an interesting observation: for the Bernoulli GLM, the -NF version scales much better to high dimensional observation spaces than the normal  $\ell$ -C2ST. Note that this does not allow us to make any general conclusions, but it might be worth further investigating this result.
F.4 Advantages of  $\ell$ -C2ST w.r.t. local-HPD
First of all, it is important to mention that having uniform HPD-values is not a sufficient condition for asserting the null hypothesis of consistency (see end of Section 3.3 in [48]). This is a clear disadvantage compared to our proposal, which provides a necessary and sufficient proxy for inspecting local posterior consistency.
Furthermore, the HPD methodology summarizes the whole information concerning  $\theta$  into a single scalar, while in  $\ell$ -C2ST we handle the  $\theta$ -vector in its multivariate form. In medium-high  $\theta$ -dimensions ( $m > 2$  as in Bernoulli GLM) or for complex posterior distributions (SLCP), such summarized information might discard too much information and not be enough to satisfactorily assess the consistency of the posterior estimator. Indeed, the tasks where local-HPD has similar statistical power to  $\ell$ -C2ST are either in low dimensions (Two Moons) and / or have a Gaussian posterior (Gaussian Mixture / Gaussian Linear Uniform). See Table 2 for a summary of those results w.r.t. data dimensionality and posterior structure. For a detailed analysis of the scalability to high dimensional data spaces see Appendix F.1.
Finally, local-HPD in its naive implementation is much less efficient than  $\ell$ -C2ST (see Appendix C). Note however that a new "amortized" version of local-HPD has recently been proposed [9] and could be interesting to look at.