SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Suppose that  $\mathcal{H}_0(x_{\mathrm{o}})$  does not hold, there exists a set  $\mathcal{C} = \left\{\theta \in \mathbb{R}^m, p(\theta \mid x_{\mathrm{o}}) \neq q(\theta \mid x_{\mathrm{o}})\right\}$  of strictly positive measure (w.r.t.  $p$  or  $q$ , which ever is non zero on that set). We can decompose  $\mathcal{C}$  into the direct sum of  $\mathcal{A} = \left\{\theta \in \mathcal{C}, q(\theta \mid x_{\mathrm{o}}) < p(\theta \mid x_{\mathrm{o}})\right\}$  and  $\mathcal{B} = \left\{\theta \in \mathcal{C}, q(\theta \mid x_{\mathrm{o}}) > p(\theta \mid x_{\mathrm{o}})\right\}$ . Either  $\mathcal{A}$  or  $\mathcal{B}$  is necessarily of strictly positive measure. Let's say  $\mathcal{A}$  (see Lemma 1 for the symmetric case).
$\mathcal{A}$  is exactly the region where  $\mathrm{Prob}(C = 1\mid \Theta = \theta) > \frac{1}{2}$  and thus where  $f_{x_{0}}(\theta) = 1;\mathcal{B}$  is the region where  $f_{x_0}(\theta) = 0$  . We therefore get that:
$$
\begin{array}{l} \operatorname {A c c} \left(f _ {x _ {\mathrm {o}}}\right) = \frac {1}{2} \left(\int_ {f _ {x _ {\mathrm {o}}} (\theta) = 0} q (\theta \mid x _ {\mathrm {o}}) \mathrm {d} \theta + \int_ {f _ {x _ {\mathrm {o}}} (\theta) = 1} p (\theta \mid x _ {\mathrm {o}}) \mathrm {d} \theta\right) \\ = \frac {1}{2} \left(\int_ {\mathcal {B}} q (\theta \mid x _ {\mathrm {o}}) \mathrm {d} \theta + \int_ {\mathcal {A}} p (\theta \mid x _ {\mathrm {o}}) \mathrm {d} \theta\right) \\ = \frac {1}{2} \left(1 + \int_ {\mathcal {A}} p (\theta \mid x _ {\mathrm {o}}) - q (\theta \mid x _ {\mathrm {o}}) \mathrm {d} \theta\right) \quad (b e c a u s e \int_ {\mathcal {A}} q + \int_ {\mathcal {B}} q = 1) \\ \end{array}
$$
But  $\forall \theta \in \mathcal{A}$ ,  $0 \leq q(\theta \mid x) < p(\theta \mid x)$  (and  $\mathcal{A}$  is of strictly positive measure), so the integral in the last equality is strictly positive and we get  $\mathrm{Acc}(f_x) > \frac{1}{2}$ , which concludes the proof.
Lemma 1. Let  $p$  and  $q$  be two probability density functions defined on a space  $S$ . If there exists a set  $\mathcal{A} \subseteq S$  of strictly positive measure such that  $\forall \theta \in \mathcal{A}$ ,  $q(\theta) < p(\theta)$ , then there exists a set  $\mathcal{B} \subseteq S$  of strictly positive measure such that  $\forall \theta' \in \mathcal{B}$ ,  $q(\theta') > p(\theta')$ .
Proof. We know that  $\int_{\mathcal{S}}p = \int_{\mathcal{S}}q = 1$  or equivalently, using  $\mathcal{S} = \mathcal{A}\cup \mathcal{B} = \{q > p\} \cup \{q\leq p\}$
$$
\int_ {\mathcal {A}} q (\theta) \mathrm {d} \theta + \int_ {\mathcal {B}} q (\theta) \mathrm {d} \theta = \int_ {\mathcal {A}} p (\theta) \mathrm {d} \theta + \int_ {\mathcal {B}} p (\theta) \mathrm {d} \theta = 1.
$$
By grouping the integrals over  $\mathcal{A}$  on one side and the ones over  $\mathcal{B}$  on the other, we get:
$$
\int_ {\mathcal {A}} q (\theta) \mathrm {d} \theta - \int_ {\mathcal {A}} p (\theta) \mathrm {d} \theta = \int_ {\mathcal {B}} p (\theta) \mathrm {d} \theta - \int_ {\mathcal {B}} q (\theta) \mathrm {d} \theta > 0.
$$
which is non-negative because  $\mathcal{A}$  is assumed to be of strictly positive measure and  $q - p > 0$  everywhere in  $\mathcal{A}$ .
The integral of  $p - q$  over  $\{p = q\}$  is zero, which implies that
$$
\int_ {q <   p} (p (\theta) - q (\theta)) \mathrm {d} \theta = \int_ {\mathcal {B} = q \leq p} (p (\theta) - q (\theta)) \mathrm {d} \theta > 0
$$
meaning that the region  $\{\theta \in S, p(\theta) < q(\theta)\}$  is of strictly positive measure, which concludes the proof.
Proof. Let  $d_{x_0}$  be an estimator of  $\mathbb{P}(C = 1 \mid \Theta; x_0)$  defined on  $S_{x_0}$  such that  $f_{x_0} = \mathbb{I}_{d_{x_0} > \frac{1}{2}}$ . As this region contains all the data points  $\Theta_n^q \sim q(\theta \mid x_0)$  ( $C_n = 0$ ) and  $\Theta_n^p \sim p(\theta \mid x_0)$  ( $C_n = 1$ ), the mean squared error  $\hat{t}_{\mathrm{MSE}}(f_{x_0})$  over the dataset  $\mathcal{D} = \{(\Theta_n, C_n)\}_{n=1}^{2N}$  is well defined (9) and
$$
\hat {t} _ {\text {M S E}} \left(f _ {x _ {\mathrm {o}}}\right) \underset {N _ {v} \rightarrow \infty} {\longrightarrow} t _ {\text {M S E}} \left(f _ {x _ {\mathrm {o}}}\right) = \frac {1}{2} \int \left(d _ {x _ {\mathrm {o}}} (\theta) - \frac {1}{2}\right) ^ {2} \left(q \left(\theta \mid x _ {\mathrm {o}}\right) + p \left(\theta \mid x _ {\mathrm {o}}\right)\right) \mathrm {d} \theta .
$$
This integral is zero if, and only if  $\left(d_{x_{\mathrm{o}}}\left(\theta\right)-\frac{1}{2}\right)^{2}=0$  for every  $\theta \in S_{x_{\mathrm{o}}}$  (all terms are non-negative and  $q(\theta \mid x_{\mathrm{o}})+p(\theta \mid x_{\mathrm{o}})>0^{9}$ , which is equivalent to  $d_{x_{\mathrm{o}}}(\theta)=\frac{1}{2}$  for every  $\theta \in S_{x_{\mathrm{o}}}$ . Assuming  $f_{x_{\mathrm{o}}} = f_{x_{\mathrm{o}}}^{\star}$  is optimal, we have that  $d_{x_{\mathrm{o}}}(\theta) = d_{x_{\mathrm{o}}}^{*}(\theta) = \mathrm{P}(C = 1 \mid \theta; x_{\mathrm{o}})$  and we conclude the proof using the result from equation (5) (and knowing that outside of  $S_{x_{\mathrm{o}}}$ ,  $p = q = 0$ ):
$$
t _ {\mathrm {M S E}} (f _ {x _ {\mathrm {o}}} ^ {\star}) = 0 \quad \Longleftrightarrow \quad d _ {x _ {\mathrm {o}}} ^ {\star} (\theta) = \mathrm {P} (C = 1 \mid \theta ; x _ {\mathrm {o}}) = \frac {1}{2}, \forall \theta \in \mathcal {S} _ {x _ {\mathrm {o}}} \quad \underbrace {\Longleftrightarrow} _ {(5)} \quad \mathcal {H} _ {0} (x _ {\mathrm {o}}) \quad \text {h o l d s}.
$$
Proof. Let  $d$  be an estimator of  $\mathbb{P}(C = 1 \mid \Theta; X)$  and  $f = \mathbb{I}_{d > \frac{1}{2}}$  the associated classifier, both defined on  $\mathcal{S} = \{w = (\theta, x) \in R^m \times \mathbb{R}^d, q(\theta, x) + p(\theta, x) > 0\}$ . Suppose that  $f = f^{\star}$  is optimal and let  $x_0 \in \mathbb{R}^d$  be a fixed observation. As explained in section 3, we have that
$$
d ^ {\star} (\theta , x _ {\mathrm {o}}) = \mathbb {P} (C = 1 \mid \theta ; x _ {\mathrm {o}}) = d _ {x _ {\mathrm {o}}} ^ {\star} (\theta) \quad \text {a n d} \quad f ^ {\star} (\theta , x _ {\mathrm {o}}) = f _ {x _ {\mathrm {o}}} ^ {\star} (\theta), \quad \forall \theta \in S _ {x _ {\mathrm {o}}} .
$$
Consider the support  $S_{q,x_{\mathrm{o}}} = \{\theta \in \mathbb{R}^{m}, q(\theta \mid x_{\mathrm{o}}) > 0\} \subset S_{x_{\mathrm{o}}}$  containing all data points  $\Theta_n^q \sim q(\theta \mid x_{\mathrm{o}})$  from our single-class validation set  $\mathcal{D}_{v0} = \{(\Theta_n^q, 0)\}_{n=1}^{N_{v0}}$ . Therefore our single-class test statistic  $\hat{t}_{\mathrm{MSE}_0}$  is well defined (12) and
$$
\hat {t} _ {\mathrm {M S E} _ {0}} \left(f ^ {\star}, x _ {\mathrm {o}}\right) \underset {N _ {v 0} \rightarrow \infty} {\longrightarrow} t _ {\mathrm {M S E} _ {0}} \left(f ^ {\star}, x _ {\mathrm {o}}\right) = \int \left(d _ {x _ {\mathrm {o}}} ^ {\star} (\theta) - \frac {1}{2}\right) ^ {2} q (\theta \mid x _ {\mathrm {o}}) \mathrm {d} \theta
$$
With the same arguments as in the proof of Theorem 2 in A.2, we get that
$$
t _ {\mathrm {M S E} _ {0}} (f ^ {\star}, x _ {\mathrm {o}}) = 0 \quad \Longleftrightarrow \quad d _ {x _ {\mathrm {o}}} ^ {\star} (\theta) = \mathbb {P} (C = 1 \mid \theta ; x _ {\mathrm {o}}) = \frac {1}{2}, \quad \forall \theta \in \mathcal {S} _ {q, x _ {\mathrm {o}}} \quad \Longleftrightarrow \quad \mathcal {H} _ {0} (x _ {\mathrm {o}}) \quad \text {h o l d s} ,
$$
where the second equivalence is true because  $S_{q,x_{\mathrm{o}}} = S_{x_{\mathrm{o}}}$  for  $p(.|x_{\mathrm{o}}) = q(.|x_{\mathrm{o}})$ . Therefore,  $t_{\mathrm{MSE}_0}(f^\star ,x_{\mathrm{o}}) = 0$  is a necessary and sufficient condition for  $\mathcal{H}_0(x_{\mathrm{o}})$ .
N.B. Single-class accuracy  $(\mathrm{Acc}_0)$  does not provide a sufficient condition for  $\mathcal{H}_0(x_{\mathrm{o}})$ .
Note that this integral can also be zero if  $S_{x_0}$  is of measure zero. But as mentioned at the beginning of this appendix, this has generally no practical implications.
Proof. Following the proof of Theorem 1 in A.1, we have that
$$
\mathcal {H} _ {0} \left(x _ {\mathrm {o}}\right) \quad \text {h o l d s} \iff \operatorname {A c c} \left(f _ {x _ {\mathrm {o}}} ^ {\star}\right) = \frac {1}{2} \iff \mathrm {P} \left(f _ {x _ {\mathrm {o}}} ^ {\star} \left(\Theta^ {q}\right) = 0\right) + \mathbb {P} \left(f _ {x _ {\mathrm {o}}} ^ {\star} \left(\Theta^ {p}\right) = 1\right) = 1.
$$
This means that  $\mathrm{Acc}_0(f^\star, x_{\mathrm{o}}) = \mathrm{P}(f_{x_{\mathrm{o}}}^\star(\Theta^q) = 0) = \frac{1}{2}$  can only be a sufficient condition for  $\mathcal{H}_0(x_{\mathrm{o}})$  if  $\mathrm{P}(f_{x_{\mathrm{o}}}^\star(\Theta^q) = 0) = \mathbb{P}(f_{x_{\mathrm{o}}}^\star(\Theta^p) = 1)$ , which is not generally true. In conclusion, evaluating  $\mathrm{Acc}_0(f^\star, x_{\mathrm{o}})$  only provides a necessary condition for the local null hypothesis (see  $(\Rightarrow)$  in A.1).
B Algorithms
B.1 Generating PP-plots with  $\ell$ -C2ST
Algorithm 5:  $\ell$ -C2ST - local PP-plots for any  $x_{\mathrm{o}}$
Input: evaluation data set  $\mathcal{D}_{v0}$ ; an observation  $x_0$ ; estimate  $d$  of the class probabilities; estimates  $\{d_1, \ldots, d_{N_{\mathcal{H}}}\}$  under the null; grid  $\mathcal{G}$  of PP-levels in (0,1); significance level  $\alpha$   
Output: empirical CDF-values  $\{\hat{F}(l;x_0)\}_{l \in \mathcal{G}}$  of predicted class-0 probabilities;  $(1 - \alpha)$ -confidence bands  $\{L_l(x_0), U_l(x_0)\}_{l \in \mathcal{G}}$   
Predict class-0 probabilities  $\{d_0(v,x_0) = 1 - d(v,x_0)\}_{v \in \mathcal{D}_{v0}} / * d$  is an estimate of class 1  
for  $l$  in  $\mathcal{G}$  do  
/* Compute empirical CDFs at  $l$ $\hat{F}(l;x_0) \gets \frac{1}{N_{v0}} \sum \mathbb{I}_{d_0(v,x_0) \leq l}$   
for  $h = 1, \ldots, N_{\mathcal{H}}$  do  
 $\hat{F}_h(l;x_0) \gets \frac{1}{N_{v0}} \sum \mathbb{I}_{d_0(v,x_0) \leq l}$   
/*Compute confidence bands at  $l$ $L_l(x_0), U_l(x_0) \gets q_{\frac{\alpha}{2}}(\{\hat{F}_h(l;x_0)\}_{h=1}^{N_{\mathcal{H}}}), q_{1 - \frac{\alpha}{2}}(\{\hat{F}_h(l;x_0)\}_{h=1}^{N_{\mathcal{H}}}) / *\text{quantiles}$
return  $\{\hat{F} (l;x_{\mathrm{o}})\}_{l\in \mathcal{G}},L_l(x_{\mathrm{o}}),U_l(x_{\mathrm{o}})\}_{l\in \mathcal{G}}$
C Run-times for each validation procedure
To complete the benchmark experiments from Section 4.1, we analyze the run-times of each validation method to compute the test statistic<sup>10</sup> for an NPE of the SLCP-task, obtained for different values of  $N_{\mathrm{train}}$ . Results are displayed in Table 1 and computed on average over all given observations  $x_{\mathrm{o}}$  and for  $N_{\mathrm{cal}}$ -values that ensure high test power. We here focus solely on the SLCP-task, as the higher dimensional observation space allows to illustrate the differences between local methods (trained on the joint data space,  $(\theta, x) \in \mathbb{R}^{5+8}$ ) and the oracle (trained on the parameter space only,  $\theta \in \mathbb{R}^5$ ).
As expected, the computation time increases with the sample size  $N_{\mathrm{cal}}$  (left vs. right part of Table 1). While the oracle C2ST has close to constant run-time for fixed  $N_{\mathrm{cal}}$ , local methods become faster with increasing  $N_{\mathrm{train}}$ . We observe that  $\ell$ -C2ST(-NF) has comparable run-times to the oracle C2ST: the amortization cost is negligible, in particular for difficult tasks involving "good" posterior estimators (high  $N_{\mathrm{train}}$ ). However, this is not the case for local-HPD, which is the most expensive method. Indeed, it involves (1) the costly computation of the HPD statistics on the joint and (2) the training of several (default is  $n_c = 11$ ) classifiers, both of which increase with the sample size and dimension of the data space.
Ncal = 5000	Ncal = 10000
Ntrain	102	103	104	105	102	103	104	105
oracle C2ST	5.47	4.52	5.95	7.56	16.36	18.03	23.3	15.33
l-C2ST	5.92	5.09	1.78	1.81	27.62	34.06	17.9	3.65
l-C2ST-NF	6.98	6.84	6.38	1.72	43.99	25.01	18.56	7.62
local-HPD	282.19	282.85	279.38	282.5	956.91	938.21	682.92	530.45
Table 1: Run-time (in seconds) to compute the test statistic for the SLCP task (mean over observations). C2ST has close to constant run-time for fixed  $N_{\mathrm{cal}}$ . Local methods become faster with increasing  $N_{\mathrm{train}}$  and  $\ell$ -C2ST(-NF) stays comparable to the oracle C2ST, even for  $N_{\mathrm{cal}} = 10000$ . While the amortization cost of  $\ell$ -C2ST is not an issue, local-HPD is always at least 30 times slower.
<sup>10</sup>Note that in order to compute the  $p$ -values, we need to compute the test statistic  $N_{\mathcal{H}}$  times under the null hypothesis. The number of classifiers we need to train depends on how many we need to compute the test statistic ( $N_{\mathcal{H}}$  for  $\ell$ -C2ST vs.  $N_{\mathcal{H}} \times n_c$  for local-HPD). In summary, if  $\ell$ -C2ST is more efficient in computing a single test statistic, it will also be more efficient to compute  $N_{\mathcal{H}}$  test statistics.
D Graphical diagnostics (  $\ell$  -C2ST-NF) for the JRNMM posterior estimator
The following figures present additional results for the interpretability of  $\ell$ -C2ST applied to the JRNMM neural posterior estimator. They complete Figure 4. Results are shown for all given observations associated to ground-truth gain values  $g_{\mathrm{o}} = -20, -15, -10, -5, 0, 5, 10, 15, 20$ .
In each Figure, the top right panel displays the empirical CDF of the classifier (blue) overlaid with the theoretical CDF of the null hypothesis (step function at 0.5) and  $95\%$  confidence region of estimated classifiers under the null displayed in gray. The pairplot displays histograms of samples from the posterior estimator  $q_{\phi}$  within the prior region and dashed lines indicate values of  $\theta_{\mathrm{o}}$  used to generate the conditioning observation  $x_{\mathrm{o}}$ . The predicted probabilities are mapped on the colors of the bins in the histogram. Blue-green (resp. orange-red) regions indicate low (resp. high) predicted probabilities of the classifier. Yellow regions correspond to chance level, thus  $q_{\phi} \approx p$ .
E On the cross-entropy loss for  $\ell$ -C2ST
This section aims at facilitating the understanding of  $\ell$ -C2ST by proving the result of equation (11).
As detailed in section 3, the first step in  $\ell$ -C2ST consists in training a classifier to distinguish between  $N_{\mathrm{cal}}$  data points  $(\Theta_n,X_n)$  and  $(\Theta_n^q,X_n)$  from the joint distributions  $p(\theta ,x)$  and  $q(\theta ,x)$  respectively. Here, the same conditioning observations  $\{X_{n}\}_{n = 1}^{N_{\mathrm{cal}}}$  are used to construct the data samples for each class (see Algorithm 1). We show that this does not affect the objective function and convergence of the classifier.
The theoretical cross-entropy loss function to distinguish between data  $(\Theta, X)$  from class  $C = 1$  and class  $C = 0$  is defined by
$$
l _ {\mathrm {C E}} (d) := - \frac {1}{2} \mathbb {E} _ {(\Theta , X) | C = 1} [ \log (d (\Theta , X)) ] - \frac {1}{2} \mathbb {E} _ {(\Theta , X) | C = 0} [ \log (1 - d (\Theta , X)) ]. \tag {20}
$$
Note that the we have equal marginals  $X \mid (C = 1) \sim p(x) = q(x) = X \mid (C = 0)$ . This allows us to take the expectation over  $X$  and approximate (20) via Monte-Carlo for only one set of conditioning observations  $\{X_{n}\}_{n=1}^{N_{\mathrm{cal}}}$ , and with data points  $\Theta_{n}$  and  $\Theta_{n}^{q}$  respectively associated to class  $C = 1$  and  $C = 0$  for a given  $X_{n}$ :
$$
\begin{array}{l} l _ {\mathrm {C E}} (d) = \mathbb {E} _ {X} \left[ - \frac {1}{2} \mathbb {E} _ {\Theta | X, C = 1} [ \log (d (\Theta , X)) ] - \frac {1}{2} \mathbb {E} _ {\Theta | X, C = 0} [ \log (1 - d (\Theta , X)) ] \right] \tag {21} \\ \approx - \frac {1}{2 N _ {\mathrm {c a l}}} \sum_ {n = 1} ^ {N _ {\mathrm {c a l}}} \log \left(d \left(\Theta_ {n}, X _ {n}\right)\right) + \log \left(1 - d \left(\Theta_ {n} ^ {q}, X _ {n}\right)\right). \\ \end{array}
$$
Therefore, we can train a classifier to minimize (20) using data from the joint distributions with same conditioning observations. The obtained estimate  $d = \arg \min \{l_{\mathrm{CE}}(d)\}$  of the class probabilities is defined for every  $(\theta ,x)\in \mathbb{R}^m\times \mathbb{R}^d$  by  $d(\theta ,x)\approx \frac{p(\theta,x)}{p(\theta,x) + q(\theta,x)}$ . As  $p(x) = q(x)$ , we recover the class probabilities of the optimal Bayes classifier on the conditional data space for any given  $x\in \mathbb{R}^d$ :
$$
d ^ {\star} (\theta , x) = \frac {p (\theta \mid x)}{p (\theta \mid x) + q (\theta \mid x _ {\mathrm {o}})} = d _ {x} ^ {\star} (\theta). \tag {22}
$$
For an example, see works related to neural ratio estimation (NRE) [22, 8]: these algorithms implicitly use a classifier to distinguish between the joint and marginal distributions  $p(\theta, x)$  and  $p(\theta)p(x)$ . Like in our case, both classes are modeled using the same observations  $X_{n}$  obtained via the simulator.
<sup>11</sup>The joint distributions  $p(\theta, x)$  and  $q(\theta, x)$  are both modeled using samples  $X \sim p(x \mid \Theta)$  obtained from the prior  $\Theta \sim p(\theta)$ , which implies that the marginals  $p(x)$  and  $q(x)$  are both defined by  $\int p(x \mid \theta) p(\theta) \, \mathrm{d}\theta$ .
F Additional Experiments on several benchmark examples
The results on the two benchmark examples in Figure 2 give first intuitions about the validity and behaviour of  $\ell$ -C2ST(-NF), our proposal, w.r.t. the oracle C2ST and the alternative local-HPD methods. In this section, Figure 5 extends Figure 2 with results on additional benchmark examples and more detailed results on the correlation between the test statistics of  $\ell$ -C2ST(-NF) and the oracle C2ST for different conditioning observations are shown in Figure 6 and Table 3. We refer the reader to Table 2 for a description of all benchmark examples (data dimensionality, posterior structure, challenges) and a summary of the main results comparing local methods.
While investigating the scalability of the algorithms to high dimensional data spaces, these additional experiments help to further analyze how well  $\ell$ -C2ST(-NF) captures the true C2ST, when it outperforms local-HPD, and better understand the benefit of the -NF version. These points are further detailed in the following subsections. The goal is to create a first guideline for when our proposal can and should be used, while raising awareness of its limitations (i.e. when it should not be used or at least be adpated for improved performance).
Dimension (θ, x)	Posterior structure	Challenge	Better local method
SLCP	low (5, 8)	4 symmetrical modes	complex posterior	l-C2ST-NF
Two Moons	low (2, 2)	bi-modal, crescent shape	global and local structure	local-HPD / l-C2ST-NF
Gaussian Mixture	low (2, 2)	2D Gaussian	large vs. small s.t.d. in GMM	all similar
Gaussian Linear Uniform	medium (10, 10)	multivariate Gaussian	dimensionality scaling	local-HPD / l-C2ST
Bernoulli GLM	medium (10, 10)	unimodal, concave	dimensionality scaling	l-C2ST(-NF)
Bernoulli GLM Raw	high (10, 100)	unimodal, concave	raw observations (no summary stats)	l-C2ST-NF
Table 2: Description of benchmark examples and summary of main results for local methods.
F.1 Scalability to high dimensions
First of all, note that in the specific case of SBI, the dimension of the parameter space is typically of order  $10^{0}$  to  $10^{1}$  and  $m \approx 10^{2}$  is already often considered as high dimensional. The observation space, however, can be of higher dimension (e.g.  $d \approx 10^{3}$  for time-series), but summary statistics are often used to reduce the dimension of the observations to the order of  $d \approx 10^{1}$ . In Section 4 we analyze the results obtained for rather low-dimensional benchmark examples (Two-Moons:  $m = 2$ ,  $d = 2$ , SLCP:  $m = 5$ ,  $d = 8$ ). As an extension of Figure 2, Figure 5 includes additional benchmark examples with low and medium dimensionality: Gaussian Mixture ( $m = 2$ ,  $d = 2$ ) Gaussian Linear Uniform ( $m = 10$ ,  $d = 10$ ) and Bernoulli GLM ( $m = 10$ ,  $d = 10$ ). Furthermore, the Bernoulli GLM Raw task allows us to analyze how our method scales to high dimensional observation spaces only (without parameter-space / task variability): it considers raw observation data with  $d = 100$ , as opposed to sufficient summary statistics in the Bernoulli GLM task.
Column 3 in Figure 5 shows that  $\ell$ -C2ST requires more data to converge to the oracle C2ST (at maximum power  $\mathrm{TPR} = 1$ ) as the data dimensionality increases:  $N_{\mathrm{cal}} \approx 2000$  for Two Moons and SLCP, but  $N_{\mathrm{cal}} \approx 5000$  for the Bernoulli GLM task. Note that the Gaussian Mixture and
Figure 5: Results on additional sbibm benchmark examples using the same experimental setup as for Figure 2 in Section 4 (50 test runs, mean / std over 10 different reference observations).
Gaussian Linear Uniform tasks were not included in this analysis, as here, the difficulty of the classification task has more impact on statistical power than the data dimensionality  $^{12}$ .
Interestingly, we observe in the Bernoulli GLM Raw task, that  $\ell$ -C2ST-NF scales well to the high-dimensional observation space (faster convergence to maximum TPR compared to Bernoulli GLM), while the normal  $\ell$ -C2ST and local-HPD significantly lose in statistical power. It should also be noted that local-HPD performs significantly worse in medium dimensions (cf. Bernoulli GLM or even SLCP) than in low dimensions (Gaussian Mixture and Two Moons), though this could be because of the complex posterior structure.
F.2 Accuracy of  $\ell$ -C2ST(-NF) w.r.t. the true C2ST
Figures 2 and 5 compare our method to the oracle C2ST, but only in terms of statistical power, as the local analysis is limited to the averaged results over 10 different reference observations  $x_{\mathrm{o}}$ . We here provide a more detailed local analysis that examines how the  $\ell$ -C2ST(-NF) test statistics correlate with those from the oracle C2ST, by plotting them against each other. The results obtained for the above mentioned 10 reference observations are shown in Figure 6a. To allow for more robust conclusions, we also show results obtained for a total of 100 different reference observations (cf. Figure 6b), as well as quantitative results on the statistical significance of the correlation indices in Table 3.
Overall, the scattered points are not too far from the diagonal, which indicates that there is some correlation between the test statistics for  $\ell$ -C2ST(-NF) and those from the oracle C2ST. This correlation becomes weaker when  $N_{\mathrm{train}}$  becomes larger, since the test statistics in these cases tend to zero and can start to be confused with noise. This observation is consistent with the results in Table 3, showing the p-values of the Pearson test, a standard tests for the statistical significance of the correlation indices between the scores.
Another general trend is that the scattered points tend to be below the diagonal, indicating that  $\ell$ -C2ST(-NF) is less sensible to local inconsistencies than the oracle C2ST. This behaviour was expected, as  $\ell$ -C2ST is trained on the joint and thus less precise. Interestingly this doesn't apply to
12Local methods a TPR  $< 1$  at  $N_{\mathrm{train}} = 1000$  (see Column 2), which means that the classification task is harder and requires more data: local methods never reach maximum TPR  $= 1$ , and even the oracle C2ST (with MSE test statistic) needs  $N_{\mathrm{cal}} = 2000$ , at least four times more than for the other tasks.
the Gaussian Mixture task. This could be due to a big variability in the local consistency of  $q$ : while trained on the joint  $\ell$ -C2ST(-NF) could overfit on the "bad" observations, resulting in higher test statistics for observations where the true C2ST statistic would be small.
Finally, across all benchmark examples we observe results that are consistent with the ones from Figure 5: higher correlation means higher statistical power.
(a) 10 observations (same as in benchmark experiment in Figures 2 and 5)