SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\gamma_{*} = \operatorname{argmax}_{\gamma}\mathcal{T}(\hat{X},\hat{Y})$  for  $\gamma \in \{med^2\cdot 2^{-4},med^2\cdot 2^{-3.8},med^2\cdot 2^{-3.6},\dots ,med^2\cdot 2^4\}$  (36)
where  $med$  denotes the median of pairwise Euclidean distances of points in  $X$  and  $Y$ .
Compared methods
We now present the details for our compared methods as follows:
-  $\mathbf{ME}^2$ : The mean embedding method learns a set of test locations and a single Gaussian kernel, and then measures the difference between two mean embeddings with a statistic following  $\chi^2$  distribution [9, 10];
2The code is downloaded from github.com/wittawatj/interpretable-test.
Algorithm 3  $\mathrm{ME}_{\mathrm{MaBiD}}$  testing
Input: Two testing samples  $X^{\prime}$  and  $Y^{\prime}$ , test locations  $\{\pmb{v}_i\}_{i=1}^\ell$ , Mahalanobis kernels parameters  $\{M_i\}_{i=1}^\ell$  and  $\{\gamma_i\}_{i=1}^\ell$ , inference direction  $\pmb{F}$ , significance level  $\alpha$  and parameter  $\beta$
Output:  $h$
1: Calculate the embedding testing samples  $\hat{X}^{\prime}$  and  $\hat{Y}^{\prime}$  based on Eqn. (2)
2: Calculate mean embeddings  $c_{\hat{X}^{\prime}} = \sum_{i = 1}^{m^{\prime}}\hat{x}_{i}^{\prime} / m^{\prime}$  and  $c_{\hat{Y}^{\prime}} = \sum_{i = 1}^{n}\hat{y}_{i}^{\prime} / n^{\prime}$
3: Calculate the pooled covariance matrix  $\Sigma_{\hat{X}^{\prime},\hat{Y}^{\prime}}$  based on Eqn. (3)
4: Calculate the Schur decomposition of the pooled covariance matrix:  $L^{\prime}L^{\prime} = \Sigma_{\hat{X}^{\prime},\hat{Y}^{\prime}}$
5: if  $F^T(c_{\hat{X}'} - c_{\hat{Y}'}) \geq 0$  then
6:  $h = \mathbb{I}[\chi_{\ell}^{2}(\mathcal{T}(\hat{X}^{\prime},\hat{Y}^{\prime}))\leq \beta \alpha ]$
7: else
8:  $h = \mathbb{I}[\chi_{\ell}^{2}(\mathcal{T}(\hat{X}^{\prime},\hat{Y}^{\prime}))\leq (2 - \beta)\alpha ]$
9: end if
10: return h
Algorithm 4 Exploring the local significant differences
Input: Two testing samples  $X, Y$ , Mahalanobis kernels' parameters  $\{M_i\}_{i=1}^{\ell}$  and  $\{\gamma_i\}_{i=1}^{\ell}$ , test locations  $\{\pmb{v}_i\}_{i=1}^{\ell}$ , partition tree  $T$
Output:  $t^*$
1: Calculate the embedding testing samples  $\hat{X}^{\prime}$  and  $\hat{Y}^{\prime}$  based on Eqn. (2)
2: Get the local two samples  $\hat{X}_{\mathcal{B}_i}'$  and  $\hat{Y}_{\mathcal{B}_i}'$  for every  $i \in [s]$  based on partition Tree  $T$
3: Calculate bi-directional masked  $p$ -value  $g(\hat{X}_{B_i}, \hat{Y}_{B_i})$  for each rectangle region  $\mathcal{B}_i$
4: Resort rectangle regions as  $\mathcal{B}_{\langle 1\rangle},\mathcal{B}_{\langle 2\rangle},\ldots ,\mathcal{B}_{\langle s\rangle}$  in a non-increasing order based on  $g(\cdot ,\cdot)$
5: Calculate bi-directional hypothesis  $h(\hat{X}_{\mathcal{B}_i}^{\prime}, \hat{Y}_{\mathcal{B}_i}^{\prime})$  for each rectangle region  $\mathcal{B}_i$  by Eqn. (7)
6: Calculate the index set  $t^*$  based on Eqn. (11)
7: return  $t^*$
- C2ST-S<sup>3</sup>: A binary classification neural network is trained and the statistic is computed as the accuracy over a hold-out set of two samples [13];
- MMDAgg<sup>4</sup>: A solution for the fundamental kernel selection problem involves the aggregation of a large number of kernels with several bandwidths, where the incomplete  $U$ -statistics are used to measure the difference between two samples Schrab et al. [75]. Notice that we set the testing sample size for MMDAgg to 5 times that of the other methods, since it does not require training;
- MMD-D<sup>5</sup>: A deep kernel approach for Maximum Mean Discrepancy (MMD), where the parameters of a neural network, two lengthscales of Gaussian kernels and a regularization parameter are optimized [39];
- C2ST-L<sup>6</sup>: A binary classification neural network is trained and the statistic is computed as the difference between outputs of the logit function corresponding to two samples [14, 18];
- AutoML<sup>7</sup>: A binary classifier is trained based on Automated Machine Learning techniques and the statistic is same to C2ST-L approach [19].
We implement methods for exploring local significant differences in Python, following their respective guidelines as follows:
- FDG: Partition the sample space based on probability binning and then compare the cardinalities of two samples over rectangle regions, where a normalized chi-squared value is computed for each bin to measure the local difference [24];
3The code is downloaded from github.com/lopezpac/classifierTests.
<sup>4</sup>The code is downloaded from github.com/antoninschrab/agginc-paper.
The code is downloaded from github.com/fengliu90/DK-for-TST.
The code is downloaded from github.com/xycheng/net_logit_test.
The code is downloaded from github.com/jmkuebler/autoML-TST-paper.
Step I: Initialization with root 1
Partition tree
$①$
Step II: Split node 1 at x-axis with median position
Step III: 1) Split node 2 at x-axis with median position 2) Split node 3 at y-axis with median position
Splitting regions
Step IV: 1) Split node 4 at y-axis with median position
2) Split node 5 at x-axis with median position
3) Split node 6 at x-axis with median position
4) Split node 7 at y-axis with median position
Splitting regions
Figure 9: An illustration for our partition tree with splitting regions.
- K-PRIM: Partition the space based on patient rule induction method along with the information of kernel density estimation, and use a statistic following  $\chi_1^2$  distribution to identify local distributional difference [28];
- MRS: Partition the sample space based on polya tree method, which splits a leaf node based on the median of randomly selected feature, and then measure the difference between local two samples based on Binomial distribution [26];
- TEAM: Partition the sample space on the median of the feature with largest sample variance, and then measure differences between local two samples with Binomial distribution [27];
- BTLDD: Estimate the conditional probabilities of two samples based on a regression model and then cluster those elements with significant different conditional probabilities [77];
- MMDT: Partition the sample space into multiple equal grids based on quantile values of the features, and then estimate the kernel densities of two samples and measure local differences based on Welch's two-sample t-test statistic [29].
For the exploration of local significant differences, we take density differences [78] between two samples in a local region as an evaluation measure for local significant differences, and follow the works of [79, 80] based on  $k$ -NN density estimator with  $k = 20$ . Here, denote by  $X$  and  $Y$  the available data for density evaluation, we have estimated density functions as follows:
$$
f _ {X} (z) := \frac {2 0}{N \cdot v _ {d} \cdot r _ {X} (z) ^ {d}} \quad \text {a n d} \quad f _ {Y} (z) := \frac {2 0}{N \cdot v _ {d} \cdot r _ {Y} (z) ^ {d}},
$$
where  $z \in [0,1]^d$ ,  $N$  denotes sample size for density estimation and  $v_{d}$  is the volume of a unit ball in  $\mathbb{R}^d$ . Denote by  $r_X(\pmb {z})^d$  and  $r_Y(\pmb {z})^d$  the distances from  $z$  to its 20-th nearest neighbors in  $X$  and  $Y$ , respectively. For density estimation, we use all data of diamond, and 1,000,000 data of other datasets due to the limitation of time complexity.
For our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test, we present the detailed training procedure in Algorithm 2 and testing procedure in Algorithm 3. Figure 9 is a pictorial illustration to present the region-splitting method and clarify that our partition tree is constructed iteratively: We initiate tree root with embedding space, and during each iteration, we select randomly one of those leaves with the largest size of training data points, and select the feature of the largest statistics value and with the median splitting position. We present the detailed description on the exploration of local significant differences in Algorithm 4.
Figure 10: Comparisons of training time for different methods on two-sample test. Note that y-axis is in log-scale.
Figure 11: Comparisons of testing time for different methods on two-sample test. Note that y-axis is in log-scale.
D Experimental comparisons
For fair comparisons, we run all experiments on a single core without parallel optimizations, and experiments are performed with Python on nodes of a computational cluster with a single CPU (Intel Core i9-10900X 3.7GHz) and a single GPU (GeForce RTX 2080 Ti), running Ubuntu with 128GB main memory. For these methods based on deep neural networks, such as C2ST-S, C2ST-L and MMD-D, we run experiments on GPU. For other methods except for AutoMLTST, we run experiments on CPU. For AutoMLTST, we run experiments on GPU for cifar and mnist, and run experiments on CPU for other datasets.
We further compare the average training time and testing time (in seconds) for different methods on two-sample test, as shown in Figure 10 and Figure 11. Notice that, the training time of MMDAgg is 0, since it has no training procedure. In testing procedure, our  $\mathrm{ME}_{\mathrm{MaBiD}}$  and ME methods calculate the rejection threshold based on the asymptotic distribution  $\chi_{\ell}^{2}$ , whereas other methods perform permutation test or wild bootstrap.
Additional experimental comparisons
Table 5 presents the comparisons of type-I error for different methods. As can be seen, type-I error is limited about  $\alpha = 0.05$  for all compared methods, which shows the effectiveness of test power in experiments for different methods.
Figure 12 shows the experimental comparisons between Mahalanobis kernels and Gaussian kernels, and it is obvious that Mahalanobis kernels achieves higher test power with better performance by exploiting local regions and directional information. Figure 13 presents experimental comparisons of test power with different training sample sizes. The testing sample size is set as 200 for blob, 1000 for higgs, 200 for codrna and 150 for sea50. As can be seen, our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test takes better test powers empirically than other methods w.r.t different training sample sizes by incorporating correlation and directional information of two samples.
Table 5: Comparisons of type-I error (mean±std).
Dataset	Ours MEMaBiD	ME	MMDAgg	MMD-D	C2ST-L	C2ST-S	AutoMLTST
blob	.043± .025	.064± .056	.061± .054	.064± .018	.037± .032	.004± .006	.019± .016
dna	.045± .036	.052± .068	.061± .035	.002± .004	.037± .025	.059± .054	.033± .034
agnostic	.047± .023	.049± .034	.051± .043	.002± .004	.042± .029	.063± .045	.046± .039
topo21	.033± .019	.049± .032	.038± .034	.049± .039	.019± .022	.051± .024	.038± .024
har	.046± .033	.033± .026	.040± .030	.045± .028	.022± .014	.061± .041	.052± .041
kropt	.033± .016	.021± .006	.013± .000	.030± .029	.046± .016	.033± .012	.033± .021
diamon	.042± .024	.029± .024	.017± .012	.046± .023	.013± .010	.021± .016	.033± .031
cifar	.015± .010	.012± .011	.025± .014	.078± .068	.021± .016	.027± .017	.020± .014
mnist	.043± .038	.013± .016	.041± .028	.092± .068	.032± .041	.057± .026	.049± .039
santan	.037± .031	.092± .043	.029± .006	.058± .019	.025± .018	.037± .027	.042± .036
codrna	.013± .010	.017± .016	.075± .080	.042± .028	.046± .047	.042± .012	.008± .012
sea50	.032± .027	.021± .012	.058± .026	.088± .059	.029± .041	.058± .056	.050± .047
adult	.024± .016	.021± .016	.029± .016	.038± .010	.033± .016	.033± .026	.062± .020
labor	.017± .016	.037± .010	.054± .059	.056± .029	.029± .024	.029± .026	.042± .026
poker	.029± .016	.013± .010	.054± .041	.078± .047	.029± .006	.058± .031	.050± .054
higgs	.046± .011	.024± .016	.043± .031	.066± .028	.068± .098	.048± .017	.034± .039
Figure 12: The comparisons of test power vs sample size for Mahalanobis kernel and Gaussian kernel.
Figure 13: The comparisons of test power with different training sample sizes.
Figure 14: The comparisons of test power vs sample size for our bi-directional masked  $p$ -value and prior Sidak.
Figure 15: The comparisons of  $\xi$  with different sample size and number of test locations.
Figure 14 presents experimental comparisons between our bi-directional masked  $p$ -value and previous Sidak [67]. As can be seen, our bi-directional masked  $p$ -value achieves higher test power by exploiting the significant level of local difference. Figure 15 analyzes the relationship between probability  $\xi = \operatorname*{Pr}[\operatorname{sgn}(F_{\mathcal{B}}^{\top}L_{\mathcal{B}}^{\prime -1}(c_{\hat{X}_{\mathcal{B}}^{\prime}} - c_{\hat{Y}_{\mathcal{B}}^{\prime}})] \geq 1)$  and sample size, by considering different numbers of test locations  $\ell \in \{2,6,10,14,18\}$ . As can be seen,  $\xi$  gradually increases as the sample size for different number of test locations, which shows the necessity of directional information in two-sample test.
Figure 16 analyzes the relationship between the number of local regions  $s$  and probability  $\xi = \operatorname{Pr}[\operatorname{sgn}(\pmb{F}_{\mathcal{B}}^{\top}\pmb{L}_{\mathcal{B}}^{\prime -1}(\pmb{c}_{\hat{X}_{\mathcal{B}}^{\prime}} - \pmb{c}_{\hat{Y}_{\mathcal{B}}^{\prime}})] \geq 1)$ . The probability  $\xi$  increases with  $s$ , and hence can effectively partition the space into local regions with different inference directions. Figure 17 presents
Figure 16: The comparisons of  $\xi$  for partition trees with  $s \in [2^0, 2^3]$ . The value of  $\xi$  increases as  $s$  increases.
Figure 17: The correlation between the optimal parameter  $\beta$  and probability  $\xi = \operatorname{Pr}[\operatorname{sgn}(\pmb{F}_{\mathcal{B}}^{\top}\pmb{L}_{\mathcal{B}}^{\prime -1}(\pmb{c}_{\hat{X}_{\mathcal{B}}^{\prime}} - \pmb{c}_{\hat{Y}_{\mathcal{B}}^{\prime}})] \geq 1)$ . Here,  $N$  denotes sample size.
additional experiments on the relationship between the optimal parameter  $\beta$  and the probability  $\xi = \operatorname*{Pr}[\pmb{F}^{\top}\pmb{L}^{\prime -1}(\pmb{c}_{\hat{X}^{\prime}} - \pmb{c}_{\hat{Y}^{\prime}})\geq 0]$ , as shown in Figure 6.