SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Datasets
Dataset	#Inst.	#Feat.	Dataset	#Inst.	#Feat.	Dataset	#Inst.	#Feat.	Dataset	#Inst.	#Feat.
dna	3,186	180	kropt	27,705	6	santan	200,000	200	adult	1,000,000	14
agnos	3,468	970	diamon	53,940	9	codrna	487,867	8	labor	1,000,000	16
topo21	8,885	266	cifar10	60,000	3072	blob	1,000,000	2	poker	1,025,010	10
har	10,299	561	mnist	70,000	784	sea50	1,000,000	3	higgs	11,000,000	4
We finally get the partition tree with  $s$  leaf nodes, associated with  $s$  rectangle regions  $\mathcal{B}_1, \mathcal{B}_2, \ldots, \mathcal{B}_s$ . Algorithm 1 presents the detailed description on tree construction and rectangle region splitting.
We take the statistic  $\mathcal{T}(\cdot ,\cdot)$  as a splitting criterion, relevant to test power and data correlations, and it is helpful to exploit local significant difference directly. We also adopt the median splitting position with equal probabilities on partitioned regions, i.e., balanced examples for each partition region, and this could yield better performance than regular grids, as shown empirically in [59, 60].
Our partition tree is different from previous  $p$ -value histogram based on Chi-square test [61, 62], where the difference is measured by cardinalities of elements in two samples over a local rectangle region. Our splitting criterion is also different from that of previous decision trees [63-66], which consider some information-theoretic criterions such as entropy, Gini index, information gain, etc. In comparisons, our statistic  $\mathcal{T}(\cdot ,\cdot)$  is more essential to reflect the test power for two-sample test.
For each rectangle region  $\mathcal{B}_i$ , let  $c_{\hat{X}_{\mathcal{B}_i}}$  and  $c_{\hat{Y}_{\mathcal{B}_i}}$  be the means of  $\hat{X}_{\mathcal{B}_i} = \mathcal{B}_i \cap \hat{X}$  and  $\hat{Y}_{\mathcal{B}_i} = \mathcal{B}_i \cap \hat{Y}$ , respectively. We make similar Schur decomposition  $L_{\mathcal{B}_i} L_{\mathcal{B}_i} = \Sigma_{\hat{X}_{\mathcal{B}_i}, \hat{Y}_{\mathcal{B}_i}}$  for covariance matrix  $\Sigma_{\hat{X}_{\mathcal{B}_i}, \hat{Y}_{\mathcal{B}_i}}$ , and introduce the local inference direction for each  $\mathcal{B}_i$  as follows:
$$
\boldsymbol {F} _ {\mathcal {B} _ {i}} = \operatorname {s g n} \left(\boldsymbol {L} _ {\mathcal {B} _ {i}} ^ {- 1} \boldsymbol {c} _ {\hat {X} _ {\mathcal {B} _ {i}}} - \boldsymbol {L} _ {\mathcal {B} _ {i}} ^ {- 1} \boldsymbol {c} _ {\hat {Y} _ {\mathcal {B} _ {i}}}\right) \in \{- 1, 0, + 1 \} ^ {\ell}. \tag {9}
$$
For different rectangle regions, we could have different or even contrary inference directions, which is helpful to exploit local differences from distributional shapes of two samples.
Exploration of local significant differences
For testing embedding samples  $\hat{X}^{\prime}$  and  $\hat{Y}^{\prime}$ , we denote by  $\hat{X}_{\mathcal{B}_i}' = \mathcal{B}_i \cap \hat{X}^{\prime}$  and  $\hat{Y}_{\mathcal{B}_i}' = \mathcal{B}_i \cap \hat{Y}^{\prime}$  with their respective means  $c_{\hat{X}_{\mathcal{B}_i}'}$  and  $c_{\hat{Y}_{\mathcal{B}_i}'}$  for each rectangle region  $\mathcal{B}_i$ , and calculate testing statistic  $\mathcal{T}_{\mathcal{B}_i} = \mathcal{T}(\hat{X}_{\mathcal{B}_i}', \hat{Y}_{\mathcal{B}_i}')$  by Eqn. (4).
We propose the new bi-directional masked  $p$ -value for each rectangle region  $\mathcal{B}_i$  as follows:
$$
g \left(\hat {X} _ {\mathcal {B} _ {i}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {i}} ^ {\prime}\right) = \left\{ \begin{array}{l l} \min  \left\{\frac {\chi_ {\ell} ^ {2} \left(\mathcal {T} _ {\mathcal {B} _ {i}}\right)}{\beta}, \frac {p _ {*} \left(1 - \chi_ {\ell} ^ {2} \left(\mathcal {T} _ {\mathcal {B} _ {i}}\right)\right)}{1 - \beta p _ {*}} \right\} & \text {f o r} \quad \boldsymbol {F} _ {\mathcal {B} _ {i}} ^ {\top} \boldsymbol {L} _ {\mathcal {B} _ {i}} ^ {\prime - 1} \left(\boldsymbol {c} _ {\hat {X} _ {\mathcal {B} _ {i}} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} _ {\mathcal {B} _ {i}} ^ {\prime}}\right) \geq 0 \\ \min  \left\{\frac {\chi_ {\ell} ^ {2} \left(\mathcal {T} _ {\mathcal {B} _ {i}}\right)}{2 - \beta}, \frac {p _ {*} \left(1 - \chi_ {\ell} ^ {2} \left(\mathcal {T} _ {\mathcal {B} _ {i}}\right)\right)}{1 - (2 - \beta) p _ {*}} \right\} & \text {f o r} \quad \boldsymbol {F} _ {\mathcal {B} _ {i}} ^ {\top} \boldsymbol {L} _ {\mathcal {B} _ {i}} ^ {\prime - 1} \left(\boldsymbol {c} _ {\hat {X} _ {\mathcal {B} _ {i}} ^ {\prime}} - \boldsymbol {c} ^ {\prime} _ {\hat {Y} _ {\mathcal {B} _ {i}} ^ {\prime}}\right) <   0, \end{array} \right. \tag {10}
$$
where  $L_{\mathcal{B}_i}^{\prime -1}$  is from Schur decomposition  $\Sigma_{\hat{X}_{\mathcal{B}_i}^{\prime},\hat{Y}_{\mathcal{B}_i}^{\prime}} = L_{\mathcal{B}_i}^{\prime -1}L_{\mathcal{B}_i}^{\prime -1},p_*\in (0,1)$  is a parameter on significance level, and  $\beta$  is an adaptive parameter. Here, we also consider two discriminative directions  $\pmb {F}_{\mathcal{B}_i}$  and  $-F_{\mathcal{B}_i}$  on each rectangle region  $\mathcal{B}_i$ , which is different from previous masked  $p$ -value [67] without directional information.
Our bi-directional masked  $p$ -value directly reflects the significant level of local difference when there is a significant difference in local  $\mathcal{B}_i$ , similarly to [67]. The smaller the bi-directional masked  $p$ -value, the more significant the local difference. On the other hand, the bi-directional masked  $p$ -value is a random number with uniform distribution over  $(0, p_*)$  when there is no significant difference in  $\mathcal{B}_i$ , since  $\chi_{\ell}^{2}(\mathcal{T}_{\mathcal{B}_{i}})$  follows a uniform distribution in such case [68].
Based on such recognition, we resort rectangle regions as  $\mathcal{B}_{\langle 1\rangle},\mathcal{B}_{\langle 2\rangle},\ldots ,\mathcal{B}_{\langle s\rangle}$  according to their bi-directional masked  $p$  -value, i.e.,
$$
g \left(\hat {X} _ {\mathcal {B} _ {\langle 1 \rangle}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {\langle 1 \rangle}} ^ {\prime}\right) \leq g \left(\hat {X} _ {\mathcal {B} _ {\langle 2 \rangle}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {\langle 2 \rangle}} ^ {\prime}\right) \leq \dots \leq g \left(\hat {X} _ {\mathcal {B} _ {\langle s \rangle}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {\langle s \rangle}} ^ {\prime}\right).
$$
Table 2: Comparisons of test powers (mean±std) on two-sample test. Bold denotes the highest mean in per row.
Dataset	Our MEMaBiD	ME	MMDAgg	MMD-D	C2ST-L	C2ST-S	AutoMLTST
blob	.985±.009	.823±.000	.935±.012	.963±.010	.972±.078	.946±.037	.980±.029
dna	.717±.068	.536±.059	.659±.070	.628±.006	.699±.028	.505±.044	.603±.085
agnos	.812±.018	.602±.033	.779±.046	.734±.006	.742±.012	.679±.051	.632±.077
topo21	.692±.006	.526±.058	.605±.077	.633±.062	.679±.046	.517±.046	.591±.006
har	.858±.065	.816±.015	.814±.026	.728±.064	.761±.093	.738±.063	.740±.058
kropt	.992±.012	.875±.027	.971±.024	.916±.066	.946±.013	.929±.031	.971±.026
diamon	.837±.066	.697±.068	.676±.047	.755±.056	.747±.086	.727±.076	.831±.062
cifar	.893±.022	.859±.075	.866±.091	.878±.090	.834±.099	.798±.019	.882±.086
mnist	.985±.017	.926±.056	.932±.068	.972±.051	.969±.042	.930±.029	.963±.074
santan	1.00±.000	.896±.060	1.00±.000	.887±.021	.911±.084	.850±.021	.954±.012
codrna	1.00±.000	.946±.085	.926±.037	.914±.076	1.00±.000	1.00±.000	.876±.067
sea50	.993±.018	.993±.018	.982±.012	.993±.018	.993±.018	.970±.053	.989±.029
adult	.996±.002	.875±.034	.967±.029	.908±.072	.761±.091	.854±.058	.992±.006
labor	.992±.012	.807±.078	.988±.010	.930±.093	.756±.059	.791±.031	1.00±.000
poker	.821±.079	.719±.096	.712±.033	.701±.056	.743±.039	.731±.052	.832±.048
higgs	.979±.024	.818±.090	.938±.047	.953±.055	.968±.043	.933±.013	.969±.030
Average	.909±.026	.795±.053	.859±.039	.843±.050	.842±.052	.806±.039	.863±.043
We then take our bi-directional hypothesis  $h(\hat{X}_{\mathcal{B}_i}', \hat{Y}_{\mathcal{B}_i}')$  with parameter  $\beta$  and  $\alpha = p_*$  as in Eqn. (7), and finally get the local regions with significant differences as
$$
\left\{\mathcal {B} _ {\langle i \rangle}: i \leq t ^ {*} \text {a n d} h \left(\hat {X} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}\right) = 1 \right\},
$$
$$
t ^ {*} = \underset {t \in [ s ]} {\arg \max } \left\{t - \left| \left\{i \in [ t ]: h \left(\hat {X} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}, \hat {Y} _ {\mathcal {B} _ {\langle i \rangle}} ^ {\prime}\right) = 1 \right\} \right| + 1 \leq \frac {\ln \left(1 - \alpha_ {*}\right)}{\ln \left(1 - p _ {*}\right)} \right\}. \tag {11}
$$
Here,  $p_*$  is selected as in Eqn. (10), and  $\alpha_* \in [p_*, 1)$  is a parameter to control the probability of mis-identifying at least one rectangle region without significant difference, also called familywise error rate [69-72]. We present theoretical analysis for familywise error rate as follows:
Theorem 5. For our exploration, the familywise error rate is upper bounded by  $\alpha_{*}$ , if 1) the  $p$ -values of local regions without differences are mutually independent; and 2) the  $p$ -values of local regions with differences are independent to those  $p$ -values of local regions without differences.
Our method is different from previous space partition methods of trees or clusters [24-27, 60], where the splitting criterion is taken as the cardinalities of samples in each region. Duong [28] partitioned and searched local regions from the estimated density function, and Kim et al. [30] identified local regions by clustering data samples from the estimated conditional probabilities. It is not easy to make accurate estimation for density and conditional probabilities without sufficient data, particularly for multiple small regions. Other relevant studies detected local differences implicitly based on interactive rank test [73] or a learned classifier [74].
We conduct experiments on 16 datasets<sup>1</sup> as summarized in Table 1. Most dataset have been studied in previous two-sample test, and features have been scaled to [0, 1] for all datasets. All experiments are performed with Python on nodes of a computational cluster with a single CPU (Intel Core i9-10900X 3.7GHz) and a single GPU (GeForce RTX 2080 Ti), running Ubuntu with 128GB main memory.
Experimental comparisons for two-sample test
We compare our  $\mathrm{ME}_{\mathrm{MaBiD}}$  with the state-of-the-art approaches on two-sample test as follows:
$^{1}$ Dataset blob is downloaded from github.com/fengliu90/DK-for-TST, and other datasets are downloaded from www.openml.org.
Table 3: Comparisons of density differences (mean±std) on the exploration of local significant differences, and the bold denotes the highest mean in per row.
Dataset	Our method	FDG	KPRIM	MRS	MMDT	BTLDD	TEAM
blob	.945±.082	.902±.075	.879±.045	.849±.075	.909±.091	.932±.046	.877±.067
diamon	.974±.054	.852±.010	.895±.089	.876±.066	.867±.104	.951±.070	.947±.022
codrna	.969±.026	.936±.037	.876±.061	.966±.045	.884±.105	.905±.050	.863±.036
sea50	.977±.056	.975±.062	.933±.074	.928±.058	.985±.045	.892±.059	.944±.065
adult	.953±.048	.862±.044	.838±.109	.911±.085	.927±.101	.875±.074	.880±.070
labor	.959±.062	.894±.080	.905±.016	.900±.037	.911±.093	.922±.048	.932±.067
poker	.945±.030	.901±.030	.882±.027	.925±.023	.927±.057	.894±.028	.884±.064
higgs	.946±.016	.932±.011	.918±.001	.940±.000	.927±.026	.926±.027	.937±.002
Average	.959±.047	.907±.044	.891±.053	.912±.049	.917±.078	.912±.050	.908±.049
- ME: Mean Embeddings over multiple test locations and a single Gaussian kernel [9, 10];
- MMD-D: Maximum Mean Discrepancy based on a Deep kernel [39];
- MMDAgg: Maximum Mean Discrepancy with Aggregating of multiple Gaussian kernels [75];
C2ST-S: Train a binary classification network and test its accuracy on a hold-out set [13];
C2ST-L: Train a binary classification network with a statistic about class probabilities [14, 18];
- AutoMLTST: Train a binary classifier based on AutoML method with a statistic as C2ST-L [19].
Following [39, 76], we train on a subset of each available data, and test on 100 random subsets from the remaining dataset, and the ratio is set as  $4:1$  for training and testing. We repeat such process 10 times for each dataset. More details are given in Appendix C. For our  $\mathrm{ME}_{\mathrm{MaBiD}}$ , we set  $\alpha = 0.05$  and take 5-fold cross validation to select  $\beta \in [1:0.2:2]$ . We limit the cardinality of test locations within 20 for ME and  $\mathrm{ME}_{\mathrm{MaBiD}}$  as in [9-11], and optimization parameters of Eqn. (5) is presented in Appendix C. We take parameter settings for other methods as in their respective inferences.
Table 2 summarizes the average of test powers and standard deviations. It is evident that our  $\mathrm{ME}_{\mathrm{MaBiD}}$  takes better performance than ME and MMDAgg, because they both take Gaussian kernels with isotropic scale, and ignore the distributional differences from different directions. Our method is still better than MMD-D with a deep kernel, and a reason is that multiple Mahalanobis kernels are more flexible than a deep kernel to capture local difference from multiple neighborhoods and directions.
From Table 2, it is also observed that our  $\mathrm{ME}_{\mathrm{MaBiD}}$  outperforms three classifier-based methods C2ST-S, C2ST-L and AutoML expect for datasets labor and poker, since those methods focus merely on the prediction information from outputs of classifiers, rather than local and directional information among data samples. For datasets labor and poker, AutoML generates new features automatically from the original mixture of continuous and symbolic features, and thus achieves better performance.
We further compare the average running time (in seconds) for different methods on two-sample test, as shown in Figure 3. As expected, ME takes the least running time since it considers only one Gaussian kernel, yet with the smallest average of test powers in Table 2. Our  $\mathrm{ME}_{\mathrm{MaBiD}}$  method takes smaller and comparable running time in contrast to other methods since our method takes relatively smaller time on training Mahalanobis kernels without permutation test in the testing process.
Experiments on the exploration of local significant differences
We compare with the state-of-the-art approaches on exploring local significant difference as follows:
- FDG: Partition space by probability binning and compare cardinalities of two samples [24];
- K-PRIM: Partition space by patient rule induction and estimate kernel density differences [28];
- MRS: Partition space by polya tree and measure difference via Binomial distributions [26];
- TEAM: Partition space by data variance and measure difference via Binomial distributions [27];
- BTLDD: Estimate conditional probabilities of two samples and cluster data via difference [77];
- MMDT: Partition space into equal grids and test density difference via Welch's statistic [29].
Figure 3: Comparisons of running time for different methods on two-sample test. Note that y-axis is in log-scale.
Figure 4: Visualization of the local significant differences with partition tree of  $s = 8$  (left) and  $s = 16$  (right) rectangle regions. The bigger the values, the larger the density difference.
For fair comparisons with some methods of density estimations, we select eight large datasets: diamond, codrna, blob, sea50, adult, labor, poker and higgs, whose instance numbers are more than 50,000 and feature numbers are smaller than 20. We randomly select 10,000 instances during the exploration of local significant differences. For the BTLDD method, we set the percentage of selected samples as  $6.25\%$  for local regions in a cluster, while for other methods, we partition the space into 64 rectangle regions, and identify 4 rectangle regions with the most local differences.
We take density differences [78] between two samples in a local region as an evaluation measure for local significant differences, and follow the works of [79, 80] based on  $k$ -NN density estimator with  $k = 20$ . We calculate the difference between two estimated density functions in an identified local region, and normalize the returned values into [0, 1].
Table 3 summarizes the average of density differences and standard deviations. As can be seen, our method takes better explorations on local significant difference than FDG, MRS and TEAM, since those methods simply take cardinalities of samples to measure significant difference. Our exploration is still better than K-PRIM, BTLDD and MMDT except for dataset sea50, because those methods estimate density or conditional probabilities from two samples, and it is not easy to make accurate estimation without sufficient data, especially for small regions with finite samples. MMDT could estimate density function well on dataset sea50, and achieves better exploration via Welch's statistic.
We further visualize the local significant differences on mnist (sample  $X$ ) and Fake-mnist (sample  $Y$ ) in Figure 4. Here, we take partition trees with  $s = 8$  (left) and  $s = 16$  (right) rectangle regions. As can be seen, our method achieves the largest local differences via new splitting criterion for partition tree, and our proposed  $\mathrm{ME}_{\mathrm{MaBiD}}$  test and bi-directional masked  $p$ -values.
Parameter analysis
We now present some parameter influence for our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test and the exploration of local significant difference. We only present the results on four datasets due to pages limit, but the trends are similar on other datasets, and more results can be found in Appendix D.
Figure 5: The comparisons of test power vs sample size for our bi-directional hypothesis and previous one/non-directional hypothesis.
Figure 6: The relationship between the optimal adaptive parameter  $\beta$  and probability  $\xi$  for our  $\mathrm{ME}_{\mathrm{MaBiD}}$ .
Figure 7: The type-I error is limited about  $\alpha = 0.05$  w.r.t different  $\beta$  for our  $\mathrm{ME}_{\mathrm{MaBiD}}$ .
Figure 8: The FWER is limited about  $\alpha_{*} = 0.05$  w.r.t. different  $s$  for exploring local significant difference.
Figure 5 illustrates the test power versus sample size for our bi-directional hypothesis, non-directional hypothesis and one-directional hypothesis. As can be seen, our bi-directional hypothesis achieves higher test power by considering the inference and its contrary direction and adaptive parameter selection. Figure 6 exploits the relationship between the optimal parameter  $\beta$  and the probability  $\xi = \operatorname*{Pr}[F^{\top}L'^{-1}(c_{\hat{X}'} - c_{\hat{Y}'}) \geq 0]$  for our  $\mathrm{ME}_{\mathrm{MaBiD}}$  method. We can easily find the positive relevance between  $\beta$  and  $\xi$ : the larger the probability  $\xi$ , the larger the optimal parameter  $\beta$ .
Figure 7 indicates that the type-I error is limited about  $\alpha = 0.05$  for different  $\beta$  in our experiments, as shown in Theorem 4, and thus our method could effectively control the rate of falsely reject the null hypothesis, which empirically verify the trustworthiness of our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test. Figure 8 empirically shows the familywise error rate is limited about  $\alpha_{*} = 0.05$  for different number of local regions  $s$ ; therefore, our exploring method could control the rate of incorrectly exploiting the local regions with significant difference, and this is nicely in accordance with Theorem 5.
This work takes one more step on the exploration of local significant differences. We propose the  $\mathrm{ME}_{\mathrm{MaBiD}}$  test by exploiting local information from multiple Mahalanobis kernels and introducing bi-directional hypothesis for testing. We partition embedding space via a new splitting criterion, and then identify local significant differences based on our bi-directional masked  $p$ -value and  $\mathrm{ME}_{\mathrm{MaBiD}}$  test. We verify the effectiveness of our proposed methods both theoretically and empirically. An interesting work is to explore other local and directional information for local significant differences.
The authors want to thank the reviewers for their helpful comments and suggestions. This research was supported by National Key R&D Program of China (2021ZD0112802), NSFC (61921006, 62376119) and CAAI-Huawei MindSpore Open Fund. W. Gao is corresponding author of this paper.
[78] M. Sugiyama, T. Kanamori, T. Suzuki, M. Plessis, S. Liu, and I. Takeuchi. Density-difference estimation. In Advances in Neural Information Processing Systems 25, pages 692-700. Curran Associates, Dutchess, NY, 2012.
[79] Y. P. Mack and M. Rosenblatt. Multivariate k-nearest neighbor density estimates. Journal of Multivariate Analysis, 9(1):1-15, 1979.
[80] S. Dasgupta and S. Kpotufe. Optimal rates for k-nn density and mode estimation. In Advances in Neural Information Processing Systems 27, pages 2555–2563. Curran Associates, Dutchess, NY, 2014.
[81] W. Jitkrittum, Z. Szabó, and A. Gretton. An adaptive test of independence with analytic kernel embeddings. In Proceedings of the 34th International Conference on Machine Learning, pages 1742-1751, Sydney, Australia, 2017.
[82] R. A. Johnson and D. W. Wichernand. Applied Multivariate Statistical Analysis. Prentice Hall, 2002.
[83] Papoulis A and U. Pillai. Probability, Random Variables and Stochastic Processes. McGraw Hill, 2001.
[84] K. V. Mardia, J. T. Kent, and J. M. Bibby. Multivariate analysis. Probability and Mathematical Statistics, 1979.
[85] L. Lei and W. Fithian. Adapt: An interactive procedure for multiple testing with side information. Journal of the Royal Statistical Society: Series B, 80(4):649-679, 2018.
[86] S. Rabanser, S. Gunnemann, and Z. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In Advances in Neural Information Processing Systems 32, pages 1396-1408. Curran Associates, Dutchess, NY, 2019.
[87] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the 4th International Conference on Learning Representations, San Juan, PR, 2016.
[88] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, 2015.
[89] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Dutchess, NY, 2019.
A Detailed Proofs for Our Theoretical Results
We begin with some useful definitions and lemmas as follows:
Definition 6. (Random Metric [9]). We say that  $\rho$  is a random metric with values in  $\mathbb{R}$ , indexed with pairs from the set of probability measures  $\mathcal{M}$ , i.e.,  $\rho = \{\rho(\mathbb{P},\mathbb{Q})\colon \mathbb{P},\mathbb{Q}\in \mathcal{M}\}$ , if it satisfies the conditions for a metric with qualification 'almost surely'. Formally, for every  $\mathbb{P},\mathbb{Q},\mathbb{U}\in \mathcal{M}$ , random variables  $\rho(\mathbb{P},\mathbb{Q}),\rho(\mathbb{P},\mathbb{U})$  and  $\rho(\mathbb{Q},\mathbb{U})$  satisfy
1.  $\rho (\mathbb{P},\mathbb{Q})\geq 0$  a.s.
2. if  $\mathbb{P} = \mathbb{Q}$ , then  $\rho (\mathbb{P},\mathbb{Q}) = 0$  a.s, if  $\mathbb{P}\neq \mathbb{Q}$  then  $\rho (\mathbb{P},\mathbb{Q})\neq 0$  a.s.
3.  $\rho (\mathbb{P},\mathbb{Q}) = \rho (\mathbb{Q},\mathbb{P})$  a.s.
4.  $\rho (\mathbb{P},\mathbb{Q})\leq \rho (\mathbb{P},\mathbb{U}) + \rho (\mathbb{U},\mathbb{Q})$  a.s.
Lemma 7. [11] Let  $\mu$  be a absolutely continuous Lebesgue measure on  $\mathbb{R}^d$ . A non-zero analytic function  $f$  can be zero at most in the set of measure 0 w.r.t.  $\mu$ .
Lemma 8. [9] If  $\kappa$  is a bounded and analytic kernel on  $\mathbb{R}^d\times \mathbb{R}^d$ , then it is analytic for every function in the RKHS associated with this kernel.
Definition 9. For test locations  $\mathcal{V} = \{\pmb{v}_1, \pmb{v}_2, \dots, \pmb{v}_{\ell}\}$  and Mahalanobis kernels  $\kappa_1, \kappa_2, \dots, \kappa_{\ell}$  given by Eqn. (1), we define