SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhi-Jian Zhou, Jie Ni, Jia-He Yao, Wei Gao
{zhouzj, nij, yaojh, gaow}  $@$  lamda.nju.edu.cn
Recent years have witnessed increasing attentions on two-sample test with diverse real applications, while this work takes one more step on the exploration of local significant differences for two-sample test. We propose the  $\mathrm{ME}_{\mathrm{MaBiD}}$ , an effective test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. On the exploration of local significant differences, we first partition the embedding space into several rectangle regions via a new splitting criterion, which is relevant to test power and data correlation. We then explore local significant differences based on our bi-directional masked  $p$ -value together with the  $\mathrm{ME}_{\mathrm{MaBiD}}$  test. Theoretically, we present the asymptotic distribution and lower bounds of test power for our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test, and control the familywise error rate on the exploration of local significant differences. We finally conduct extensive experiments to validate the effectiveness of our proposed methods on two-sample test and the exploration of local significant differences.
Two-sample test has attracted much attention with diverse applications such as cancer detection [1], distribution-shift detection [2], generative modeling [3, 4], etc. The basic problem is to assess whether two i.i.d. samples are drawn from the same distribution. Various kernel-based methods have been developed for two-sample test such as Maximum Mean Discrepancy (MMD) [5-8] and Mean Embedding (ME) [9-11]. Another relevant approach is to construct a binary classifier and assess two samples according to classification performance [12-20]. For an overview of two-sample test, we refer to a survey [21, and references therein].
In many real applications, however, it is necessary to take one more step to explore and understand local significant differences, rather than only two-sample test. For example, a scientific problem in galaxy morphology is to identify some local regions of significant differences between two kinds of galaxies, which is important to discover galaxy formation and evolution history [22]. On the analysis of mass cytometry data in cell biology, researchers are always interested in finding local regions of significantly different abundance between disease and healthy samples [23].
Several attempts have been made to explore local significant differences in the past years. A feasible solution is to partition space into several regions, and identify significantly different regions according to their cardinalities of samples [24-27]. Another relevant work is to identify local significant differences by estimating kernel densities [28, 29] and conditional probabilities [30]. Generally, it is not easy to deal with complex data by simply counting cardinalities of samples, regardless of data intrinsic correlations, and it is also difficult to make accurate estimation of kernel densities and conditional probabilities without sufficient data, especially for many regions with finite samples.
Figure 1: An illustration of different contours of Mahalanobis and Gaussian kernels for two-sample test.
This work presents a new two-sample test from local and directional information, and further explore local significant differences. The main contributions can be summarized as follows:
- We propose the effective  $\mathrm{ME}_{\mathrm{MaBiD}}$  test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. Intuitively, Mahalanobis kernels are more flexible to exploit local differences from neighborhoods and feature maps, and the bi-directional hypothesis is beneficial to improve the sensitivity of two-sample test with proper parameter adaptation.
- We partition the embedding space into several rectangle regions based on a new splitting criterion, which is relevant to test power and data correlation. We introduce the bi-directional masked  $p$ -value for each rectangle region, and finally explore local regions with significant difference based on our bi-directional masked  $p$ -value together with the  $\mathrm{ME}_{\mathrm{MaBiD}}$  test.
- We present theoretical guarantees for our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test via the asymptotic distribution, as well as the lower bounds on the test power for our test. We also present the upper bounds on familywise error rate for our exploration of local significant differences.
- We conduct extensive experiments to validate the effectiveness and efficiency of our methods. Specifically, our methods achieve better performance on most datasets for two-sample test and exploring local significant differences, along with comparable or smaller running time.
The rest of this work is organized as follows: Section 2 presents our  $\mathrm{ME}_{\mathrm{MaBiD}}$  test. Section 3 explores local significant differences. Section 4 conducts extensive experiments, and Section 5 concludes with future work. All technical proofs are given in Appendix A.
2 Our  $\mathbf{ME}_{\mathrm{MaBiD}}$  Test for Two-Sample Testing
Let  $\mathbb{P}$  and  $\mathbb{Q}$  denote two (unknown) Borel probability measures over an instance space  $\mathcal{X} \subseteq \mathbb{R}^d$ , and  $X = \{\pmb{x}_i\}_{i=1}^m$  and  $Y = \{\pmb{y}_j\}_{j=1}^n$  are two i.i.d. samples from  $\mathbb{P}$  and  $\mathbb{Q}$ , respectively. The goal of two-sample test is to assess whether  $X$  and  $Y$  are drawn from the same distribution; in other words, we aim to assess whether  $\mathbb{P} = \mathbb{Q}$  from two samples  $X$  and  $Y$ .
We introduce some necessary notations used in this work. Write  $[\tau] = \{1,2,\dots,\tau\}$  for integer  $\tau \geq 2$ , and  $|A|$  denotes the cardinality of set  $A$ . Let  $\mathbf{I}_d$  be the identity matrix of size  $d\times d$ . For a vector  $\pmb{a} = [a_1,a_2,\dots ,a_d]$ , denote by  $\operatorname{sgn}(\pmb{a}) = [\operatorname{sgn}(a_1),\operatorname{sgn}(a_2),\dots ,\operatorname{sgn}(a_d)]$  with  $\operatorname{sgn}(a_i) = a_i / |a_i|$  for  $a_i\neq 0$ ; otherwise,  $\operatorname{sgn}(a_i) = 0$ . Let  $\chi_{\ell}^{2}$  be the  $\chi^2$  distribution with  $\ell$  degree of freedom, as well as the  $p$ -value function  $\chi_{\ell ,\alpha}^{2}(\cdot)$ . Denote by  $\chi_{\ell ,\alpha}^{2}$  the  $\alpha$ -quantile of distribution  $\chi_{\ell}^{2}$  for  $\alpha \in (0,1)$ .
Learning multiple Mahalanobis kernels via maximizing test power in training
Following ME test [9, 10], we begin with a set of test locations  $\mathcal{V} = \{\pmb{v}_1,\pmb{v}_2,\dots,\pmb{v}_{\ell}\} \subset \mathcal{X}$  to construct discriminative features. For every  $\pmb{v}_i\in \mathcal{V}$ , we introduce a Mahalanobis kernel as follows:
$$
\kappa_ {i} (\boldsymbol {x}, \boldsymbol {v} _ {i}) = \exp \left(- (\boldsymbol {x} - \boldsymbol {v} _ {i}) ^ {\top} M _ {i} (\boldsymbol {x} - \boldsymbol {v} _ {i}) / 2 \gamma_ {i} ^ {2}\right) \text {f o r} \gamma_ {i} > 0 \text {a n d p o s i t i v e d e f i n i t e m a t r i x} M _ {i}. \tag {1}
$$
Here, we propose multiple Mahalanobis kernels for two-sample test, which is motivated from multiple kernel learning [31, 32] and Mahalanobis distance [33-35]. The advantage of multiple Mahalanobis kernels is to exploit intrinsic structures and correlations from different directions and regions, and adjust geometrical distribution of data so as to enlarge the distance between different samples [36, 37].
This is different from previous Gaussian kernel  $\kappa_{i}(\pmb{x},\pmb{v}_{i}) = \exp (-\| \pmb{x} - \pmb{v}_{i}\|^{2} / 2\gamma^{2})$  [10, 38], which deals with every direction isotropically without difference. Figure 1 presents an illustration of different contours of Mahalanobis and Gaussian kernels for two-sample test. As we can see, Mahalanobis kernels are more flexible to exploit different directional information than Gaussian kernels. Our work is also different from previous deep kernel approaches [4, 39], which train single one deep neural network combined with Gaussian kernel for variations in distribution smoothness and shape.
We then embed each element in  $X = \{\pmb{x}_i\}_{i=1}^m$  and  $Y = \{\pmb{y}_j\}_{j=1}^n$  into an  $\ell$ -dimensional space as
$$
\hat {\boldsymbol {x}} _ {i} = \left(\kappa_ {1} \left(\boldsymbol {x} _ {i}, \boldsymbol {v} _ {1}\right), \dots , \kappa_ {\ell} \left(\boldsymbol {x} _ {i}, \boldsymbol {v} _ {\ell}\right)\right) ^ {\top} \text {a n d} \hat {\boldsymbol {y}} _ {j} = \left(\kappa_ {1} \left(\boldsymbol {y} _ {j}, \boldsymbol {v} _ {1}\right), \dots , \kappa_ {\ell} \left(\boldsymbol {y} _ {j}, \boldsymbol {v} _ {\ell}\right)\right) ^ {\top}, \text {r e s p e c t i v e l y}. \tag {2}
$$
Denote by  $\hat{X} = \{\hat{\pmb{x}}_i\}_{i = 1}^m$  and  $\hat{Y} = \{\hat{\pmb{y}}_j\}_{j = 1}^n$ . We define the pooled covariance matrix as
$$
\Sigma_ {\hat {X}, \hat {Y}} = \sum_ {i = 1} ^ {m} \frac {(\hat {\boldsymbol {x}} _ {i} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {i} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{m + n - 2} + \sum_ {i = 1} ^ {n} \frac {(\hat {\boldsymbol {y}} _ {i} - \boldsymbol {c} _ {\hat {Y}}) (\hat {\boldsymbol {y}} _ {i} - \boldsymbol {c} _ {\hat {Y}}) ^ {\top}}{m + n - 2} + \epsilon \mathbf {I} _ {d}, \tag {3}
$$
where  $c_{\hat{X}} = \sum_{i=1}^{m} \hat{\pmb{x}}_i / m$  and  $c_{\hat{Y}} = \sum_{j=1}^{n} \hat{\pmb{y}}_i / n$ , and  $\epsilon \mathbf{I}_d$  is introduced to guarantee the positive definiteness for small constant  $\epsilon > 0$ . We consider the Hotelling  $T^2$  statistic, as in [40-42],
$$
\mathcal {T} (\hat {X}, \hat {Y}) = m n \left(\boldsymbol {c} _ {\hat {X}} - \boldsymbol {c} _ {\hat {Y}}\right) ^ {\top} \Sigma_ {\hat {X}, \hat {Y}} ^ {- 1} \left(\boldsymbol {c} _ {\hat {X}} - \boldsymbol {c} _ {\hat {Y}}\right) / (m + n). \tag {4}
$$
Test power is the probability of correctly identifying two different samples. Maximizing  $\mathcal{T}(\hat{X},\hat{Y})$  is essentially equivalent to maximizing a lower bound of test power [10, 11], and we learn test locations and Mahalanobis kernels as follows:
$$
\{\mathcal {V}, M _ {1}, \dots , M _ {\ell}, \gamma_ {1}, \dots , \gamma_ {\ell} \} \in \arg \max  \left\{\mathcal {T} (\hat {X}, \hat {Y}) \right\}. \tag {5}
$$
We take gradient method [43] to solve the above optimization, as done by Jitkrittum et al. [10], and the details are presented in Appendix B.
We decompose  $\Sigma_{\hat{X},\hat{Y}} = LL$  via the Schur method [44] to remove feature correlations, and it follows
$$
\mathcal {T} (\hat {X}, \hat {Y}) = \frac {m n}{m + n} (\pmb {c} _ {\hat {X}} - \pmb {c} _ {\hat {Y}}) ^ {\top} \Sigma_ {\hat {X}, \hat {Y}} ^ {- 1} (\pmb {c} _ {\hat {X}} - \pmb {c} _ {\hat {Y}}) = \frac {m n}{m + n} \left\| \pmb {L} ^ {- 1} \pmb {c} _ {\hat {X}} - \pmb {L} ^ {- 1} \pmb {c} _ {\hat {Y}} \right\| _ {2} ^ {2}.
$$
Hence,  $\mathcal{T}(\hat{X},\hat{Y})$  essentially measures the difference between two samples via the  $L_{2}$ -norm of vector  $\pmb{L}^{-1}\pmb{c}_{\hat{X}} - \pmb{L}^{-1}\pmb{c}_{\hat{Y}}$ . We further exploit their inference direction, defined by
$$
\boldsymbol {F} = \operatorname {s g n} \left(\boldsymbol {L} ^ {- 1} \boldsymbol {c} _ {\hat {X}} - \boldsymbol {L} ^ {- 1} \boldsymbol {c} _ {\hat {Y}}\right) \in \{- 1, 0, + 1 \} ^ {\ell}. \tag {6}
$$
Bi-directional hypothesis for testing
Let  $\hat{\mathbb{P}}$  and  $\hat{\mathbb{Q}}$  be the corresponding embedding distributions from the original  $\mathbb{P}$  and  $\mathbb{Q}$ , respectively. Denote by  $\pmb{\mu}_{\hat{\mathbb{P}}} = E_{\hat{x}^{\prime} \sim \hat{\mathbb{P}}}[\hat{x}^{\prime}]$  and  $\pmb{\mu}_{\hat{\mathbb{Q}}} = E_{\hat{y}^{\prime} \sim \hat{\mathbb{Q}}}[\hat{y}^{\prime}]$ . We consider the following null hypothesis
$$
H _ {0}: \boldsymbol {\mu} _ {\mathbb {P}} = \boldsymbol {\mu} _ {\hat {\mathbb {Q}}}.
$$
The null hypothesis  $H_0$  can be used to test whether  $\mathbb{P} = \mathbb{Q}$  by the following lemma:
Lemma 1. We have  $\pmb{\mu}_{\hat{\mathbb{P}}} = \pmb{\mu}_{\hat{\mathbb{Q}}}$  iff  $\mathbb{P} = \mathbb{Q}$ , for bounded Mahalanobis kernels  $\{\kappa_j\}_{j=1}^{\ell}$  and for test locations  $\{\pmb{v}_j\}_{j=1}^\ell$  drawn i.i.d. from a absolutely-continuous distribution w.r.t. Lebesgue measure.
Let  $\hat{X}^{\prime} = \{\hat{x}_i^{\prime}\}_{i = 1}^{m^{\prime}}$  and  $\hat{Y}^{\prime} = \{\hat{y}_j^{\prime}\}_{j = 1}^{n^{\prime}}$  denote two embedding testing samples. We make similar Schur decomposition  $\Sigma_{\hat{X}^{\prime},\hat{Y}^{\prime}} = L^{\prime}L^{\prime}$ , and calculate testing statistic  $\mathcal{T}(\hat{X}^{\prime},\hat{Y}^{\prime})$  according to Eqn. (4). Based on Lemma 1, we can present the asymptotic distribution of statistic  $\mathcal{T}(\hat{X}^{\prime},\hat{Y}^{\prime})$  as follows:
Theorem 2. The testing statistic  $\mathcal{T}(\hat{X}',\hat{Y}')$  is almost surely asymptotically distributed as  $\chi_{\ell}^{2}$  if  $\mathbb{P} = \mathbb{Q}$ ; otherwise,  $\chi_{\ell}^{2}(\mathcal{T}(\hat{X}',\hat{Y}))\to 0$  as  $m'n' / (m' + n')\to \infty$ .
From Theorem 2, we propose the bi-directional hypothesis, by considering inference direction,
$$
h \left(\hat {X} ^ {\prime}, \hat {Y} ^ {\prime}\right) = \left\{ \begin{array}{l l} \mathbb {I} \left[ \chi_ {\ell} ^ {2} \left(\mathcal {T} \left(\hat {X} ^ {\prime}, \hat {Y} ^ {\prime}\right)\right) \leq \beta \alpha \right] & \text {f o r} \quad \boldsymbol {F} ^ {\top} \boldsymbol {L} ^ {\prime - 1} \left(\boldsymbol {c} _ {\hat {X} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} ^ {\prime}}\right) \geq 0 \\ \mathbb {I} \left[ \chi_ {\ell} ^ {2} \left(\mathcal {T} \left(\hat {X} ^ {\prime}, \hat {Y} ^ {\prime}\right)\right) \leq (2 - \beta) \alpha \right] & \text {f o r} \quad \boldsymbol {F} ^ {\top} \boldsymbol {L} ^ {\prime - 1} \left(\boldsymbol {c} _ {\hat {X} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} ^ {\prime}}\right) <   0, \end{array} \right. \tag {7}
$$
Figure 2: An illustration of different rejection regions on two samples for our bi-directional hypothesis and previous non/one-directional hypothesis. Our rejection region can be adaptive according to different datasets.
where  $\alpha \in (0,1)$  is the significance level of hypothesis test, and  $\beta \in [1,2]$  is an adaptive parameter.
Our bi-directional hypothesis is essentially about designing a rejection region of null hypothesis [45]. How to design an efficient rejection region is an interesting problem from the early work [46], and some techniques has been developed for selecting rejection regions [47-49]. We consider the most discriminative directions  $\pmb{F}$  and  $-\pmb{F}$  in our bi-directional hypothesis, which could improve the sensitivity for two-sample test by selecting appropriate parameters according to different datasets.
Our bi-directional hypothesis can be viewed as a generalization of previous hypotheses, that is,
- By setting  $\beta = 1$ , our test has been the non-directional hypothesis [9, 11, 50] regardless of direction information, which is also referred to as two-sided/tailed hypothesis [49, 51, 52];
- By setting  $\beta = 2$  and  $\pmb{F} = (1,1,\dots ,1)^{\top}$ , our test has become one-directional hypothesis [53-55], which is also referred to as one-sided/tailed hypothesis [52, 56].
Notice that previous non/one-directional hypotheses fix the structures of rejection region for a given significance level  $\alpha$ , whereas our bi-directional hypothesis could adjust rejection region according to inference direction  $F$  w.r.t. different datasets, which can be illustrated in Figure 2. Here, we consider an illustrative dataset, and our bi-directional hypothesis gives the rejection region adaptive to dataset, which could yield higher test power in two-sample test, as shown in Figure 5 (in Section 4).
We can also present some distribution and probability information for  $\pmb{F}^{\top}\pmb{L}^{\prime -1}(\pmb{c}_{\hat{X}^{\prime}} - \pmb{c}_{\hat{Y}^{\prime}}) \geq 0$  in Eqn. (7). Denote by  $L_{\hat{\mathbb{P}},\hat{\mathbb{Q}}} = E_{\hat{X}^{\prime}\sim \hat{\mathbb{P}}^{m^{\prime}},\hat{Y}^{\prime}\sim \hat{\mathbb{Q}}^{n^{\prime}}}[\pmb{L}^{\prime}]$  and  $\xi = \operatorname*{Pr}[F^{\top}L^{\prime -1}(\pmb{c}_{\hat{X}^{\prime}} - \pmb{c}_{\hat{Y}^{\prime}}) \geq 0]$ . We have
Lemma 3. For inference direction  $\mathbf{F}$  in Eqn. (6) and for embedding samples  $\hat{X}^{\prime}$  and  $\hat{Y}^{\prime}$ , we have
$$
\boldsymbol {L} ^ {\prime - 1} (\boldsymbol {c} _ {\hat {X} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} ^ {\prime}}) \sim \mathcal {N} \big (\boldsymbol {L} _ {\hat {\mathbb {P}}, \hat {\mathbb {Q}}} ^ {- 1} (\boldsymbol {\mu} _ {\hat {\mathbb {P}}} - \boldsymbol {\mu} _ {\hat {\mathbb {Q}}}), \omega^ {- 1} \boldsymbol {I} _ {\ell} \big) a n d s g n (\boldsymbol {F} ^ {\top} \boldsymbol {L} ^ {\prime - 1} (\boldsymbol {c} _ {\hat {X} ^ {\prime}} - \boldsymbol {c} _ {\hat {Y} ^ {\prime}})) \sim \mathcal {T P} (\xi)
$$
with  $\omega = m'n' / (m' + n')$  and  $\xi = 1 - \Phi(-\sqrt{\omega/\ell} \pmb{F}^\top \pmb{L}_{\hat{\mathbb{P}},\hat{\mathbb{Q}}}^{-1}(\pmb{\mu}_{\hat{\mathbb{P}}} - \pmb{\mu}_{\hat{\mathbb{Q}}}))$ . Here,  $\Phi(\cdot)$  is the distribution function of standard Gaussian, and  $\mathcal{TP}(\xi)$  denotes a distribution over  $\{-1, +1\}$  with probability  $\xi$  on the selection of  $+1$ .
The selection of parameter  $\beta$  is highly positive-relevant to the probability  $\xi$ . This is because a larger  $\xi$  implies larger difference between two samples in the inference direction  $F$ , and we should select a larger  $\beta$  to enlarge the rejection region and improve the sensitivity of dataset. Figure 6 (in Section 4) shows such positive relevance between the optimal parameter  $\beta$  and probability  $\xi$  empirically.
We finally present theoretical analysis on test power and type-I error of our bi-directional hypothesis. Let  $f(x \colon n_1, n_2, \lambda)$  be the density function of noncentral  $F$ -distribution with  $n_1$  and  $n_2$  degrees of freedom and non-centrality parameter  $\lambda$ , and denote by  $F_{n_1, n_2, \alpha}$  the  $\alpha$ -quantile of central  $F$ -distribution with  $n_1$  and  $n_2$  degrees of freedom for  $\alpha \in (0, 1)$ . We define the following probability, from the work of [57],
$$
q \left(n _ {1}, n _ {2}, \lambda , \alpha\right) = \int_ {F _ {n _ {1}, n _ {2}, \alpha}} ^ {\infty} f (x \mid n _ {1}, n _ {2}, \lambda) d x. \tag {8}
$$
Theorem 4. For our bi-directional hypothesis  $h(\hat{X}', \hat{Y}')$ , the test power can be lower bounded by
$$
q (\ell , \nu - \ell , \lambda , \beta \alpha) \cdot \xi + \Phi \left(- \left(\chi_ {\ell , (2 - \beta) \alpha} ^ {2}\right) ^ {1 / 2} - (\omega / \ell) ^ {1 / 2} \boldsymbol {F} ^ {\top} \boldsymbol {L} _ {\hat {\mathbb {P}}, \hat {\mathbb {Q}}} ^ {- 1} \left(\boldsymbol {\mu} _ {\hat {\mathbb {P}}} - \boldsymbol {\mu} _ {\hat {\mathbb {Q}}}\right)\right)
$$
Algorithm 1 Construction of partition tree
Input: Two embedding samples  $\hat{X}$  and  $\hat{Y}$   
Output: Partition tree  $T$   
Initialize: Tree  $T$  with only a root  $[0,1]^{\ell}$   
1: for  $t = 1,\dots,s - 1$  do  
2: Randomly select one of the leaf node  $\mathcal{B}$  of the largest  $|\hat{X} \cap \mathcal{B}|$   
3: Initialize the best splitting feature  $j^{*} = 0$   
4: for  $j = 1,\dots,\ell$  do  
5: Calculate the median value in  $j$ -th dimension  $\tau_{j} = \text{median}\{\hat{x}_{i,j} : \hat{x}_{i} \in \hat{X} \cap \mathcal{B}\}$   
6: Split two samples  $\hat{X}_{\mathcal{B}}$  and  $\hat{Y}_{\mathcal{B}}$  into  $\hat{X}_{\mathcal{B}_l^j}, \hat{Y}_{\mathcal{B}_l^j}$  and  $\hat{X}_{\mathcal{B}_r^j}, \hat{Y}_{\mathcal{B}_r^j}$ , according to  $\tau_{j}$   
7: Calculate  $\mathcal{T}(\hat{X}_{\mathcal{B}_l^j}, \hat{Y}_{\mathcal{B}_l^j})$  and  $\mathcal{T}(\hat{X}_{\mathcal{B}_r^j}, \hat{Y}_{\mathcal{B}_r^j})$  based on Eqn. (4)  
8: Update  $j^{*} = j$  if  $\mathcal{T}(\hat{X}_{\mathcal{B}_l^j}, \hat{Y}_{\mathcal{B}_l^j})\mathcal{T}(\hat{X}_{\mathcal{B}_r^j}, \hat{Y}_{\mathcal{B}_r^j}) > \mathcal{T}(\hat{X}_{\mathcal{B}_l^{j*}}, \hat{Y}_{\mathcal{B}_l^{j*}})\mathcal{T}(\hat{X}_{\mathcal{B}_r^{j*}}, \hat{Y}_{\mathcal{B}_r^{j*}})$   
9: end for  
10: Update node  $\mathcal{B}$  with two children  $\mathcal{B}_l^{j*}$  and  $\mathcal{B}_r^{j*}$  w.r.t the splitting feature  $j^{*}$  and position  $\tau_{j*}$   
11: end for
if  $F^{\top}L_{\hat{\mathbb{P}},\hat{\mathbb{Q}}}^{-1}\pmb{\mu}_{\hat{\mathbb{P}}}>F^{\top}L_{\hat{\mathbb{P}},\hat{\mathbb{Q}}}^{-1}\pmb{\mu}_{\hat{\mathbb{Q}}}$ ; and the test power can also be lower bounded by
$$
q (\ell , \nu - \ell , \lambda , (2 - \beta) \alpha) (1 - \xi) + 1 - \Phi \left(\left(\chi_ {\ell , \beta \alpha} ^ {2}\right) ^ {1 / 2} - (\omega / \ell) ^ {1 / 2} \boldsymbol {F} ^ {\top} \boldsymbol {L} _ {\hat {\mathbb {P}}, \hat {\mathbb {Q}}} ^ {- 1} \left(\boldsymbol {\mu} _ {\hat {\mathbb {P}}} - \boldsymbol {\mu} _ {\hat {\mathbb {Q}}}\right)\right)
$$
if  $\pmb{F}^{\top}\pmb{L}_{\hat{\mathbb{P}},\hat{\mathbb{Q}}}^{-1}\pmb{\mu}_{\hat{\mathbb{P}}} < \pmb{F}^{\top}\pmb{L}_{\hat{\mathbb{P}},\hat{\mathbb{Q}}}^{-1}\pmb{\mu}_{\hat{\mathbb{Q}}}$ ; and the type-I error rate is equal to  $\alpha$  if  $\pmb{\mu}_{\hat{\mathbb{P}}} = \pmb{\mu}_{\hat{\mathbb{Q}}}$ . Here,  $\omega = m'n' / (m' + n')$ ,  $\nu = m' + n' - 1$  and  $\lambda = \omega \| L_{\hat{\mathbb{P}},\hat{\mathbb{Q}}}^{-1}(\pmb{\mu}_{\hat{\mathbb{P}}} - \pmb{\mu}_{\hat{\mathbb{Q}}}) \|_2^2$ .
This theorem presents lower bounds on test power, and the performance of the statistical test is maintained under the general condition. Notice that the type-I error in our hypothesis test is controlled only by the significant level  $\alpha$ , regardless of different  $\xi$  and  $\beta$ .
We call our test as  $ME_{MaBiD}$  test because of multiple Mahalanobis kernels in training and our bidirectional hypothesis in testing.
3 Explore Local Significant Differences for Two-sample Test
On the exploration of local significant difference, most previous studies [24-28] partition the instance space into several regions, and then exploit the difference on each region from two samples. Motivated from polya tree method [26, 58], we first partition the embedding instance space with a new splitting criterion, which is relevant to test power and data correlations. We then exploit local regions (i.e., leaves nodes of partition tree) with significant difference.
Partition of the embedding instance space
Our partition tree is constructed iteratively as follows: We initiate the tree root with embedding space  $(0,1]^{\ell}$ . In each iteration, each node is associated with a rectangle region, and all leaves constitute a partition of embedding instance space. The following procedure is repeated  $s - 1$  iterations ( $s \geq 2$ ):
- Randomly select a leaf node, denote by  $\mathcal{B}$ , uniformly over leaf nodes of the largest  $|\hat{X} \cap \mathcal{B}|$ .
- Let  $\tau_{j} =$  median  $\{\hat{x}_{i,j}\colon \hat{x}_i\in \hat{X}\cap \mathcal{B}\}$  for  $j\in [\ell ]$  . We select the best splitting feature
$$
j ^ {*} \in \arg \max _ {j \in [ \ell ]} \left\{\mathcal {T} (\hat {X} _ {\mathcal {B} _ {l} ^ {j}}, \hat {Y} _ {\mathcal {B} _ {l} ^ {j}}) \times \mathcal {T} (\hat {X} _ {\mathcal {B} _ {r} ^ {j}}, \hat {Y} _ {\mathcal {B} _ {r} ^ {j}}) \right\},
$$
with  $\hat{X}_{\mathcal{B}_l^j} = \mathcal{B}_l^j \cap \hat{X}$ ,  $\hat{X}_{\mathcal{B}_r^j} = \mathcal{B}_r^j \cap \hat{X}$ ,  $\hat{Y}_{\mathcal{B}_l^j} = \mathcal{B}_l^j \cap \hat{Y}$  and  $\hat{Y}_{\mathcal{B}_r^j} = \mathcal{B}_r^j \cap \hat{Y}$ . Here,  $\mathcal{B}_l^j$  and  $\mathcal{B}_r^j$  are left and right children of  $\mathcal{B}$  w.r.t. the  $j$ -th splitting feature and splitting position  $\tau_j$ , respectively, and  $\mathcal{T}(\cdot, \cdot)$  is defined by Eqn. (4).
- Select the splitting position  $\tau_{j^{*}} =$  median  $\{\hat{x}_{i,j^{*}}:\hat{x}_i\in \hat{X}\cap \mathcal{B}\}$