Title: On the Exploration of Local Significant Differences For Two-Sample Test
Venue: NeurIPS 2023
Accepted: yes
Keywords: two-sample test

local significant difference

directional information
Abstract: Recent years have witnessed increasing attentions on two-sample test with diverse real applications, while this work takes one more step on the exploration of local significant differences for two-sample test. We propose the ME$_\text{MaBiD}$, an effective test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. On the exploration of local significant differences, we first partition the embedding space into several rectangle regions via a new splitting criterion, which is relevant to test power and data correlation. We then explore local significant differences based on our bi-directional masked $p$-value together with the ME$_\text{MaBiD}$ test. Theoretically, we present the asymptotic distribution and lower bounds of test power for our ME$_\text{MaBiD}$ test, and control the familywise error rate on the exploration of local significant differences. We finally conduct extensive experiments  to validate the effectiveness of our proposed methods on two-sample test and the exploration of local significant differences.

On the Exploration of Local Significant Differences For Two-Sample Test

Abstract

Recent years have witnessed increasing attentions on two-sample test with diverse real applications, while this work takes one more step on the exploration of local significant differences for two-sample test. We propose the ME MaBiD , an effective test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. On the exploration of local significant differences, we first partition the embedding space into several rectangle regions via a new splitting criterion, which is relevant to test power and data correlation. We then explore local significant differences based on our bi-directional masked p -value together with the ME MaBiD test. Theoretically, we present the asymptotic distribution and lower bounds of test power for our MEMaBiD test, and control the familywise error rate on the exploration of local significant differences. We finally conduct extensive experiments to validate the effectiveness of our proposed methods on two-sample test and the exploration of local significant differences.

1 Introduction

Two-sample test has attracted much attention with diverse applications such as cancer detection [1], distribution-shift detection [2], generative modeling [3, 4], etc. The basic problem is to assess whether two i.i.d. samples are drawn from the same distribution. Various kernel-based methods have been developed for two-sample test such as Maximum Mean Discrepancy (MMD) [5-8] and Mean Embedding (ME) [9-11]. Another relevant approach is to construct a binary classifier and assess two samples according to classification performance [12-20]. For an overview of two-sample test, we refer to a survey [21, and references therein].

In many real applications, however, it is necessary to take one more step to explore and understand local significant differences, rather than only two-sample test. For example, a scientific problem in galaxy morphology is to identify some local regions of significant differences between two kinds of galaxies, which is important to discover galaxy formation and evolution history [22]. On the analysis of mass cytometry data in cell biology, researchers are always interested in finding local regions of significantly different abundance between disease and healthy samples [23].

Several attempts have been made to explore local significant differences in the past years. A feasible solution is to partition space into several regions, and identify significantly different regions according to their cardinalities of samples [24-27]. Another relevant work is to identify local significant differences by estimating kernel densities [28, 29] and conditional probabilities [30]. Generally, it is not easy to deal with complex data by simply counting cardinalities of samples, regardless of data intrinsic correlations, and it is also difficult to make accurate estimation of kernel densities and conditional probabilities without sufficient data, especially for many regions with finite samples.

Contour of Mahalanobis kernel

test locations

elements in X

elements in Y

GLYPH(cmap:d835)1

GLYPH(cmap:d835)2

GLYPH(cmap:d835)2

Contour of Gaussian kernel

GLYPH(cmap:d835)1

This work presents a new two-sample test from local and directional information, and further explore local significant differences. The main contributions can be summarized as follows:

We propose the effective ME MaBiD test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. Intuitively, Mahalanobis kernels are more flexible to exploit local differences from neighborhoods and feature maps, and the bi-directional hypothesis is beneficial to improve the sensitivity of two-sample test with proper parameter adaptation.

We partition the embedding space into several rectangle regions based on a new splitting criterion, which is relevant to test power and data correlation. We introduce the bi-directional masked p -value for each rectangle region, and finally explore local regions with significant difference based on our bi-directional masked p -value together with the ME MaBiD test.

We present theoretical guarantees for our ME MaBiD test via the asymptotic distribution, as well as the lower bounds on the test power for our test. We also present the upper bounds on familywise error rate for our exploration of local significant differences.

Weconduct extensive experiments to validate the effectiveness and efficiency of our methods. Specifically, our methods achieve better performance on most datasets for two-sample test and exploring local significant differences, along with comparable or smaller running time.

The rest of this work is organized as follows: Section 2 presents our ME MaBiD test. Section 3 explores local significant differences. Section 4 conducts extensive experiments, and Section 5 concludes with future work. All technical proofs are given in Appendix A.

2 Our MEMaBiD Test for Two-Sample Testing

Let P and Q denote two (unknown) Borel probability measures over an instance space X ⊆ R d , and X = { x i } i m =1 and Y = { y j } n j =1 are two i.i.d. samples from P and Q , respectively. The goal of two-sample test is to assess whether X and Y are drawn from the same distribution; in other words, we aim to assess whether P = Q from two samples X and Y .

glyph[negationslash]

We introduce some necessary notations used in this work. Write [ τ ] = { 1 , 2 , · · · , τ } for integer τ ≥ 2 , and | A | denotes the cardinality of set A . Let I d be the identity matrix of size d × d . For a vector a = [ a 1 , a 2 , · · · , a d ] , denote by sgn ( a ) = [ sgn ( a 1 ) , sgn ( a 2 ) , · · · , sgn ( a d )] with sgn ( a i ) = a i / | a i | for a i = 0 ; otherwise, sgn ( a i ) = 0 . Let χ 2 glyph[lscript] be the χ 2 distribution with glyph[lscript] degree of freedom, as well as the p -value function χ 2 glyph[lscript] ( · ) . Denote by χ 2 glyph[lscript],α the α -quantile of distribution χ 2 glyph[lscript] for α ∈ (0 , 1) .

Learning multiple Mahalanobis kernels via maximizing test power in training

Following ME test [9, 10], we begin with a set of test locations V = { v 1 , v 2 , . . . , v glyph[lscript] } ⊂ X to construct discriminative features. For every v i ∈ V , we introduce a Mahalanobis kernel as follows:

Here, we propose multiple Mahalanobis kernels for two-sample test, which is motivated from multiple kernel learning [31, 32] and Mahalanobis distance [33-35]. The advantage of multiple Mahalanobis kernels is to exploit intrinsic structures and correlations from different directions and regions, and adjust geometrical distribution of data so as to enlarge the distance between different samples [36, 37].

This is different from previous Gaussian kernel κ i ( x , v i ) = exp( -‖ x -v i ‖ 2 / 2 γ 2 ) [10, 38], which deals with every direction isotropically without difference. Figure 1 presents an illustration of different contours of Mahalnobis and Gaussian kernels for two-sample test. As we can see, Mahalanobis kernels are more flexible to exploit different directional information than Gaussian kernels. Our work is also different from previous deep kernel approaches [4, 39], which train single one deep neural network combined with Gaussian kernel for variations in distribution smoothness and shape.

We then embed each element in X = { x i } i m =1 and Y = { y j } n j =1 into an glyph[lscript] -dimensional space as

Denote by ˆ X = { ˆ x i } i m =1 and ˆ Y = { ˆ y j } n j =1 . We define the pooled covariance matrix as where c ˆ X = ∑ m i =1 ˆ x i /m and c ˆ Y = ∑ n j =1 ˆ y i /n , and glyph[epsilon1] I d is introduced to guarantee the positive definiteness for small constant glyph[epsilon1] > 0 . We consider the Hotelling T 2 statistic , as in [40-42],

Test power is the probability of correctly identifying two different samples. Maximizing T ( ˆ X, ˆ Y ) is essentially equivalent to maximizing a lower bound of test power [10, 11], and we learn test locations and Mahalanobis kernels as follows:

We take gradient method [43] to solve the above optimization, as done by Jitkrittum et al. [10], and the details are presented in Appendix B.

We decompose Σ ˆ X, ˆ Y = LL via the Schur method [44] to remove feature correlations, and it follows

Hence, T ( ˆ X, ˆ Y ) essentially measures the difference between two samples via the L 2 -norm of vector L -1 c ˆ X -L -1 c ˆ Y . We further exploit their inference direction , defined by

Bi-directional hypothesis for testing

Let ˆ P and ˆ Q be the corresponding embedding distributions from the original P and Q , respectively. Denote by µ ˆ P = E ˆ x ′ ∼ ˆ P [ ˆ x ′ ] and µ ˆ Q = E ˆ y ′ ∼ ˆ Q [ ˆ y ′ ] . We consider the following null hypothesis

The null hypothesis H 0 can be used to test whether P = Q by the following lemma:

Lemma 1. We have µ ˆ P = µ ˆ Q iff P = Q , for bounded Mahalanobis kernels { κ j } glyph[lscript] j =1 and for test locations { v j } glyph[lscript] j =1 drawn i.i.d. from a absolutely-continuous distribution w.r.t. Lebesgue measure.

Let ˆ X ′ = { ˆ x ′ i } m ′ i =1 and ˆ Y ′ = { ˆ y ′ j } n ′ j =1 denote two embedding testing samples. We make similar Schur decomposition Σ ˆ X ′ , ˆ Y ′ = L ′ L ′ , and calculate testing statistic T ( ˆ X ′ , ˆ Y ′ ) according to Eqn. (4). Based on Lemma 1, we can present the asymptotic distribution of statistic T ( ˆ X ′ , ˆ Y ′ ) as follows:

Theorem 2. The testing statistic T ( ˆ X ′ , ˆ Y ′ ) is almost surely asymptotically distributed as χ 2 glyph[lscript] if P = Q ; otherwise, χ 2 glyph[lscript] ( T ( ˆ X ′ , ˆ Y ′ )) → 0 as m ′ n ′ / ( m ′ + n ′ ) →∞ .

From Theorem 2, we propose the bi-directional hypothesis , by considering inference direction,

(0,0)

GLYPH(cmap:d835)1

GLYPH(cmap:d835)2

Fixed rejection region

GLYPH(cmap:d835)2

Fixed rejection region

(0,0)

fixed

F

1

= (1,1) T

GLYPH(cmap:d835)1

Our bi-directional hypothesis

GLYPH(cmap:d835)1

The non-directional hypothesis

The one-directional hypothesis

F

= (-1,1) T

(0,0)

GLYPH(cmap:d835)2

GLYPH(cmap:d835)1

Two samples X and Y

Adaptive rejection region

according to different datasets

elements in X

elements in Y

GLYPH(cmap:d835)2

where α ∈ (0 , 1) is the significance level of hypothesis test, and β ∈ [1 , 2] is an adaptive parameter.

Our bi-directional hypothesis is essentially about designing a rejection region of null hypothesis [45]. How to design an efficient rejection region is an interesting problem from the early work [46], and some techniques has been developed for selecting rejection regions [47-49]. We consider the most discriminative directions F and -F in our bi-directional hypothesis, which could improve the sensitivity for two-sample test by selecting appropriate parameters according to different datasets.

Our bi-directional hypothesis can be viewed as a generalization of previous hypotheses, that is,

By setting β = 1 , our test has been the non-directional hypothesis [9, 11, 50] regardless of direction information, which is also referred to as two-sided/tailed hypothesis [49, 51, 52];

By setting β = 2 and F = (1 , 1 , · · · , 1) glyph[latticetop] , our test has become one-directional hypothesis [53-55], which is also referred to as one-sided/tailed hypothesis [52, 56].

Notice that previous non/one-directional hypotheses fix the structures of rejection region for a given significance level α , whereas our bi-directional hypothesis could adjust rejection region according to inference direction F w.r.t. different datasets, which can be illustrated in Figure 2. Here, we consider an illustrative dataset, and our bi-directional hypothesis gives the rejection region adaptive to dataset, which could yield higher test power in two-sample test, as shown in Figure 5 (in Section 4).

We can also present some distribution and probability information for F glyph[latticetop] L ′-1 ( c ˆ X ′ -c ˆ Y ′ ) ≥ 0 in Eqn. (7). Denote by L ˆ P , ˆ Q = E ˆ X ′ ∼ ˆ P m ′ , ˆ Y ′ ∼ ˆ Q n ′ [ L ′ ] and ξ = Pr[ F glyph[latticetop] L ′-1 ( c ˆ X ′ -c ˆ Y ′ ) ≥ 0] . We have

Lemma 3. For inference direction F in Eqn. (6) and for embedding samples ˆ X ′ and ˆ Y ′ , we have

with ω = m ′ n ′ / ( m ′ + n ′ ) and ξ = 1 -Φ( -√ ω/glyph[lscript] F glyph[latticetop] L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q )) . Here, Φ( · ) is the distribution function of standard Gaussian, and T P ( ξ ) denotes a distribution over {-1 , +1 } with probability ξ on the selection of +1 .

The selection of parameter β is highly positive-relevant to the probability ξ . This is because a larger ξ implies larger difference between two samples in the inference direction F , and we should select a larger β to enlarge the rejection region and improve the sensitivity of dataset. Figure 6 (in Section 4) shows such positive relevance between the optimal parameter β and probability ξ empirically.

We finally present theoretical analysis on test power and type-I error of our bi-directional hypothesis. Let f ( x : n 1 , n 2 , λ ) be the density function of noncentral F -distribution with n 1 and n 2 degrees of freedom and non-centrality parameter λ , and denote by F n 1 ,n 2 ,α the α -quantile of central F -distribution with n 1 and n 2 degrees of freedom for α ∈ (0 , 1) . We define the following probability, from the work of [57],

Theorem 4. For our bi-directional hypothesis h ( ˆ X ′ , ˆ Y ′ ) , the test power can be lower bounded by

Algorithm 1 Construction of partition tree

Input

: Two embeddding samples ˆ X and ˆ Y

Output

: Partition tree T

Initialize : Tree

T with only a root [0 , 1] glyph[lscript]

1: for t = 1 , . . . , s -1 do

3: Initialize the best splitting feature j ∗ = 0

2: Randomly select one of the leaf node B of the largest | ˆ X ∩ B|

4: for j = 1 , . . . , glyph[lscript] do

5: Calculate the median value in j -th dimension τ j = median { ˆ x i,j : ˆ x i ∈ ˆ X ∩ B}

6: Split two samples ˆ X B and ˆ Y B into ˆ X B j l , ˆ Y B j l and ˆ X B j r , ˆ Y B j r , according to τ j

7: Calculate T ( ˆ X B j l , ˆ Y B j l ) and T ( ˆ X B j r , ˆ Y B j r ) based on Eqn. (4)

9: end for

8: Update j ∗ = j if T ( ˆ X B j l , ˆ Y B j l ) T ( ˆ X B j r , ˆ Y B j r ) > T ( ˆ X B j ∗ l , ˆ Y B j ∗ l ) T ( ˆ X B j ∗ r , ˆ Y B j ∗ r )

∗ ∗

10: Update node B with two children B j l and B j r w.r.t the splitting feature j ∗ and position τ j ∗ 11: end for

if F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ P > F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ Q ; and the test power can also be lower bounded by

This theorem presents lower bounds on test power, and the performance of the statistical test is maintained under the general condition. Notice that the type-I error in our hypothesis test is controlled only by the significant level α , regardless of different ξ and β .

We call our test as MEMaBiD test because of multiple Mahalanobis kernels in training and our bidirectional hypothesis in testing.

3 Explore Local Significant Differences for Two-sample Test

On the exploration of local significant difference, most previous studies [24-28] partition the instance space into several regions, and then exploit the difference on each region from two samples. Motivated from pólya tree method [26, 58], we first partition the embedding instance space with a new splitting criterion, which is relevant to test power and data correlations. We then exploit local regions (i.e., leaves nodes of partition tree) with significant difference.

Partition of the embedding instance space

Our partition tree is constructed iteratively as follows: We initiate the tree root with embedding space (0 , 1] glyph[lscript] . In each iteration, each node is associated with a rectangle region, and all leaves constitute a partition of embedding instance space. The following procedure is repeated s -1 iterations ( s ≥ 2 ):

Randomly select a leaf node, denote by B , uniformly over leaf nodes of the largest | ˆ X ∩ B| .

Let τ j = median { ˆ x i,j : ˆ x i ∈ ˆ X ∩ B} for j ∈ [ glyph[lscript] ] . We select the best splitting feature

with ˆ X B j l = B j l ∩ ˆ X , ˆ X B j r = B j r ∩ ˆ X , ˆ Y B j l = B j l ∩ ˆ Y and ˆ Y B j r = B j r ∩ ˆ Y . Here, B j l and B j r are left and right children of B w.r.t. the j -th splitting feature and splitting position τ j , respectively, and T ( · , · ) is defined by Eqn. (4).

Select the splitting position τ j ∗ = median { ˆ x i,j ∗ : ˆ x i ∈ ˆ X ∩ B} .

We finally get the partition tree with s leaf nodes, associated with s rectangle regions B 1 , B 2 , . . . , B s . Algorithm 1 presents the detailed description on tree construction and rectangle region splitting.

We take the statistic T ( · , · ) as a splitting criterion, relevant to test power and data correlations, and it is helpful to exploit local significant difference directly. We also adopt the median splitting position with equal probabilities on partitioned regions, i.e., balanced examples for each partition region, and this could yield better performance than regular grids, as shown empirically in [59, 60].

Our partition tree is different from previous p -value histogram based on Chi-square test [61, 62], where the difference is measured by cardinalities of elements in two samples over a local rectangle region. Our splitting criterion is also different from that of previous decision trees [63-66], which consider some information-theoretic criterions such as entropy, Gini index, information gain, etc. In comparisons, our statistic T ( · , · ) is more essential to reflect the test power for two-sample test.

For each rectangle region B i , let c ˆ X B i and c ˆ Y B i be the means of ˆ X B i = B i ∩ ˆ X and ˆ Y B i = B i ∩ ˆ Y , respectively. We make similar Schur decomposition L B i L B i = Σ ˆ X B i , ˆ Y B i for covariance matrix Σ ˆ X B i , ˆ Y B i , and introduce the local inference direction for each B i as follow:

For different rectangle regions, we could have different or even contrary inference directions, which is helpful to exploit local differences from distributional shapes of two samples.

Exploration of local significant differences

For testing embedding samples ˆ X ′ and ˆ Y ′ , we denote by ˆ X ′ B i = B i ∩ ˆ X ′ and ˆ Y ′ B i = B i ∩ ˆ Y ′ with their respective means c ˆ X ′ B i and c ˆ Y ′ B i for each rectangle region B i , and calculate testing statistic B = ( ˆ X ′ , ˆ Y ′ ) by Eqn. (4).

We propose the new bi-directional masked p -value for each rectangle region B i as follows:

where L ′-1 B i is from Schur decomposition Σ ˆ X ′ B i , ˆ Y ′ B i = L ′-1 B i L ′-1 B i , p ∗ ∈ (0 , 1) is a parameter on significance level, and β is an adaptive parameter. Here, we also consider two discriminative directions F B i and -F B i on each rectangle region B i , which is different from previous masked p -value [67] without directional information.

Our bi-directional masked p -value directly reflects the significant level of local difference when there is a significant difference in local B i , similarly to [67]. The smaller the bi-directional masked p -value, the more significant the local difference. On the other hand, the bi-directional masked p -value is a random number with uniform distribution over (0 , p ∗ ) when there is no significant difference in B i , since χ 2 glyph[lscript] ( T B i ) follows a uniform distribution in such case [68].

Based on such recognition, we resort rectangle regions as B 〈 1 〉 , B 〈 2 〉 , . . . , B 〈 s 〉 according to their bi-directional masked p -value, i.e.,

We then take our bi-directional hypothesis h ( ˆ X ′ B i , ˆ Y ′ B i ) with parameter β and α = p ∗ as in Eqn. (7), and finally get the local regions with significant differences as

where

Here, p ∗ is selected as in Eqn. (10), and α ∗ ∈ [ p ∗ , 1) is a parameter to control the probability of mis-identifying at least one rectangle region without significant difference, also called familywise error rate [69-72]. We present theoretical analysis for familywise error rate as follows:

Theorem 5. For our exploration, the familywise error rate is upper bounded by α ∗ , if 1) the p -values of local regions without differences are mutually independent; and 2) the p -values of local regions with differences are independent to those p-values of local regions without differences.

Our method is different from previous space partition methods of trees or clusters [24-27, 60], where the splitting criterion is taken as the cardinalities of samples in each region. Duong [28] partitioned and searched local regions from the estimated density function, and Kim et al. [30] identified local regions by clustering data samples from the estimated conditional probabilities. It is not easy to make accurate estimation for density and conditional probabilities without sufficient data, particularly for multiple small regions. Other relevant studies detected local differences implicitly based on interactive rank test [73] or a learned classifier [74].

4 Experiments

We conduct experiments on 16 datasets 1 as summarized in Table 1. Most dataset have been studied in previous two-sample test, and features have been scaled to [0 , 1] for all datasets. All experiments are performed with Python on nodes of a computational cluster with a single CPU (Intel Core i9-10900X 3.7GHz) and a single GPU (GeForce RTX 2080 Ti), running Ubuntu with 128GB main memory.

Experimental comparisons for two-sample test

We compare our MEMaBiD with the state-of-the-art approaches on two-sample test as follows:

ME: Mean Embeddings over multiple test locations and a single Gaussian kernel [9, 10];

MMDAgg: Maximum Mean Discrepancy with Aggregating of multiple Gaussian kernels [75];

MMD-D: Maximum Mean Discrepancy based on a Deep kernel [39];

C2ST-S: Train a binary classification network and test its accuracy on a hold-out set [13];

AutoMLTST: Train a binary classifier based on AutoML method with a statistic as C2ST-L [19].

C2ST-L: Train a binary classification network with a statistic about class probabilities [14, 18];

Following [39, 76], we train on a subset of each available data, and test on 100 random subsets from the remaining dataset, and the ratio is set as 4 : 1 for training and testing. We repeat such process 10 times for each dataset. More details are given in Appendix C. For our ME MaBiD , we set α = 0 . 05 and take 5-fold cross validation to select β ∈ [1 : 0 . 2 : 2] . We limit the cardinality of test locations within 20 for ME and MEMaBiD as in [9-11], and optimization parameters of Eqn. (5) is presented in Appendix C. We take parameter settings for other methods as in their respective inferences.

Table 2 summarizes the average of test powers and standard deviations. It is evident that our ME MaBiD takes better performance than ME and MMDAgg, because they both take Gaussian kernels with isotropic scale, and ignore the distributional differences from different directions. Our method is still better than MMD-D with a deep kernel, and a reason is that multiple Mahalanobis kernels are more flexible than a deep kernel to capture local difference from multiple neighborhoods and directions.

From Table 2, it is also observed that our MEMaBiD outperforms three classifier-based methods C2ST-S, C2ST-L and AutoML expect for datasets labor and poker, since those methods focus merely on the prediction information from outputs of classifiers, rather than local and directional information among data samples. For datasets labor and poker, AutoML generates new features automatically from the original mixture of continuous and symbolic features, and thus achieves better performance.

We further compare the average running time (in seconds) for different methods on two-sample test, as shown in Figure 3. As expected, ME takes the least running time since it considers only one Gaussian kernel, yet with the smallest average of test powers in Table 2. Our ME MaBiD method takes smaller and comparable running time in contrast to other methods since our method takes relatively smaller time on training Mahalanobis kernels without permutation test in the testing process.

Experiments on the exploration of local significant differences

We compare with the state-of-the-art approaches on exploring local significant difference as follows:

FDG: Partition space by probability binning and compare cardinalities of two samples [24];

MRS: Partition space by pólya tree and measure difference via Binomial distributions [26];

K-PRIM: Partition space by patient rule induction and estimate kernel density differences [28];

TEAM: Partition space by data variance and measure difference via Binomial distributions [27];

MMDT: Partition space into equal grids and test density difference via Welch's statistic [29].

BTLDD: Estimate conditional probabilities of two samples and cluster data via difference [77];

blob

dna

agnos

topo21

diamon

cifar

mnist

santan

codrna

sea50

adult

labor

poker

higgs

Datasets

10 0

10 1

10 2

10 3

Running time (seconds)

Our MEMaBiD

ME

MMD-D

MMDAgg

C2ST-S

C2ST-L

AutoMLTST

Ours (0.93)

MMDT (0.90)

FDG (0.85)

KPRIM (0.92)

MRS (0.91)

TEAM (0.87)

Difference between mnist and Fake-mnist

Mean of

mnist  (S

ample

X

)

Mean of Fake-mnist (Sample

Y

)

(a) partition tree with

s

= 8  rectangle regions (left)

Difference between mnist and Fake-mnist

Mean of

mnist  (S

ample

X

)

Mean of Fake-mnist (Sample

Y

)

(b) partition tree with

s

= 16 rectangle regions (right)

Ours (0.96)

MMDT (0.91)

FDG (0.91)

KPRIM (0.93)

MRS (0.94)

TEAM (0.91)

For fair comparisons with some methods of density estimations, we select eight large datasets: diamond, codrna, blob, sea50, adult, labor, poker and higgs, whose instance numbers are more than 50,000 and feature numbers are smaller than 20. We randomly select 10 , 000 instances during the exploration of local significant differences. For the BTLDD method, we set the percentage of selected samples as 6 . 25% for local regions in a cluster, while for other methods, we partition the space into 64 rectangle regions, and identify 4 rectangle regions with the most local differences.

We take density differences [78] between two samples in a local region as an evaluation measure for local significant differences, and follow the works of [79, 80] based on k -NN density estimator with k = 20 . We calculate the difference between two estimated density functions in an identified local region, and normalize the returned values into [0 , 1] .

Table 3 summarizes the average of density differences and standard deviations. As can be seen, our method takes better explorations on local significant difference than FDG, MRS and TEAM, since those methods simply take cardinalities of samples to measure significant difference. Our exploration is still better than K-PRIM, BTLDD and MMDT except for dataset sea50, because those methods estimate density or conditional probabilities from two samples, and it is not easy to make accurate estimation without sufficient data, especially for small regions with finite samples. MMDT could estimate density function well on dataset sea50, and achieves better exploration via Welch's statistic.

We further visualize the local significant differences on mnist (sample X ) and Fake-mnist (sample Y ) in Figure 4. Here, we take partition trees with s = 8 (left) and s = 16 (right) rectangle regions. As can be seen, our method achieves the largest local differences via new splitting criterion for partition tree, and our proposed MEMaBiD test and bi-directional masked p -values.

Parameter analysis

Wenow present some parameter influence for our MEMaBiD test and the exploration of local significant difference. We only present the results on four datasets due to pages limit, but the trends are similar on other datasets, and more results can be found in Appendix D.

50

100

150

200

250

300

350

400

Sample size

0.1

0.4

0.7

1.0

Test power

blob

bi-directional hypothesis

non-directional hypothesis

one-directional hypothesis

25

50

75

100

125

150

175

200

225

Sample size

0.1

0.4

0.7

1.0

Test power

codrna

bi-directional hypothesis

non-directional hypothesis

one-directional hypothesis

25

50

75

100

125

150

175

200

225

Sample size

0.1

0.4

0.7

1.0

Test power

sea50

bi-directional hypothesis

non-directional hypothesis

one-directional hypothesis

500

1000

1500

2000

2500

3000

3500

4000

Sample size

0.1

0.4

0.7

1.0

Test power

higgs

bi-directional hypothesis

non-directional hypothesis

one-directional hypothesis

0.5

0.6

0.7

0.8

0.9

1.0

1.0

1.2

1.4

1.6

1.8

2.0

The optimal

β

blob

Probability

ξ

= Pr[

F

B

/latticetop

L

′-

B

ˆ

1

(

c

X

′ -

c

Y

ˆ

′

)

0]

0.5

0.6

0.7

0.8

0.9

1.0

1.2

1.4

1.6

1.8

2.0

The optimal

β

codrna

Probability

ξ

= Pr[

F

B

/latticetop

L

′-

B

ˆ

1

(

c

X

′ -

c

Y

ˆ

1.0

′

)

0]

0.5

0.6

0.7

0.8

0.9

1.0

1.2

1.4

1.6

1.8

2.0

The optimal

β

sea50

Probability

ξ

= Pr[

F

B

/latticetop

L

′-

B

ˆ

1

(

c

X

′ -

c

Y

ˆ

1.0

′

)

0]

0.5

0.6

0.7

0.8

0.9

1.0

1.2

1.4

1.6

1.8

2.0

The optimal

β

higgs

Probability

ξ

= Pr[

F

B

/latticetop

L

′-

B

ˆ

1

(

c

X

′ -

c

Y

ˆ

1.0

′

)

0]

50

100

150

200

250

300

350

400

Sample size

0.00

0.02

0.04

0.06

Type-I error

blob

β

=1.0

β

=1.2

β

=1.4

β

=1.6

β

=1.8

β

=2.0

50

75

100

125

150

175

200

225

250

Sample size

0.00

0.02

0.04

0.06

Type-I error

codrna

β

=1.0

β

=1.2

β

=1.4

β

=1.6

β

=1.8

β

=2.0

50

75

100

125

150

175

200

225

250

Sample size

0.00

0.02

0.04

0.06

Type-I error

sea50

β

=1.0

β

=1.2

β

=1.4

β

=1.6

β

=1.8

β

=2.0

100

200

300

400

500

600

700

800

900

Sample size

0.00

0.02

0.04

0.06

Type-I error

higgs

β

=1.0

β

=1.2

β

=1.4

β

=1.6

β

=1.8

β

=2.0

Figure 7:

The type-I error is limited about

α

= 0

.

05

w.r.t different

β

for our MEMaBiD.

50

100

150

200

250

300

350

400

Sample size

0.00

0.02

0.04

0.06

0.08

Familywise error rate

blob

s

=2 1

s

=2 2

s

=2 3

s

=2 4

s

=2 5

s

=2 6

50

75

100

125

150

175

200

225

250

Sample size

0.00

0.02

0.04

0.06

0.08

Familywise error rate

codrna

s

=2 1

s

=2 2

s

=2 3

s

=2 4

s

=2 5

s

=2 6

50

75

100

125

150

175

200

225

250

Sample size

0.00

0.02

0.04

0.06

0.08

Familywise error rate

sea50

s

=2 1

s

=2 2

s

=2 3

s

=2 4

s

=2 5

s

=2 6

100

200

300

400

500

600

700

800

900

Sample size

0.00

0.02

0.04

0.06

0.08

Familywise error rate

higgs

s

=2 1

s

=2 2

s

=2 3

s

=2 4

s

=2 5

s

=2 6

Figure 5 illustrates the test power versus sample size for our bi-directional hypothesis, non-directional hypothesis and one-directional hypothesis. As can be seen, our bi-directional hypothesis achieves higher test power by considering the inference and its contrary direction and adaptive parameter selection. Figure 6 exploits the relationship between the optimal parameter β and the probability ξ = Pr[ F glyph[latticetop] L ′-1 ( c ˆ X ′ -c ˆ Y ′ ) ≥ 0] for our MEMaBiD method. We can easily find the positive relevance between β and ξ : the larger the probability ξ , the larger the optimal parameter β .

Figure 7 indicates that the type-I error is limited about α = 0 . 05 for different β in our experiments, as shown in Theorem 4, and thus our method could effectively control the rate of falsely reject the null hypothesis, which empirically verify the trustworthiness of our ME MaBiD test. Figure 8 empirically shows the familywise error rate is limited about α ∗ = 0 . 05 for different number of local regions s ; therefore, our exploring method could control the rate of incorrectly exploiting the local regions with significant difference, and this is nicely in accordance with Theorem 5.

5 Conclusion

This work takes one more step on the exploration of local significant differences. We propose the MEMaBiD test by exploiting local information from multiple Mahalanobis kernels and introducing bi-directional hypothesis for testing. We partition embedding space via a new splitting criterion, and then identify local significant differences based on our bi-directional masked p -value and MEMaBiD test. We verify the effectiveness of our proposed methods both theoretically and empirically. An interesting work is to explore other local and directional information for local significant differences.

Acknowledgments and Disclosure of Funding

The authors want to thank the reviewers for their helpful comments and suggestions. This research was supported by National Key R&D Program of China (2021ZD0112802), NSFC (61921006, 62376119) and CAAI-Huawei MindSpore Open Fund. W. Gao is corresponding author of this paper.

References

S. Shell, S.-M. Park, A. R. Radjabi, R. Schickel, E. O. Kistner, D. A. Jewell, C. Feig, E. Lengyel, and M. E. Peter. Let-7 expression defines two differentiation stages of cancer. National Academy of Sciences , 104(27):11400-11405, 2007.

G. Boracchi, D. Carrera, C. Cervellera, and D. Macciò. Quanttree: Histograms for change detection in multivariate data streams. In Proceedings of the 35th International Conference on Machine Learning , pages 638-647, Stockholm, Sweden, 2018.

C.-L. Li, W.-C Chang, Y. Cheng, Y.-M. Yang, and B. Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems 30 , pages 2203-2213. Curran Associates, Dutchess, NY, 2017.

D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A.J. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In Proceedings of the 5th International Conference on Learning Representations , Toulon, France, 2017.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research , 13(1):723-773, 2012.

W. Zaremba, A. Gretton, and M. B. Blaschko. B-test: A non-parametric, low variance kernel two-sample test. In Advances in Neural Information Processing Systems 26 , pages 755-763. Curran Associates, Dutchess, NY, 2013.

G. Wynne and A. B. Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research , 23(73):1-51, 2022.

S. Shekhar, I. Kim, and A. Ramdas. A permutation-free kernel two-sample test. In Advances in Neural Information Processing Systems 35 , pages 18168-18180. Curran Associates, Dutchess, NY, 2022.

K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. In Advances in Neural Information Processing Systems 28 , pages 1981-1989. Curran Associates, Dutchess, NY, 2015.

W. Jitkrittum, Z. Szabó, K. P. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems 29 , pages 181-189. Curran Associates, Dutchess, NY, 2016.

M. Scetbon and G. Varoquaux. Comparing distributions: glyph[lscript] 1 geometry improves kernel twosample testing. In Advances in Neural Information Processing Systems 32 , pages 12306-12316. Curran Associates, Dutchess, NY, 2019.

P. Golland and B. Fischl. Permutation tests for classification: towards statistical significance in image-based studies. In Proceedings of the 18th International Conference on Information Processing in Medical Imaging , pages 330-341, Cumbria, England, 2003.

D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In Proceedings of the 5th International Conference on Learning Representations , Toulon, France, 2017.

X.-Y. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks. CoRR/abstract , 1909.11298, 2019.

H. Cai, B. Goggin, and Q. Jiang. Two-sample test based on classification probability. Statistical Analysis and Data Mining: The ASA Data Science Journal , 13(1):5-13, 2020.

I. Kim, A. Ramdas, A. Singh, and L. Wasserman. Classification accuracy as a proxy for two-sample testing. Annals of Statistics , 49(1):411-434, 2021.

S. Jang, S. Park, I. Lee, and O. Bastani. Sequential covariate shift detection using classifier two-sample tests. In Proceedings of the 39th International Conference on Machine Learning , pages 9845-9880, Baltimore, MD, 2022.

X. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions on Information Theory , 68(10): 6631-6662, 2022.

J.-M. Kübler, V. Stimper, S. Buchholz, K. Muandet, and B. Schölkopf. AutoML Two-Sample Test. In Advances in Neural Information Processing Systems 35 , pages 15929-15941. Curran Associates, Dutchess, NY, 2022.

S. Hediger, Loris L. Michel, and J. Näf. On the use of random forest for two-sample testing. Computational Statistics & Data Analysis , 170:107435-107468, 2022.

K. Muandet, K. Fukumizu, B. Sriperumbudur, and B. Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends R © in Machine Learning , 10(1-2): 1-141, 2017.

P .E. Freeman, I. Kim, and A.B. Lee. Local two-sample testing: A new tool for analysing high-dimensional astronomical data. Monthly Notices of the Royal Astronomical Society , 471 (3):3273-3282, 2017.

A. T. L. Lun, A. C. Richard, and J. C. Marioni. Testing for differential abundance in mass cytometry data. Nature Methods , 14(7):707-709, 2017.

M. Roederer and R. R. Hardy. Frequency difference gating: A multivariate method for identifying subsets that differ between samples. Journal of the International Society for Analytical Cytology , 45(1):56-64, 2001.

T. Duong, I. Koch, and M. P. Wand. Highest density difference region estimation with application to flow cytometric data. Journal of Mathematical Methods in Biosciences , 51(3):504-521, 2009.

J. Soriano and L. Ma. Probabilistic multi-resolution scanning for two-sample differences. Journal of the Royal Statistical Society: Series B , 79(2):547-572, 2017.

J. A. Pura, X.-C. Li, C. Chan, and J.-C. Xie. Team: A multiple testing algorithm on the aggregation tree for flow cytometry analysis. Annals of Applied Statistics , 17(1):621-640, 2023.

T. Duong. Local significant differences from nonparametric two-sample tests. Journal of Nonparametric Statistics , 25(3):635-645, 2013.

J. D. Dworkin, K. A. Linn, A. J. Solomon, T. D. Satterthwaite, A. Raznahan, R. Bakshi, and R. T. Shinohara. A local group differences test for subject-level multivariate density neuroimaging outcomes. Biostatistics , 22(3):646-661, 2021.

I. Kim, Ann. B. Lee, and J. Lei. Global and local two-sample tests via regression. Electronic Journal of Statistics , 13(2):5253-5305, 2019.

S. Sonnenburg, G. Rätsch, and C. Schäfer. A general and efficient multiple kernel learning algorithm. In Advances in Neural Information Processing Systems 18 , pages 1273-1280. MIT Press, Cambridge, MA, 2005.

M. Gönen and E. Alpaydın. Multiple kernel learning algorithms. Journal of Machine Learning Research , 12:2211-2268, 2011.

R. De Maesschalck, D. Jouan-Rimbaud, and D. .L. Massart. The mahalanobis distance. Chemometrics and Intelligent Laboratory Systems , 50(1):1-18, 2000.

Q. K. Weinberger and K. L. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research , 10(9):207-244, 2009.

J. Wang, A. Kalousis, and A. Woznica. Parametric local metric learning for nearest neighbor classification. In Advances in Neural Information Processing Systems 25 , page 1610-1618. Curran Associates, Dutchess, NY, 2012.

E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell. Distance metric learning with application to clustering with side-information. In Advances in Neural Information Processing Systems 15 , pages 505-512. MIT Press, Cambridge, MA, 2002.

S. Xiang, F. Nie, and C. Zhang. Learning a mahalanobis distance metric for data clustering and classification. Pattern Recognition , 41(12):3600-3612, 2008.

A. Gretton, B. K. Sriperumbudur, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Balakrishnan, and K. Fukumizu. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems 25 , pages 1214-1222. Curran Associates, Dutchess, NY, 2012.

F. Liu, W.-K. Xu, J. Lu, G.-Q. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In Proceedings of the 37th International Conference on Machine Learning , pages 6316-6326, Virtual, 2020.

B. S. Everitt. A monte carlo investigation of the robustness of hotelling's one-and two-sample t 2 tests. Journal of the American Statistical Association , 74(365):48-51, 1979.

H. Hotelling. The generalization of student's ratio. Annals of Mathematical Statistics , 2(3): 360-378, 1931.

N. Deb, B. B. Bhattacharya, and B. Sen. Efficiency lower bounds for distribution-free hotellingtype two-sample tests based on optimal transport. CoRR/abstract , 2104.01986, 2021.

S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

E. Deadman, N. J. Higham, and R. Ralha. Blocked schur algorithms for computing the matrix square root. In Proceedings of the 11th International Workshop on Applied Parallel Computing , pages 171-182, Helsinki, Finland, 2012.

L. Wasserman. All of Statistics: A Concise Course in Statistical Inference . Springer, 2004.

J. Neyman and E. S. Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London: Series A , 231:289-337, 1933.

J. Cornfield. Sequential trials, sequential analysis and the likelihood principle. The American Statistician , 20(2):18-23, 1966.

C. R. Genovese, K. Roeder, and L. Wasserman. False discovery control with p-value weighting. Biometrika , 93(3):509-524, 2006.

F. Zhang and J. Gou. Refined critical boundary with enhanced statistical power for nondirectional two-sided tests in group sequential designs with multiple endpoints. Statistical Papers , 62(3):1265-1290, 2021.

H. F. Kaiser. Directional statistical decisions. Psychological Review , 67(3):160-167, 1960.

A. Cohen and H. B. Sackrowitz. Directional tests for one-sided alternatives in multivariate models. Annals of Statistics , 26(6):2321-2338, 1998.

O. Ibe. Fundamentals of Applied Probability and Random Processes . Academic Press, 2014.

L. Anselin. Lagrange multiplier test diagnostics for spatial dependence and spatial heterogeneity. Geographical analysis , 20(1):1-17, 1988.

D. Follmann. A simple multivariate test for one-sided alternatives. Journal of the American Statistical Association , 91(434):854-861, 1996.

M. J. McIntosh. Calculating sample size for follmann's simple multivariate test for one-sided alternatives. American Statistician , 76(1):16-21, 2022.

J. M. Lachin. Applications of the wei-lachin multivariate one-sided test for multiple outcomes on possibly different scales. PloS One , 9(10):e108784, 2014.

F. A. Graybill. Theory and Application of the Linear Model . Duxbury Press, 1976.

L. Ma and W.-H. Wong. Coupling optional pólya trees and the two sample problem. Journal of the American Statistical Association , 106(496):1553-1565, 2011.

G. Boracchi, C. Cervellera, and D. Macciò. Uniform histograms for change detection in multivariate data. In Proceedings of the 30th International Joint Conference on Neural Networks , pages 1732-1739, Brisbane, Australia, 2017.

A.-J. Liu, J. Lu, and G.-Q. Zhang. Concept drift detection via equal intensity k-means space partitioning. IEEE Transactions on Cybernetics , 51(6):3198-3211, 2021.

G. V. Kass. An exploratory technique for investigating large quantities of categorical data. Journal of the Royal Statistical Society: Series C , 29(2):119-127, 1980.

J. R. Quinlan. Induction of decision trees. Machine Learning , 1(1):81-106, 1986.

J. H. Friedman. A recursive partitioning decision rule for nonparametric classification. IEEE Transactions on Computers , 26(4):404-408, 1977.

J. R. Quinlan. Simplifying decision trees. International Journal of Man-machine Studies , 27(3): 221-234, 1987.

S. B. Gelfand, CS Ravishankar, and E. J. Delp. An iterative growing and pruning algorithm for classification tree design. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics , pages 818-823, Cambridge, MA, 1989.

L. Rokach and O. Maimon. Top-down induction of decision trees classifiers-a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C , 35(4):476-487, 2005.

B. Duan, A. Ramdas, and L. A. Wasserman. Familywise error rate control by interactive unmasking. In Proceedings of the 37th International Conference on Machine Learning , pages 2720-2729, Virtual, 2020.

G. S. Mudholkar and D. K Srivastava. A class of robust stepwise alternatives to hotelling's t 2 tests. Journal of Applied Statistics , 27(5):599-619, 2000.

Y. Hochberg and A. C. Tamhane. Multiple Comparison Procedures . John Wiley & Sons, Inc., 1987.

S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics , 6(2):65-70, 1979.

F. Bretz, W. Maurer, W. Brannath, and M. Posch. A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine , 28(4):586-604, 2009.

A. C. Tamhane and J. Gou. Advances in p-value based multiple test procedures. Journal of Biopharmaceutical Statistics , 28(1):10-27, 2018.

B. Duan, A. Ramdas, and L. Wasserman. Interactive rank testing by betting. In Proceedings of the 1st Conference on Causal Learning and Reasoning , pages 201-235, Eureka, Canada, 2022.

W. Li, G. Dasarathy, K. N. Ramamurthy, and V. Berisha. A label efficient two-sample test. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence , pages 1168-1177, Eindhoven, Netherlands, 2022.

A. Schrab, I. Kim, B. Guedj, and A. Gretton. Efficient aggregated kernel tests using incomplete u -statistics. In Advances in Neural Information Processing Systems 35 , pages 18793-18807. Curran Associates, Dutchess, NY, 2022.

F. Liu, W. Xu, J. Lu, and D. J. Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. In Advances in Neural Information Processing Systems 34 , pages 5848-5860. Curran Associates, Dutchess, NY, 2021.

F. Cazáis and A. Lhéritier. Beyond two-sample-tests: Localizing data discrepancies in highdimensional spaces. In Proceedings of the IEEE International Conference on Data Science and Advanced Analytics , pages 1-10, Paris, France, 2015.

M. Sugiyama, T. Kanamori, T. Suzuki, M. Plessis, S. Liu, and I. Takeuchi. Density-difference estimation. In Advances in Neural Information Processing Systems 25 , pages 692-700. Curran Associates, Dutchess, NY, 2012.

Y. P. Mack and M. Rosenblatt. Multivariate k-nearest neighbor density estimates. Journal of Multivariate Analysis , 9(1):1-15, 1979.

S. Dasgupta and S. Kpotufe. Optimal rates for k-nn density and mode estimation. In Advances in Neural Information Processing Systems 27 , pages 2555-2563. Curran Associates, Dutchess, NY, 2014.

W. Jitkrittum, Z. Szabó, and A. Gretton. An adaptive test of independence with analytic kernel embeddings. In Proceedings of the 34th International Conference on Machine Learning , pages 1742-1751, Sydney, Australia, 2017.

R. A. Johnson and D. W. Wichernand. Applied Multivariate Statistical Analysis . Prentice Hall, 2002.

Papoulis A and U. Pillai. Probability, Random Variables and Stochastic Processes . McGraw Hill, 2001.

K. V. Mardia, J. T. Kent, and J. M. Bibby. Multivariate analysis. Probability and Mathematical Statistics , 1979.

L. Lei and W. Fithian. Adapt: An interactive procedure for multiple testing with side information. Journal of the Royal Statistical Society: Series B , 80(4):649-679, 2018.

S. Rabanser, S. Günnemann, and Z. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In Advances in Neural Information Processing Systems 32 , pages 1396-1408. Curran Associates, Dutchess, NY, 2019.

A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the 4th International Conference on Learning Representations , San Juan, PR, 2016.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations , San Diego, CA, 2015.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Dutchess, NY, 2019.

A Detailed Proofs for Our Theoretical Results

A.1 Proof of Lemma 1

We begin with some useful definitions and lemmas as follows:

Definition 6. (Random Metric [9]). We say that ρ is a random metric with values in R , indexed with pairs from the set of probability measures M , i.e., ρ = { ρ ( P , Q ): P , Q ∈ M} , if it satisfies the conditions for a metric with qualification 'almost surely'. Formally, for every P , Q , U ∈ M , random variables ρ ( P , Q ) , ρ ( P , U ) and ρ ( Q , U ) satisfy

ρ ( P , Q ) ≥ 0 a.s.

glyph[negationslash]

glyph[negationslash]

Lemma 7. [11] Let µ be a absolutely continuous Lebesgue measure on R d . A non-zero analytic function f can be zero at most in the set of measure 0 w.r.t. µ .

Lemma 8. [9] If κ is a bounded and analytic kernel on R d × R d , then it is analytic for every function in the RKHS associated with this kernel.

Definition 9. For test locations V = { v 1 , v 2 , · · · , v glyph[lscript] } and Mahalanobis kernels κ 1 , κ 2 , · · · , κ glyph[lscript] given by Eqn. (1), we define

where [ µ j P ] ( v j ) = E x ∼ P [ κ j ( x , v j )] and [ µ j Q ] ( v j ) = E y ∼ Q [ κ j ( y , v j )] .

We show that the above distance metric for two probability measures is a random metric as follows: Lemma 10. If v 1 , v 2 , · · · , v glyph[lscript] are drawn i.i.d. from a absolutely continuous distribution G , then ρ µ 1 ,µ 2 , ··· ,µ glyph[lscript] ( · , · ) is a random metric for bounded kernels { κ 1 , κ 2 , · · · , κ glyph[lscript] } .

Proof. For j ∈ [ glyph[lscript] ] , we first introduce a function and it is sufficient to prove that ρ µ j ( P , Q ) is a random metric for each j ∈ [ glyph[lscript] ] from Eqn. (12).

It is well-known that Mahalanobis kernels in Eqn. (1) are characteristic and analytic from [81], and the corresponding mapping

glyph[negationslash]

are injective for κ j , where µ j P and µ j Q denote the images of measures P and Q , respectively. Hence, the image of µ j is a subset of analytic functions for analytic and bounded κ j , according to Lemma 8.

If P = Q , then we have

glyph[negationslash]

We now prove that if P = Q then f j = 0 almost surely, by applying Lemma 7 to analytic function f j = µ j P -µ j Q with distribution G . For injective map µ j , there exists at least one point a such that f j ( a ) = 0 , and there exists a ball around a with non-zero f j from the continuity of f j . Hence, f is almost everywhere nonzero based on Lemma 7, and this follows that

Hence, ρ µ j is random metric from Definition 6 from the symmetry and triangle inequality of ρ µ j .

Proof of Lemma 1. We first have

glyph[negationslash]

glyph[negationslash]

and further rewrite Eqn. (12) as

From Lemma 10, we can see that ρ µ 1 ,µ 2 , ··· ,µ glyph[lscript] ( P , Q ) = 0 if and only if P = Q , and this implies that µ ˆ P = µ ˆ Q if and only if P = Q . This completes the proof of Lemma 1.

A.2 Proof of Theorem 2

We begin with a useful lemma from [11] as follows.

Lemma 11. For symmetric and positive definite matrix Σ , function h (Σ) = Σ -1 / 2 is continuous and well-defined on the positive definite space.

Based on this lemma, we present the detailed proof of Theorem 2 as follows:

Proof of Theorem 2 . For embedding testing sample ˆ X ′ = { ˆ x ′ 1 , · · · , ˆ x ′ m ′ } and ˆ Y ′ = { ˆ y ′ 1 , · · · , ˆ y ′ n ′ } , recall the pooled covariance matrix in Eqn. (3) as and we also define

We first prove that testing statistic T ( ˆ X ′ , ˆ Y ′ ) is almost surely asymptotically distributed as χ 2 glyph[lscript] with glyph[lscript] degrees of freedom under the condition µ ˆ P = µ ˆ Q for small glyph[epsilon1] .

For i.i.d. samples ˆ X ′ and ˆ Y ′ , c ˆ x is independent to c ˆ y . If µ ˆ P = µ ˆ Q , then ¯ z = c ˆ X -c ˆ Y follows a multivariate normal distribution with mean µ = µ ˆ P -µ ˆ Q = 0 and covariance matrix ( m ′ + n ′ )Σ ˆ P , ˆ Q /m ′ n ′ , from [41, 82] and the Slutsky's theorem [83], that is

where d → denotes convergence in distribution.

The matrix L is symmetric and invertible in the Schur decomposition Σ ˆ X ′ , ˆ Y ′ = LL . Our statistic can be formalized as

and by applying the Slutsky's theorem, we have

This follows that

where ( L ′-1 ¯ z ) j is the j -th dimension value of L ′-1 ¯ z , and √ m ′ n ′ / ( m ′ + n ′ )( L ′-1 ¯ z ) j follows the standard normal distribution. This proves the χ 2 glyph[lscript] distribution for T ( ˆ X ′ , ˆ Y ′ ) from the sum of glyph[lscript] squares of standard (i.i.d.) normal random variables.

glyph[negationslash]

We then prove that χ 2 glyph[lscript] ( T ( ˆ X ′ , ˆ Y ′ )) → 0 as m ′ n ′ / ( m ′ + n ′ ) →∞ if µ ˆ P = µ ˆ Q , and hence our test rejects H 0 almost surely.

From Lemma 11, we have

In a similar manner, ¯ z = c ˆ X -c ˆ Y converges to µ ˆ P -µ ˆ Q in probability. For µ ˆ P = µ ˆ Q , we have glyph[negationslash]

Then, ‖ L ′-1 ¯ z ‖ 2 2 is a continuous function with entries ¯ z and L ′-1 , and it is convergent to some positive constant. We have ( m ′ + n ′ ) / ( m ′ n ′ ) χ 2 glyph[lscript],α → 0 , and

This follows that

and hence χ 2 glyph[lscript] ( T ( ˆ X ′ , ˆ Y ′ )) → 0 as m ′ n ′ / ( m ′ + n ′ ) →∞ if µ ˆ P = µ ˆ Q .

A.3 Proof of Lemma 3

Recall the pooled covariance matrix Σ ˆ X ′ , ˆ Y ′ and Σ ˆ P , ˆ Q = E ˆ X ′ ∼ ˆ P m ′ , ˆ Y ′ ∼ ˆ Q n ′ [Σ ˆ X ′ , ˆ Y ′ ] in the proof of Theorem 2, and ¯ z = ( c ˆ X ′ -c ˆ Y ′ ) follows a multivariate normal distribution with mean µ ˆ P -µ ˆ Q and covariance matrix ( m ′ + n ′ )Σ ˆ P , ˆ Q /m ′ n ′ . We first observe

and this follows that

This is because E [ L ′-1 ¯ z ] = L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) and the covariance matrix is given by

Write B = √ m ′ n ′ / ( m ′ + n ′ ) L ′-1 diag ( F )¯ z , and we have

by using the symmetry of Σ -1 ˆ X ′ , ˆ Y ′ . It is easy to get

and covariance matrix is given by

glyph[negationslash]

This yields that

and all random variables in B are mutually independent. Define

and ¯ B is normally distributed with mean √ m ′ n ′ / ( m ′ + n ′ ) F glyph[latticetop] L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) /glyph[lscript] and variance 1 /glyph[lscript] . It is easy to see that which yields that

We further have

where Φ( · ) is the cumulative distribution function of standard Gaussian distribution. For continuous normal distribution, we have

and this follows that

Hence, the sgn ( F glyph[latticetop] L ′-1 ( c ˆ X ′ -c ˆ Y ′ )) follows a two-point distribution T P ( ξ ) with parameter

This completes the proof.

A.4 Proof of Theorem 4

Recall the pooled covariance matrix Σ ˆ X ′ , ˆ Y ′ and Σ ˆ P , ˆ Q = E ˆ X ′ ∼ ˆ P m ′ , ˆ Y ′ ∼ ˆ Q n ′ [Σ ˆ X ′ , ˆ Y ′ ] in the proof of Theorem 2, and ¯ z = ( c ˆ X ′ -c ˆ Y ′ ) follows a multivariate normal distribution with mean µ ˆ P -µ ˆ Q and covariance matrix ( m ′ + n ′ )Σ ˆ P , ˆ Q /m ′ n ′ . We have L ˆ P , ˆ Q L ˆ P , ˆ Q = Σ ˆ P , ˆ Q .

We begin with a useful lemma and corollary as follows.

Proof. Recall B = √ m ′ n ′ / ( m ′ + n ′ ) L ′-1 diag ( F )¯ z and F glyph[latticetop] L ′-1 ¯ z = √ ( m ′ + n ′ ) /m ′ n ′ 1 glyph[latticetop] B in the proof of Lemma 3, and Eqn. (14) is equivalent to

Recall that

and from Eqn. (15), we have

From the independence of S 2 and ¯ B , it is sufficient to prove that, for every δ ≥ 0 ,

It's easy to see that ¯ B is normally distributed with mean √ m ′ n ′ / ( m ′ + n ′ ) F glyph[latticetop] L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) /glyph[lscript] and variance 1 /glyph[lscript] . We define

and from Eqn. (16), we have where the equality holds from b = 0 , i.e., µ ˆ P = µ ˆ Q . This completes the proof.

Lemma 14. If F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ P > F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ Q , then the test power of our bi-directional hypothesis can be lower bounded by

where

ω

=

m

n

/

(

m

+

n

)

,

ν

=

m

+

n

-

1

and

λ

=

ω

‖

L

-

ˆ

P

1

,

ˆ

Q

(

µ

ˆ

P

-

Proof. From Lemma 12, we first have, for F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ P > F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ Q ,

by substituting c = χ 2 glyph[lscript],β α into Eqn. (14). From µ ˆ P = µ ˆ Q , we also have, from the work of [84], glyph[negationslash]

where λ = m ′ n ′ ( µ ˆ P -µ ˆ Q ) glyph[latticetop] Σ -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) / ( m ′ + n ′ ) . Given the significance level βα , we have

and this follows that

Recall B = √ m ′ n ′ / ( m ′ + n ′ ) L ′-1 diag ( F )¯ z and F glyph[latticetop] L ′-1 ¯ z = √ ( m ′ + n ′ ) /m ′ n ′ 1 glyph[latticetop] B in the proof of Lemma 3, and that ¯ B = 1 glyph[latticetop] B /glyph[lscript] is normally distributed with mean √ m ′ n ′ / ( m ′ + n ′ ) F glyph[latticetop] L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) /glyph[lscript] and variance 1 /glyph[lscript] . We have

′

′

′

′

′

′

µ

ˆ

)

‖

2

.

Q

2

since ¯ B is normally distributed with mean √ m ′ n ′ / ( m ′ + n ′ ) F glyph[latticetop] L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) /glyph[lscript] and variance 1 /glyph[lscript] . Combining with Eqns. (17)-(19), we have

Combining with Eqns. (20)-(21), we give a lower bound for the test power of bi-directional hypothesis

Lemma 15. If F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ P < F glyph[latticetop] L -1 ˆ P , ˆ Q µ ˆ Q , then the test power of our bi-directional hypothesis can be lower bounded by

Proof.

Recall that

in the proof of Lemma 14, and this follows that

,

Lemma 16. For our bi-directional hypothesis, the type-I error rate is equal to α if µ ˆ P = µ ˆ Q .

Proof. We first consider the case F glyph[latticetop] L ′-1 ¯ z > 0 . By substituting c = χ 2 glyph[lscript],β α into Eqn. (14), we have

Recall B = √ m ′ n ′ / ( m ′ + n ′ ) L ′-1 diag ( F )¯ z and F glyph[latticetop] L ′-1 ¯ z = √ ( m ′ + n ′ ) /m ′ n ′ 1 glyph[latticetop] B in the proof of Lemma 3, and that ¯ B = 1 glyph[latticetop] B /glyph[lscript] is normally distributed with mean √ m ′ n ′ / ( m ′ + n ′ ) F glyph[latticetop] L -1 ˆ P , ˆ Q ( µ ˆ P -µ ˆ Q ) /glyph[lscript] and variance 1 /glyph[lscript] . We have and we have

since ¯ B is normally distributed with mean 0 .

For the case F glyph[latticetop] L ′-1 ¯ z < 0 , we similarly substitute c = χ 2 glyph[lscript], (2 -β ) α into Eqn. (14), and it follows that

From Eqns. (24)-(25), we have

This completes the proof.

Theorem 4 follows from Lemmas 14-16.

A.5 Proof of Theorem 5

Recall that B 〈 1 〉 , B 〈 2 〉 , . . . , B 〈 s 〉 are rectangle regions of a non-increasing order w.r.t. g ( · , · ) . For each rectangle region B 〈 i 〉 , we could define its local null hypothesis

From Theorem 2, the testing statistic T ( X ′ B 〈 i 〉 , Y ′ B 〈 i 〉 ) follows the χ 2 distribution with freedom of glyph[lscript] degrees under the local null hypothesis H 0 , 〈 i 〉 . Denote by χ 2 glyph[lscript] ( T ( ˆ X ′ B i , ˆ Y ′ B i )) the p -value, and we have

Lemma 17. [68] The p -value χ 2 glyph[lscript] ( T ( ˆ X ′ B , 〈 i 〉 , ˆ Y ′ B , 〈 i 〉 )) follows a uniform distribution U [0 , 1] under the local null hypothesis H 0 , 〈 i 〉 : µ ˆ P B 〈 i 〉 = µ ˆ Q B 〈 i 〉 .

We take an interactive multi-step testing procedure to identify the index set of rectangle regions of local significant differences. Define the candidate rejection set R ( t ) = { H 0 , 〈 i 〉 } t i =1 for 1 ≤ t ≤ s with R (0) = ∅ , and exclude one null hypothesis H 0 , 〈 t 〉 at the t -th step. We could generate a sequence as follows:

and it holds that H 0 , 〈 i 〉 = R ( i ) \ R ( i -1) . Recall the local bi-directional hypothesis

Denote by p ∗ the parameter of significant level for the local two-sample test and masked p -value. From Theorem 4, we have Pr[ h 〈 i 〉 = 1] = p ∗ under the null local hypothesis H 0 , 〈 i 〉 . We further present some useful lemmas as follows.

Lemma18. [67] We have E [ h 〈 i 〉 ] = p ∗ for i ∈ [ s ] and h 〈 1 〉 , h 〈 2 〉 , · · · , h 〈 s 〉 are mutually independent, if µ ˆ P B 〈 i 〉 = µ ˆ Q B 〈 i 〉 for every i ∈ [ s ] and the p -values are uniformly distributed.

Denote by ˜ B the set of rectangle regions that the local two samples ˆ X ′ B 〈 i 〉 and ˆ Y ′ B 〈 i 〉 are actually drawn from one identical distribution, and we define

Lemma 19. [67] If there is some rectangle region such that µ ˆ P B 〈·〉 = µ ˆ Q B 〈·〉 , and if the p -value follows a uniform distribution under null hypothesis H 0 , 〈·〉 , then we have glyph[negationslash]

We also define the weighted mirror-conservativeness , motivated from [85, 67], as follows.

Definition 20. We say that a density function f ( · ) satisfies the weighted mirror-conservativeness if it holds that, for some given p ∈ (0 , 1] ,

Here, we introduce an additional parameter w to incorporate two directions for our method, which is different from the previous mirror-conservativeness [85, 67]. We could also present two sufficient conditions for weighted mirror-conservativeness from [85, 67]: i) the non-decrease of f and ii) the convexity of cumulative density function of p -value.

∈

Lemma 21. If the density function of the p -value satisfies the weighted mirror-conservativeness under local null hypothesis H 0 , 〈·〉 , and if there is some rectangle region such that µ ˆ P B 〈·〉 = µ ˆ Q B 〈·〉 , then we have, for every i [ s ] , glyph[negationslash]

where g 〈 k 〉 = g ( ˆ X ′ B 〈 k 〉 , ˆ Y ′ B 〈 k 〉 ) .

Proof. We first prove

from our bi-directional hypothesis and masked p -value with β ∈ [1 , 2] in Eqn. (7), and it is sufficient to consider the following two cases:

where the last inequality holds from the weighted mirror-conservativeness with p -value's uniform distribution U [0 , 1] .

We define the information available for choosing H 0 , 〈 i 〉 as a filtration (sequence of nested σ -fields)

and also define the filtration

This follows that

where the last equality holds from the law of total expectation. We have

where the last equation holds from the fact that {F h 〈 i 〉 , H 0 , 〈 i 〉 ∈ H 0 , { g 〈 k 〉 } s k =1 } is a subset of F 〈 i 〉 . We further have, since h 〈 i 〉 is independent of other information in F 〈 i 〉 ,

which completes the proof.

We say that a random variable Z follows a Bernoulli distribution with parameter p , denoted by Z ∼ B ern ( p ) , if

We also say that a random variable Z follows a negative binomial distribution with parameters r and p , denoted by Z ∼ NB ( r, p ) , if

It is necessary to introduce a definition as follows:

Definition 22. We say that a random variable Z is stochastically dominated by a distribution G , denoted by if for random variable X ∼ G , it holds that

We further introduce some useful lemmas as follows:

Lemma 23. [67] Let Z 1 , · · · , Z s be i.i.d random variables with Z i ∼ B ern ( p ∗ ) for some p ∗ > 0 , and write ˜ N t = ∑ t j =1 Z j and ˜ G t = σ ( ˜ N t , { Z j } s j = t +1 ) for t ∈ [ s ] . We have

where the stopping index ˜ τ is parameterized by some constant v ( ≥ 1) , defined by

We further introduce a weighted version of Lemma 23 as follows:

Lemma 24. [67] Let { W j } s j =1 be a sequence of weights, drawn from a Bernoulli distribution, s.t. ∑ s j =1 W j = u for fixed constant u ≤ s ; and Z j | σ ( { Z k , W k } s k = j +1 , W j = 1 ) ∼ B ern ( p ∗ ) . Write N w t = ∑ t j =1 W j Z j , and we have

where the stopping index τ w is parameterized by some constant v ( ≥ 1) , defined by

We now introduce a different version of Lemma 24 by considering different parameter for Bernoulli distribution as follows:

Lemma 25. [67] Let Z j | σ ( { Z k , W k } s k = j +1 , W j = 1 ) follow a Bernoulli distribution with parameter p ( { Z k , W k } s k = j +1 ) for j ∈ [ s ] , respectively. We have

We now present the detailed proof of Theorem 5 as follows.

Proof of Theorem 5 . We first consider µ ˆ P B 〈 i 〉 = µ ˆ Q B 〈 i 〉 for every i ∈ [ s ] . From Lemma 18, { h 〈 i 〉 } s i =1 are s i.i.d. random variables with h 〈 i 〉 ∼ B ern ( p ∗ ) . Recall that the stopping rule in our testing, i.e., Eqn. (11), which is equivalent to

where α ∗ is a parameter to control familywise error rate and I ( t ) = { i ∈ [ t ] : h ( ˆ X ′ B 〈 i 〉 , ˆ Y ′ B 〈 i 〉 ) = 1 } . The stopping rule can be rewritten as t -|I ( t ) | < v with

Let Z j = h 〈 j 〉 and ˜ N t = ∑ t j =1 Z j . We define the stopping index ˜ τ as follows

From Lemma 23, the number of rejections at the stopping index is given by

If µ ˆ P B 〈 i 〉 = µ ˆ Q B 〈 i 〉 for every i ∈ [ s ] , then the number of false rejections is

and hence the familywise error rate (FWER) is upper bounded by

glyph[negationslash]

Let Z j = h 〈 j 〉 and W j = I ( H 0 , 〈 j 〉 ∈ H 0 ) . We define the stopping index τ w as follows:

where the last inequality follows from Eqn. (29).

We now consider that there is some rectangle region with µ ˆ P B 〈·〉 = µ ˆ Q B 〈·〉 . In such case, we provide an upper bound for familywise error rate without the information of masked p -values, and prove that the number of false rejections is stochastically dominated by NB ( v, p ∗ ) .

where v is given in Eqn. (29). It is easy to see that

from Lemma 19. Denote by u = |H 0 | , and we have ∑ s j =1 W j = u and u ≤ s . From Lemma 24, we have the number of false rejections

Recall that t - |I ( t ) | < v is our stopping rule on the exploration of local significant differences. Denote by τ w T the stopping index in our exploration, and we have

Since N w t is non-decreasing with respect to t , it is easy to obtain

and we have the number of false rejections

We upper bound the familywise error rate without considering the masked p -values as follows:

We finally take masked p -values { g 〈 k 〉 } s k =1 into consideration. From Lemma 21, it is easy to observe where

This follows that and we further have, from Lemma 25,

We finally upper bound the familywise error rate by considering the masked p -values { g 〈 k 〉 } s k =1 as

This completes the proof.

B Optimization for Test Locations and Mahalanobis Kernels

We take gradient method [43] for the optimization of Eqn. (5) as in the work of [10]. Specifically, we calculate gradients, and update test locations and Mahalanobis kernels iteratively. In the following of this section, we present the calculation of some crucial gradients in optimization.

For test location v j with j ∈ [ glyph[lscript] ] , we have

where, for i ∈ [ glyph[lscript] ] ,

where Tr [ · ] denotes the trace.

We further have

where all elements are zeros except for the j -th element. We also have

where the i -th element is 1 . We similarly calculate ∂ T ( ˆ X, ˆ Y ) /∂ c ˆ Y × ∂ c ˆ Y /∂ v j,i .

For the third term in Eqn. (33), we have

where

glyph[negationslash]

glyph[negationslash]

and this follows that

We similarly have

glyph[negationslash]

For gamma parameter γ j with j ∈ [ glyph[lscript] ] , we have

We further have

where glyph[negationslash]

glyph[negationslash]

We similarly calculate

where

glyph[negationslash]

glyph[negationslash]

and this follows that

We similarly have

glyph[negationslash]

For Mahalanobis matrix M j with j ∈ [ glyph[lscript] ] , we have where we denote by M j,a,b the element in a -th row and b -th column in M j . We further have

with and

where J a,b is the single-entry matrix ( 1 at ( a, b ) and zero elsewhere). We similarly calculate ∂ c ˆ Y /∂M j,a,b and have

where

Here, we have

glyph[negationslash]

We similarly have ∂ (ˆ x r -c ˆ X )(ˆ x r -c ˆ X ) glyph[latticetop] t,j / ∂M j,a,b and glyph[negationslash]

Project Mahalanobis matrix onto a positive definite cone

We can not guarantee the positive-definiteness of Mahalanobis matrices during the optimization process via gradient ascend. Motivated from [43], we project Mahalanobis matrix onto a positive definite cone as follows:

Present the spectral (eigenvalue) decomposition of a Mahalanobis matrix M as

where λ 1 , λ 2 , · · · , λ d are their eigenvalues with corresponding eigenvectors p 1 , p 2 , · · · , p d .

Project the Mahalanobis matrix M onto a positive definite cone

C Datasets and Parameter Setting

Datasets

We partition datasets into several disjoint subsets, and then randomly draw data elements from each subset based on the sample fraction, i.e., the proportion of samples to be selected from each glyph[negationslash]

subset. We construct two different samples by using two different sample fractions in above stratified sampling process, and construct two samples drawn from one identical distribution by adapting a same sample fraction, as done in [60].

We provide the details of constructing two samples for each dataset as follows:

blob is constructed as the mixture of nine Gaussian modes. We first write

and

To construct different distributions P and Q , we adapt different covariance structures and

To construct identical distribution for two samples, i.e., P = Q , we have

We set the sample size for training to 900 and for testing to 224 .

dna is a categorical dataset with 3 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 30 , 0 . 35 , 0 . 35] and for the other sample to [0 . 45 , 0 . 25 , 0 . 3] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 30 , 0 . 35 , 0 . 35] for two samples. We set the sample size for training to 1000 and for testing to 250 .

agnos (agnostic) is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 35 , 0 . 65] and for the other sample to [0 . 65 , 0 . 35] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 35 , 0 . 65] for two samples. We set the sample size for training to 1000 and for testing to 250 .

topo21 is a regression dataset with continuous target variables. Based on the sorted target variables, we divide the data into 4 equal parts. For constructing two different samples, we set the sample fraction for one sample to [0 . 1 , 0 . 3 , 0 . 2 , 0 . 4] and for the other sample to [0 . 5 , 0 . 2 , 0 . 1 , 0 . 2] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 1 , 0 . 3 , 0 . 2 , 0 . 4] for two samples. We set the sample size for training to 2200 and for testing to 550 .

har is a categorical dataset with 6 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 10 , 0 . 20 , 0 . 10 , 0 . 20 , 0 . 20 , 0 . 10] and for the other sample to [0 . 15 , 0 . 15 , 0 . 20 , 0 . 15 , 0 . 20 , 0 . 15] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 10 , 0 . 20 , 0 . 10 , 0 . 20 , 0 . 20 , 0 . 10] for two samples. We set the sample size for training to 2200 and for testing to 550 .

kropt is a categorical dataset with 18 classes, where we only consider categories 13 , 14 , 15 and 16 which have the majority of the data.. For constructing two different samples, we set the sample fraction for one sample to [0 . 15 , 0 . 2 , 0 . 3 , 0 . 35] and for the other sample to [0 . 35 , 0 . 35 , 0 . 15 , 0 . 15] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 25 , 0 . 25 , 0 . 25 , 0 . 25] for two samples. We set the sample size for training to 2000 and for testing to 500 .

diamon (diamonds) is a regression dataset with continuous target variables. Based on the sorted target variables, we divide the data into 4 equal parts. For constructing two different samples, we set the sample fraction for one sample to [0 . 35 , 0 . 2 , 0 . 2 , 0 . 25] and for the other sample to [0 . 2 , 0 . 3 , 0 . 3 , 0 . 2] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 25 , 0 . 25 , 0 . 25 , 0 . 25] for two samples. We set the sample size for training to 2000 and for testing to 500 .

Original cifar10 (samples P ) is compared to adversarial-cifar10 (samples Q ) following [19], where adversarial-cifar10 is constructed by [86] for distribution shift detection. To construct two samples with one identical distribution, we select randomly between original cifar10 and adversarial-cifar10 and drawn two samples from the same dataset. The image is scaled to 32 × 32 and we set the sample size for training to 200 and for testing to 50 .

mnist contains 70000 handwritten digit images, we compare true mnist data (samples P ) to Fake-mnist data (samples Q ) following [39], where the Fake-mnist data is drawn from a pre-trained deep convolutional generative adversarial network [87] and the image is scaled to 32 × 32 . To construct two samples with one identical distribution, we select randomly between original mnist and Fake-mnist and drawn two samples from the same dataset. We set the sample size for training to 600 and for testing to 150 .

santan (santandercustomersatisfaction) is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 8 , 0 . 2] and for the other sample to [0 . 25 , 0 . 75] ; To construct two samples with one identical distribution, we set the same sample fraction [0 . 5 , 0 . 5] for two samples. We set the sample size for training to 1000 and for testing to 250 .

codrna is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 7 , 0 . 3] and for the other sample to [0 . 35 , 0 . 65] ; To construct two samples with one identical distribution, we set same sample fraction [0 . 5 , 0 . 5] for two samples. We set the sample size for training to 1000 and for testing to 250 .

sea50 is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 7 , 0 . 3] and for the other sample to [0 . 3 , 0 . 7] ; To construct two samples with one identical distribution, we set same sample fraction [0 . 5 , 0 . 5] for two samples. We set the sample size for training to 5000 and for testing to 1250 .

adult is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 6 , 0 . 4] and for the other sample to [0 . 4 , 0 . 6] ; To construct two samples with one identical distribution, we set same sample fraction [0 . 5 , 0 . 5] for two samples. We set the sample size for training to 5000 and for testing to 1250 .

labor is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 6 , 0 . 4] and for the other sample to [0 . 45 , 0 . 55] ; To construct two samples with one identical distribution, we set same sample fraction [0 . 5 , 0 . 5] for two samples. We set the sample size for training to 5000 and for testing to 1250 .

poker is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 2 , 0 . 8] and for the other sample to [0 . 8 , 0 . 2] ; To construct two samples with one identical distribution, we set same sample fraction [0 . 5 , 0 . 5] for two samples. We set the sample size for training to 3000 and for testing to 750 .

Algorithm 2 MEMaBiD training

Input : Two training samples X and Y , number of test locations glyph[lscript] , optimization epoch K , step size η Output : { v i } glyph[lscript] i =1 , { M i } glyph[lscript] i =1 , { γ i } glyph[lscript] i =1 and F

2: for k = 1 , 2 , . . . , K do

1: Initialize { v i } glyph[lscript] i =1 , { M i } glyph[lscript] i =1 = I d and { γ i } glyph[lscript] i =1 by Eqn. (36)

3: Calculate the embeddding training samples ˆ X and ˆ Y based on Eqn (2)

4: Calculate the statistic T ( ˆ X, ˆ Y ) based on Eqn. (4)

6: Gradient ascend based on Adam optimization method with step size η

5: Calculate gradient ∇ v j T ( ˆ X, ˆ Y ) , ∇ γ j T ( ˆ X, ˆ Y ) and ∇ M j T ( ˆ X, ˆ Y ) via Eqns. (32), (34), (35)

7: end for

9: Calculate the pooled covariance matrix Σ ˆ X, ˆ Y based on Eqn. (3)

8: Calculate the embeddding training samples ˆ X and ˆ Y based on Eqn (2)

10: Calculate the Schur decomposition of the pooled covariance matrix: LL = Σ ˆ X, ˆ Y .

12: return { v i } glyph[lscript] i =1 , { M i } glyph[lscript] i =1 , { γ i } glyph[lscript] i =1 and F

11: Calculate the inference direction F based on Eqn. (6)

higgs is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0 . 0 , 1 . 0] and for the other sample to [1 . 0 , 0 . 0] ; To construct two samples with one identical distribution, we set same sample fraction [1 . 0 , 0 . 0] for two samples. We set the sample size for training to 16000 and for testing to 4000 .

In optimization, we adapt Adam optimization method from the pytorch library in python [88, 89]. Table 4 presents the details of the hyperparameter settings for each dataset, including the number of test locations, learning rate.

Experimental settings

At initialization, we usually set the Mahalanobis matrices { M i } glyph[lscript] i =1 to identity matrices. We further provide an alternative initialization for the Mahalanobis matrices based on the correlation information of two training samples as follows:

where S j = { x 1 -v j , ..., x m -v j , y 1 -v j , ..., y n -v j } , S j,i is the i -th element of S j , and c S j = ∑ m + n i =1 S j,i / ( m + n ) . I d is an identity matrix of size d × d to guarantee the positive definiteness with small constant δ > 0 . Here, the elements x and y are come from training samples X and Y , and m and n are the numbers of elements in X and Y respectively.

For initialization of test locations, we first fit a Gaussian distribution for each sample and draw half of the number of test locations from each distribution. This could be expensive for high dimensional dataset, and we can simplify the process by directly sampling test locations from original dataset.

For initialization of the bandwidth parameters of Mahalanobis kernels, we let γ j = γ for every j ∈ [ glyph[lscript] ] and then linearly search for γ that maximize the statistic from a candidate list with fixed test locations and Mahalanobis matrices, which can be formalized as follows:

where med denotes the median of pairwise Euclidean distances of points in X and Y .

Compared methods

We now present the details for our compared methods as follows:

ME 2 : The mean embeddding method learns a set of test locations and a single Gaussian kernel, and then measures the difference between two mean embedddings with a statistic following χ 2 distribution [9, 10];

Algorithm 3 MEMaBiD testing

Input : Two testing samples X ′ and Y ′ , test locations { v i } glyph[lscript] i =1 , Mahalanobis kernels parameters { M i } glyph[lscript] i =1 and { γ i } glyph[lscript] i =1 , inference direction F , significance level α and parameter β Output : h

1: Calculate the embeddding testing samples ˆ X ′ and ˆ Y ′ based on Eqn. (2)

10:

2: Calculate mean embeddings c ˆ X ′ = ∑ m ′ i =1 ˆ x ′ i /m ′ and c ˆ Y ′ = ∑ n j =1 ˆ y ′ i /n ′ 3: Calculate the pooled covariance matrix Σ ˆ X ′ , ˆ Y ′ based on Eqn. (3) 4: Calculate the Schur decomposition of the pooled covariance matrix: L ′ L ′ = Σ ˆ X ′ , ˆ Y ′ . 5: if F T ( c ˆ X ′ -c ˆ Y ′ ) ≥ 0 then 6: h = I [ χ 2 glyph[lscript] ( T ( ˆ X ′ , ˆ Y ′ )) ≤ βα ] 7: else 8: h = I [ χ 2 glyph[lscript] ( T ( ˆ X ′ , ˆ Y ′ )) ≤ (2 -β ) α ] 9: end if return h

Algorithm 4 Exploring the local significant differences

Input : Two testing samples X,Y , Mahalanobis kernels' parameters { M i } glyph[lscript] i =1 and { γ i } glyph[lscript] i =1 , test locations { v i } glyph[lscript] i =1 , partition tree T

Output : t ∗

1: Calculate the embeddding testing samples ˆ X ′ and ˆ Y ′ based on Eqn. (2)

2: Get the local two samples ˆ X ′ B i and ˆ Y ′ B i for every i ∈ [ s ] based on partition Tree T

4: Resort rectangle regions as B 〈 1 〉 , B 〈 2 〉 , . . . , B 〈 s 〉 in a non-increasing order based on g ( · , · )

3: Calculate bi-directional masked p -value g ( ˆ X ′ B i , ˆ Y ′ B i ) for each rectangle region B i

5: Calculate bi-directional hypothesis h ( ˆ X ′ B i , ˆ Y ′ B i ) for each rectangle region B i by Eqn. (7)

7: return t ∗

6: Calculate the index set t ∗ based on Eqn. (11)

C2ST-S 3 : A binary classification neural network is trained and the statistic is computed as the accuracy over a hold-out set of two samples [13];

MMDAgg 4 : A solution for the fundamental kernel selection problem involves the aggregation of a large number of kernels with several bandwidths, where the incomplete U -statistics are used to measure the difference between two samples Schrab et al. [75]. Notice that we set the testing sample size for MMDAgg to 5 times that of the other methods, since it does not require training;

MMD-D 5 : A deep kernel approach for Maximum Mean Discrepancy (MMD), where the parameters of a neural network, two lengthscales of Gaussian kernels and a regularization parameter are optimized [39];

C2ST-L 6 : A binary classification neural network is trained and the statistic is computed as the difference between outputs of the logit function corresponding to two samples [14, 18];

AutoML 7 : A binary classifier is trained based on Automated Machine Learning techniques and the statistic is same to C2ST-L approach [19].

Weimplement methods for exploring local significant differences in Python, following their respective guidelines as follows:

FDG : Partition the sample space based on probability binning and then compare the cardinalities of two samples over rectangle regions, where a normalized chi-squared value is computed for each bin to measure the local difference [24];

Step Ⅳ: 1) Split node 4 at y-axis with median position

2) Split node 5 at x-axis with median position

3) Split node 6 at x-axis with median position

4) Split node 7 at y-axis with median position

Splitting regions

y

x

8

9

10

11

12

13

14

15

Partition tree

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Step Ⅲ: 1) Split node 2 at x-axis with median position

2) Split node 3 at y-axis with median position

Splitting regions

y

x

4

5

6

7

Partition tree

1

2

3

4

5

6

7

Step Ⅱ: Split node 1 at x-axis with median position

Splitting regions

y

x

2

3

Partition tree

1

2

3

Step Ⅰ: Initialization with root 1

Splitting regions

y

x

1

Partition tree

1

K-PRIM : Partition the space based on patient rule induction method along with the information of kernel density estimation, and use a statistic following χ 2 1 distribution to identify local distributional difference [28];

MRS : Partition the sample space based on pólya tree method, which splits a leaf node based on the median of randomly selected feature, and then measure the difference between local two samples based on Binomial distribution [26];

TEAM : Partition the sample space on the median of the feature with largest sample variance, and then measure differences between local two samples with Binomial distribution [27];

BTLDD : Estimate the conditional probabilities of two samples based on a regression model and then cluster those elements with significant different conditional probabilities [77];

MMDT : Partition the sample space into multiple equal grids based on quantile values of the features, and then estimate the kernel densities of two samples and measure local differences based on Welch's two-sample t-test statistic [29].

For the exploration of local significant differences, we take density differences [78] between two samples in a local region as an evaluation measure for local significant differences, and follow the works of [79, 80] based on k -NN density estimator with k = 20 . Here, denote by X and Y the available data for density evaluation, we have estimated density functions as follows:

where z ∈ [0 , 1] d , N denotes sample size for density estimation and v d is the volume of a unit ball in R d . Denote by r X ( z ) d and r Y ( z ) d the distances from z to its 20 -th nearest neighbors in X and Y , respectively. For density estimation, we use all data of diamond, and 1 , 000 , 000 data of other datasets due to the limitation of time complexity.

For our MEMaBiD test, we present the detailed training procedure in Algorithm 2 and testing procedure in Algorithm 3. Figure 9 is a pictorial illustration to present the region-splitting method and clarify that our partition tree is constructed iteratively: We initiate tree root with embedding space, and during each iteration, we select randomly one of those leaves with the largest size of training data points, and select the feature of the largest statistics value and with the median splitting position. We present the detailed description on the exploration of local significant differences in Algorithm 4.

Running time (seconds)

10 3

10 2

10 1

10 0

blob dna

agnos topo21

ME

diamon mnist

C2ST-S

santan

C2ST-L

poker higgs

MMD-D

cifar

Datasets

blob

dna

agnos

topo21

diamon

cifar

mnist

santan

codrna

sea50

adult

labor

poker

higgs

Datasets

10

2

10

1

10 0

10 1

10 2

10 3

Running time (seconds)

Our MEMaBiD

ME

MMD-D

MMDAgg

C2ST-S

C2ST-L

AutoMLTST

D Experimental comparisons

Running time

For fair comparisons, we run all experiments on a single core without parallel optimizations, and experiments are performed with Python on nodes of a computational cluster with a single CPU (Intel Core i9-10900X 3.7GHz) and a single GPU (GeForce RTX 2080 Ti), running Ubuntu with 128GB main memory. For these methods based on deep neural networks, such as C2ST-S, C2ST-L and MMD-D, we run experiments on GPU. For other methods except for AutoMLTST, we run experiments on CPU. For AutoMLTST, we run experiments on GPU for cifar and mnist, and run experiments on CPU for other datasets.

We further compare the average training time and testing time (in seconds) for different methods on two-sample test, as shown in Figure 10 and Figure 11. Notice that, the training time of MMDAgg is 0 , since it has no training procedure. In testing procedure, our ME MaBiD and ME methods calculate the rejection threshold based on the asymptotic distribution χ 2 glyph[lscript] , whereas other methods perform permutation test or wild bootstrap.

Additional experimental comparisons

Table 5 presents the comparisons of type-I error for different methods. As can be seen, type-I error is limited about α = 0 . 05 for all compared methods, which shows the effectiveness of test power in experiments for different methods.

Figure 12 shows the experimental comparisons between Mahalanobis kernels and Gaussian kernels, and it is obvious that Mahalanobis kernels achieves higher test power with better performance by exploiting local regions and directional information. Figure 13 presents experimental comparisons of test power with different training sample sizes. The testing sample size is set as 200 for blob, 1000 for higgs, 200 for codrna and 150 for sea50. As can be seen, our ME MaBiD test takes better test powers empirically than other methods w.r.t different training sample sizes by incorporating correlation and directional information of two samples.

Our MEMaBiD

AutoMLTST

adult labor

Dataset

Ours MEMaBiD

ME

MMDAgg

MMD-D

C2ST-L

C2ST-S

AutoMLTST

blob

.043

±

.025

.064

±

.056

.061

±

.054

.064

±

.018

.037

±

.032

.004

±

.006

.019

±

.016

dna

.045

±

.036

.052

±

.068

.061

±

.035

.002

±

.004

.037

±

.025

.059

±

.054

.033

±

.034

agnostic

.047

±

.023

.049

±

.034

.051

±

.043

.002

±

.004

.042

±

.029

.063

±

.045

.046

±

.039

topo21

.033

±

.019

.049

±

.032

.038

±

.034

.049

±

.039

.019

±

.022

.051

±

.024

.038

±

.024

.046

±

.033

.033

±

.026

.040

±

.030

.045

±

.028

.022

±

.014

.061

±

.041

.052

±

.041

.033

±

.016

.021

±

.006

.013

±

.000

.030

±

.029

.046

±

.016

.033

±

.012

.033

±

.021

diamon

.042

±

.024

.029

±

.024

.017

±

.012

.046

±

.023

.013

±

.010

.021

±

.016

.033

±

.031

cifar

.015

±

.010

.012

±

.011

.025

±

.014

.078

±

.068

.021

±

.016

.027

±

.017

.020

±

.014

mnist

.043

±

.038

.013

±

.016

.041

±

.028

.092

±

.068

.032

±

.041

.057

±

.026

.049

±

.039

santan

.037

±

.031

.092

±

.043

.029

±

.006

.058

±

.019

.025

±

.018

.037

±

.027

.042

±

.036

codrna

.013

±

.010

.017

±

.016

.075

±

.080

.042

±

.028

.046

±

.047

.042

±

.012

.008

±

.012

sea50

.032

±

.027

.021

±

.012

.058

±

.026

.088

±

.059

.029

±

.041

.058

±

.056

.050

±

.047

adult

.024

±

.016

.021

±

.016

.029

±

.016

.038

±

.010

.033

±

.016

.033

±

.026

.062

±

.020

labor

.017

±

.016

.037

±

.010

.054

±

.059

.056

±

.029

.029

±

.024

.029

±

.026

.042

±

.026

poker

.029

±

.016

.013

±

.010

.054

±

.041

.078

±

.047

.029

±

.006

.058

±

.031

.050

±

.054

higgs

.046

±

.011

.024

±

.016

.043

±

.031

.066

±

.028

.068

±

.098

.048

±

.017

.034

±

.039

50

100

150

200

250

300

350

400

Sample size

0.1

0.4

0.7

1.0

Test power

blob

Mahalanobis kernel

Gaussian kernel

25

50

75

100

125

150

175

200

225

Sample size

0.1

0.4

0.7

1.0

Test power

higgs

Mahalanobis kernel

Gaussian kernel

25

50

75

100

125

150

175

200

225

Sample size

0.1

0.4

0.7

1.0

Test power

higgs

Mahalanobis kernel

Gaussian kernel

500

1000

1500

2000

2500

3000

3500

4000

Sample size

0.1

0.4

0.7

1.0

Test power

higgs

Mahalanobis kernel

Gaussian kernel

Figure 12:

The comparisons of test power vs sample size for Mahalanobis kernel and Gaussian kernel.

100

200

300

400

500

600

700

800

Training sample size

0.1

0.4

0.7

1.0

Test power

blob

Our method

ME

MMD-D

C2ST-L

C2ST-S

AutoMLTST

100

200

300

400

500

600

700

800

Training sample size

0.1

0.4

0.7

1.0

Test power

codrna

Our method

ME

MMD-D

C2ST-L

C2ST-S

AutoMLTST

100

200

300

400

500

600

700

800

Training sample size

0.1

0.4

0.7

1.0

Test power

sea50

Our method

ME

MMD-D

C2ST-L

C2ST-S

AutoMLTST

4000

6000

8000

10000 12000 14000 16000 18000

Training sample size

0.1

0.4

0.7

1.0

Test power

higgs

Our method

ME

MMD-D

C2ST-L

C2ST-S

AutoMLTST

Figure 13:

The comparisons of test power with different training sample sizes.

300

400

500

600

700

800

900

1000

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

Test power

blob

Masked p-value

Sidak

300

400

500

600

700

800

900

1000

Sample size

0.4

0.5

0.6

0.7

0.8

0.9

Test power

codrna

Masked p-value

Sidak

300

400

500

600

700

800

900

1000

Sample size

0.4

0.5

0.6

0.7

0.8

Test power

codrna

Masked p-value

Sidak

6000

8000

10000 12000 14000 16000 18000 20000

Sample size

0.1

0.4

0.7

1.0

Test power

higgs

Masked p-value

Sidak

Figure 14:

The comparisons of test power vs sample size for our bi-directional masked

p

-value and prior Sidak.

20

30

40

50

60

70

80

90

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

ξ

blob

ℓ

=2

ℓ

=6

ℓ

=10

ℓ

=14

ℓ

=18

50

150

250

350

450

550

650

750

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

ξ

codrna

ℓ

=2

ℓ

=6

ℓ

=10

ℓ

=14

ℓ

=18

20

30

40

50

60

70

80

90

Sample size

0.6

0.7

0.8

0.9

1.0

ξ

MNIST

ℓ

=2

ℓ

=6

ℓ

=10

ℓ

=14

ℓ

=18

125

375

625

875

1125

1375

1625

1875

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

ξ

higgs

ℓ

=2

ℓ

=6

ℓ

=10

ℓ

=14

ℓ

=18

Figure 14 presents experimental comparisons between our bi-directional masked p -value and previous Sidak [67]. As can be seen, our bi-directional masked p -value achieves higher test power by exploiting the significant level of local difference. Figure 15 analyzes the relationship between probability ξ = Pr[ sgn ( F glyph[latticetop] B L ′-1 B ( c ˆ X ′ B -c ˆ Y ′ B )] ≥ 1) and sample size, by considering different numbers of test locations glyph[lscript] ∈ { 2 , 6 , 10 , 14 , 18 } . As can be seen, ξ gradually increases as the sample size for different number of test locations, which shows the necessity of directional information in two-sample test.

Figure 16 analyzes the relationship between the number of local regions s and probability ξ = Pr[ sgn ( F glyph[latticetop] B L ′-1 B ( c ˆ X ′ B -c ˆ Y ′ B )] ≥ 1) . The probability ξ increases with s , and hence can effectively partition the space into local regions with different inference directions. Figure 17 presents

20

30

40

50

60

70

80

90

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

ξ

blob

s

=2 0

s

=2 1

s

=2 2

s

=2 3

50

75

100

125

150

175

200

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

ξ

codrna

s

=2 0

s

=2 1

s

=2 2

s

=2 3

20

30

40

50

60

70

80

90

Sample size

0.6

0.7

0.8

0.9

1.0

ξ

mnist

s

=2 0

s

=2 1

s

=2 2

s

=2 3

120

180

240

300

360

420

480

540

Sample size

0.5

0.6

0.7

0.8

0.9

1.0

ξ

higgs

s

=2 0

s

=2 1

s

=2 2

s

=2 3

Figure 16:

The comparisons of

ξ

for partition trees with

s

∈

[2

0

,

2

3

]

. The value of

ξ

increases as

s

increases.

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2.0

β

0.4

0.5

0.6

0.7

Test power

dna

N=125,

ξ

=.57

N=150,

ξ

=.59

N=200,

ξ

=.62

N=225,

ξ

=.65

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2.0

β

0.2

0.3

0.4

0.5

0.6

0.7

Test power

topo21

N=400,

ξ

=.57

N=450,

ξ

=.58

N=500,

ξ

=.61

N=550,

ξ

=.65

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2.0

β

0.85

0.90

0.95

1.00

Test power

blob

N=150,

ξ

=.85

N=200,

ξ

=.88

N=250,

ξ

=.96

N=300,

ξ

=.98

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2.0

β

0.0

0.2

0.4

0.6

0.8

Test power

higgs

N=1000,

ξ

=.90

N=1500,

ξ

=.96

N=2000,

ξ

=.98

N=2500,

ξ

=.99

additional experiments on the relationship between the optimal parameter β and the probability ξ = Pr[ F glyph[latticetop] L ′-1 ( c ˆ X ′ -c ˆ Y ′ ) ≥ 0] , as shown in Figure 6.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a bidirectional two-sample test that can address both positive and negative gaps from one sample to the other. The test is proposed based on Hotelling's statistic, and the direction information is encoded via the mean difference. By searching test locations and an adequate Mahalanobis kernel, and more importantly, by detecting the local inference, the test exhibits a higher testing power than the baseline.\n\nSTRENGTHS:\n**Originality**\\\nThe proposed test is novel. \n\n**quality**\\\nThe authors spent great effort delivering theorems and performing thorough experiments to validate the test. \n\n**clarity**\\\nThe writing and the structure of the paper are adequately clear, though it can be improved in some parts. \n\n**significance**\\\nThe proposed test has extensive applications in experimental design and observation studies, especially for the case where the global difference is small but the local difference is large.\n\nWEAKNESSES:\n**Tradeoff between local difference and overall sample complexity**\\\nThe testing power typically depends on the problem's difficulty and the sample complexity. An easier problem and higher sample complexity lead to a higher testing power for a two-sample test. One important idea of the proposed test is to shift the testing focus to the local region instead of global, and hence potentially change the original testing problem to an easier one. The price to pay is the sample complexity used to compute the statistic for a region is certainly smaller than the sample complexity used for a global test. The paper seems not to address the tradeoff between the decreasing problem's difficulty and the decreasing sample complexity used to compute the statistic. \n\n**Clarity**\\\nThe authors could use pictorial illustrations for presenting the region-splitting method; right now it is not super clear to me how the method work. In addition, the paper could include a problem formulation section to present what's the null and alternative hypothesis here; the introduction of the local test sort of blurs the typical setting for the two-sample test.\n\nQUESTIONS:\nDo you split the dataset into a training set and a test set, optimize the kernel parameters and test locations and split the region base on the training set, and then perform the two-sample test based on the test set? If so, how does the size of the training set impact the testing power, and do you have experimental results for different sizes of training sets?\n\nLIMITATIONS:\nThe following papers propose two-sample tests by detecting local differences inexplicitly. \n\n1. Duan, Boyan, Aaditya Ramdas, and Larry Wasserman. \"Interactive rank testing by betting.\" Conference on Causal Learning and Reasoning. PMLR. \\\n2. Li, Weizhi, et al. \"A label efficient two-sample test.\" Uncertainty in Artificial Intelligence. PMLR, 2022.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a two-sample test that explores local variations in distribution. The authors provide a motivation for this specific formulation by highlighting two practical scenarios in cosmology and biology. They proceed with a literature review and conclude that none of the existing approaches is fully satisfactory. Guided by the problem's requirements and their intuition they incorporate space partitions, Mahalanobis kernels, and bi-directional hypothesis testing in developing the new test. The proposed algorithm is experimentally compared with existing methods, and it is found to empirically outperform them.\n\nSTRENGTHS:\nThe paper is quite novel in both its motivation and the algorithm's construction. It is commendable that authors justify further specialization of two-sample tests ( \"it is necessary to take one more step to find and understand local significant differences, rather than only two-sample test\" ), by invoking two compelling practical problems. \n\nI am impressed by the unorthodox design of this test, which incorporates methods from various sub-fields to create an algorithm that addresses the requirements and and seems to perform well in practice.\n\nWEAKNESSES:\nIt is unclear which elements of the design (bi-directional p-values, tree partition, kernel design) contribute to overall performance of the algorithm. \n\nClarity of the writeup goes down starting from the \"Construction of partition tree\" section. A few examples:\n\n* It is not explained why authors need masked p-value.\n* Why is it important to guarantee equal probabilities in partitioned rectangle regions?\n* Theorem 5 makes assumptions that I don't understand \".. if p-values are mutually independent for regions without difference, and are independent to regions with difference\". Does your test control FWER?\n\nFinally,  Bi-directional hypothesis for testing looks like designing non-centered rejection regions for the L^{-1}(c_{\\hat x} - c_{\\hat y}) (you don't have to that L2 norm of the embedding)\n\n\nedit: I have read the rebuttal, and now I have less confidence that the clarity of the paper will improve. I am also more worried that the lack of clarity hides technical flaws. Moving  to 2.\n\nQUESTIONS:\nI suggest authors improve the quality of the writeup.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn recent years, the focus on two-sample testing has significantly increased due to its wide range of practical applications. This study advances this exploration by highlighting local significant differences in two-sample testing. The authors introduce the ME, a robust test for two-sample testing that leverages local information through multiple Mahalanobis kernels and proposes a bi-directional hypothesis for testing. To pinpoint local significant differences, the embedding space is initially partitioned into several rectangular regions using a novel splitting criterion tied to test power and data correlation. Following this, local significant differences are identified using the bi-directional masked-value in conjunction with the ME test. The authors provide theoretical guarantees for their ME test, including lower bounds on test power, and effectively control the familywise error rate when investigating local significant differences. The effectiveness of the proposed methods in two-sample testing and the examination of local significant differences are then validated through extensive experiments.\n\nSTRENGTHS:\n1. This paper focuses on an important problem. Two-sample testing techniques have an important role in detecting distribution changes. T\n\n2. The proposed method focuses on the local difference between two distributions, which is a promising research direction based on existing two-sample testing techniques. Recent testing techniques focus on learning good representations of data via maxing test power (Liu et al., 2020), which is another way to look at different areas with different weights but more flexible (bringing some issues). This paper proposes an effective way to implement this kind of technique.\n\n3. The experiment is solid, verifying that the proposed method is good and effective.\n\nWEAKNESSES:\n1. There should be more discussion in the introduction. To show the difference between the deep-kernel-based method and your proposed method. I can understand the merit of the proposed method, but more discussions are needed in this paper.\n\n2. The motivation to use Mahalanobis kernels is still not clear. Can we have more choices? What is your selection standard for kernels?\n\n3. Procedures of permutation test should be introduced in preliminary. it is better for readers to understand the procedures of two-sample testing.\n\nQUESTIONS:\nSee the weakness above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the exploration of local significant differences in the context of two-sample tests. The authors propose a new test called ME_{MaBiD}, which utilizes multiple Mahalanobis kernels and introduces a bi-directional hypothesis to exploit local information. The embedding space is partitioned into rectangle regions based on a new splitting criterion, and local significant differences are identified using the bi-directional masked p-value and the ME_{MaBiD} test. The paper provides theoretical guarantees for the ME_{MaBiD} test, including lower bounds on test power and control of familywise error rate. Extensive experiments validate the effectiveness and efficiency of the proposed methods for two-sample testing and the exploration of local significant differences.\n\nSTRENGTHS:\nNovel Approach: The paper introduces a new test, ME_{MaBiD}, which offers a novel approach to exploring local significant differences in two-sample tests. The utilization of multiple Mahalanobis kernels and the introduction of a bi-directional hypothesis provide a unique perspective for analyzing local information.\n\nRigorous Theoretical Analysis: The paper provides theoretical guarantees for the MEMaBiD test, including the asymptotic distribution of the proposed statistic with Mahalanobis kernels, lower bounds on test power, and control of familywise error rate. This demonstrates a strong foundation for the proposed method and enhances its credibility.\n\nExploration of Local Significant Differences: The paper addresses the important aspect of identifying local significant differences, going beyond traditional two-sample testing. By partitioning the embedding space into rectangle regions and introducing the bi-directional masked p-value, the paper offers a comprehensive framework for discovering and understanding local variations.\n\nExtensive Experimental Validation: The paper conducts extensive experiments to validate the effectiveness and efficiency of the proposed methods. The superior performance of the ME_{MaBiD} test on various datasets for two-sample testing and the exploration of local significant differences demonstrates the practical value and applicability of the proposed approach.\n\n------------------------after rebuttal------------------------\nI'm leaning towards improving my evaluation due to the authors' commitment to enhancing clarity, which directly addresses my concern. Their responses have clarified the unique contributions of their work, particularly in exploring local significant differences, while their explanations on theoretical foundations underscore the method's credibility.\n\nWEAKNESSES:\nIn general, I think the weakness lies mainly in clarity. It fails to give a formal definition of the input and output of the problem and defined further what is counted as local or not. Without such a formal definition, it would take more time to go into the details of the paper. Another issue with clarity is that the paper does not provide enough reasoning on why the proposed techniques are working towards the goal of the paper. For example, in Lines 67-68, why do Mahalanobis kernels exploit more information than the Gaussian kernels? In Eq.7 why the two directions are necessary, but they are not used in the exploration of local significance differences? For Lemma 1, would the learned \\v_j satisfy the condition of drawn i.i.d. from some distribution? Why the alternative hypotheses H1 and H2 are important and practical? These factors are important to accurately evaluate the contribution of the paper but are not sufficiently discussed. Without such discussion,  the paper's main contributions seem to replace the Gaussian kernel with the Mahalanobis kernel and one direction test by two-directions, which is novel but may not be significant enough considering the motivation.\n\nQUESTIONS:\nIn addition to what has been asked in the Weaknesses part. the type-I error rate in Theorem 4, and M_1, \\ldots, M_\\ell in Eq 5 are not defined. For the Construction of partition tree step one, is the leaf node randomly selected, or just select the largest? The paper seems to go with both.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}