On the Exploration of Local Significant Differences For Two-Sample Test
Authors: Zhijian Zhou
Jie Ni
Jia-He Yao
Wei Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: two-sample test
local significant difference
directional information

Abstract: Recent years have witnessed increasing attentions on two-sample test with diverse real applications, while this work takes one more step on the exploration of local significant differences for two-sample test. We propose the ME$_\text{MaBiD}$, an effective test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. On the exploration of local significant differences, we first partition the embedding space into several rectangle regions via a new splitting criterion, which is relevant to test power and data correlation. We then explore local significant differences based on our bi-directional masked $p$-value together with the ME$_\text{MaBiD}$ test. Theoretically, we present the asymptotic distribution and lower bounds of test power for our ME$_\text{MaBiD}$ test, and control the familywise error rate on the exploration of local significant differences. We finally conduct extensive experiments  to validate the effectiveness of our proposed methods on two-sample test and the exploration of local significant differences.

On the Exploration of Local Significant Differences For Two-Sample Test
Zhi-Jian Zhou, Jie Ni, Jia-He Yao, Wei Gao
National Key Laboratory for Novel Software Technology. Nanjing University, China School of Artificial Intelligence, Nanjing University, China [zhouzj, nij, yaojh, gaow landa.nju.eddu.cn
Abstract
Recent years have witnessed increasing attentions on two-sample test with diverse real applications, while this work takes one more step on the exploration of local significant differences for two-sample test. We propose the MEMBBID, an effective test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. On the exploration of local significant differences, we first partition the embedding space into several rectangle regions via a new splitting criterion, which is relevant to test power and data correlation. We then explore local significant differences based on our bi-directional masked p-value together with the ME test. Theoretically, present the asymptotic distribution and lower bounds of test power for MEMBRD test, and control the familywise error rate on the exploration of local significant differences. We finally conduct extensive experiments validate the effectiveness of our proposed methods on two-sample test and the exploration local significant differences.
Introduction
Two-sample test has attracted much attention with diverse applications such cancer detection [1], distribution-shifft detection [2), generative modeling [3, 4], etc. The basic problem is to assess whether two i.i.d. samples drawn from the same distribution. Various kemel-hased methods have been developed for two-sample test such as Maximum Mean Discrepancy (MMD) [5-8] and Mean Embedding (ME) [9-11]. Another relevant approach is construct binary classifier and assess two samples according to classification performance [12-20]. For an overview of two-sample test, we refer survey [21, and references therein].
In many real applications, however, it is necessary to take one more step to explore and understand local significant differences, rather than only two-sample test. For example, scientific problem in galaxy morphology is to identify some local regions significant differences between two kinds of galaxies, which is important discover galaxy formation and evolution history [22]. On the analysis of mass cytometry data in cell biology, researchers are always interested in finding local regions of significantly different abundance between disease and healthy samples [23].
Several attempts have been made to explore local significant differences in the past years. feasible solution is to partition space into several regions, and identify significantly different regions according to their cardinalities of samples [24-27]. Another relevant work is to identify local significant differences by estimating kernel densities [28, 29] and conditional probabilities [30]. Generally, it not easy to deal with complex data by simply counting cardinalities of samples, regardless of data intrinsic correlations, and it is also difficult to make accurate estimation of kernel densities and conditional probabilities without sufficient data, especially for many regions with finite samples.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

test locations
elemmnts X
A elements in
0
Contour Mahalanobis kernel
Contour Gaussian kernel
Figure 1: An illustration of different contours of Mahalnobis and Gaussian kernels for two-sample test.
This work presents new two-sample test from local and directional information, and further explore local significant differences. The main contributions can be summarized as follows:
We propose the effective MEMABID test for two-sample testing, and the basic idea is to exploit local information by multiple Mahalanobis kernels and introduce bi-directional hypothesis for testing. Intuitively, Mahalanobis kernels are more flexible to exploit local differences from neighborhoods and feature maps, and the bi-directional hypothesis is beneficial to improve the sensitivity of two-sample test with proper parameter adaptation.
We partition the embedding space into several rectangle regions based on a new splitting criterion, which is relevant test power and data correlation. We introduce the bi-directional masked p-value for each rectangle region, and finally explore local regions with significant difference based our bi-directional masked p-value together with the MEMaBID test.
We present theoretical guarantees for our MEMаBID test via the asymptotic distribution, as well as the lower bounds on the test power for our test. We also present the upper bounds on familywise error rate for our exploration of local significant differences.
We conduct extensive experiments to validate the effectiveness and efficiency of our methods. Specifically, our methods achieve better performance on most datasets for two-sample test and exploring local significant differences, along with comparable or smaller running time.
The rest of this work organized as follows: Section 2 presents our MEMBBD test. Section 3 explores local significant differences. Section 4 conducts extensive experiments, and Section 5 concludes with future work. All technical proofs are given in Appendix A.
2 Our MEMaBiD Test for Two-Sample Testing
Let and Q denote two (unknown) Borel probability measures over an instance space X' C Rd and X {æx; 1 and Y {y>} are two i.i.d. samples from P and Q. respectively. The goal of two-sample test to assess whether X and Y are drawn from the same distribution; in other words, Q from two samples and Y.
we aim to assess whether
We introduce some necessary notations used in this work. Write [r] = (1,2, 1..2.....77.. , for integer 2, and |A| denotes the cardinality of set A. Let La be the identity matrix of size d d. For vector a = [a1, Q2 ... ad), denote by sgn( a) =
... sgg((az) with sgn(a;) = a,/ (a,) 2 be the x2 distribution with degree of freedom, well as the p-value function x2( (-). Denote by Xa,a the c-quantile of distribution X² for E (0, 1).
for a; # 0; otherwise, sgn(a,) 0. Let xe
Learning multiple Mahalanohis kernels via maximizing test power in training
Following ME test [9, 10], we begin with set of test locations V (vi, U2. ....... 0z} C X' to construct discriminative features. For every vz E V. we introduce a Mahalanobis kernel as follows:
(x, v,) exp (- T.)T (1 22)/2n?) for 7 and positive definite matrix M,. (1)
Here, we propose multiple Mahalanobis kernels for two-sample test, which is motivated from multiple kernel learning [31, 32) and Mahalanobis distance [33-35]. The advantage of multiple Mahalanobis kernels is to exploit intrinsic structures and correlations from different directions and regions, and adjust geometrical distribution of data so to enlarge the distance between different samples [36, 37].

---

This is different from previous Gaussian kernel (2 u.) = exp( 1 - -v,1 -2,)²/2>²) [10, 38), which deals with every direction isotropically without difference. Figure 1 presents an illustration of different contours of Mahalnobis and Gaussian kernels for two-sample test. As we can see, Mahalanobis kernels are more flexible to exploit different directional information than Gaussian kernels. Our work is also different from previous deep kernel approaches [4, 39], which train single one deep neural network combined with Gaussian kernel for variations in distribution smoothness and shape.
We then embed each element in X = { and Y = {リ into an é-dimensional space as i (K1 (mi 11), .. Ka, uz)) T and ÿn = (Na (vj, 01), .. Re(U,, vz)) respectively (2)
Denote by X = { and γ = {ūン}ジ=1 We define the pooled covariance matrix as
(Ø; cg (0:
( - cp (ū: cc) ela, m
Ex,
m + n-
where Cx = Σ 1== 2./m and cp = Ej== Ui/n, and eI, is introduced to guarantee the positive definiteness for small constant 0. We consider the Hotelling T statistic, as in [40-42],
T(X mn(cc cy) 1 - -m n).
(4)
Test power is the probability of correctly identifying two different samples. Maximizing T(X, Y) is essentially equivalent maximizing a lower bound of test power [10, 11], and we learn test locations and Mahalanobis kernels as follows:
M,, ... Mr, 71. ,e) E arg max (T(X,
(5)
We take gradient method [43] to solve the above optimization, as done by Jitkrittum et al. [10], and the details are presented Appendix B.
We decompose Σx = LL via the Schur method [44] to remove feature correlations, and it follows
T(X,8)= mn m + n (cx - cx) 22 - cy) m + n
mn
L - 1 L
Hence, T(X. essentially measures the difference between two samples via the Ln- of vector L L L L--e.-.. Cp. We further exploit their inference direction, defined by
F sgn (L (L='ex L (x) +-1,0,+113.
(6)
Bi-directional hypothesis for testing
Let Tp and @ be the corresponding embedding distributions from the original P and Q respectively.
Denote by Ue = Exart [ and fa = E Eg We consider the following null hypothesis
Ho: Mp 1
The null hypothesis Ho can be used to test whether P = Q by the following lemma:
Lemma 1. We have uh = pby iff 1 Q. bounded Mahalanobis kernels {ng)i=1 and for test locations {u,}s drawn i...d from a absolutely-commimuous distribution w.r.t. Lebesgue measure.
Let X (= and & \ denote embedding testing samples. We make similar Schur decomposition y L'L', and calculate testing statistic T(X', Y') according to Eqn. (4). Based on Lemma 1, we can present the asymptotic distribution of statistic T(X', Y") as follows: Theorem 2. The testing statistic T(X', Y') is almost surely asymptotically distributed as Xa if
P Q: otherwise, (T('', Y")) 0 as m'n' /mm + n') co.
From Theorem 2, we propose the bi-directional hypothesis, by considering inference direction,
(x 80) < sa]
for F pTL'-1(cg- -cy.) 2 0
h(X', 84 =
(7)
xT 87) < 3)a]
for F FT - -c.:) <0,
3

---

X
elements in
x
Adaptive rejection region according to difffrent datasets
xf
Fixed rejection region
x
Fixed rejection region
F- (-1,1)
4
[0,0]
[0.0]
fixed (1.1)
b
Two samples
Our di-directional hypathesis
The non-directional hypothesis":
The one-directional hypothesis
Figure illustration of different rejection regions on two samples our bi-directional hypothesis and
previous non/one-directional hypothesis. Our rejection region can be adaptive according to different datasets.
where a E (0,1) is the significance level of hypothesis test, and / E [1,2] is an adaptive parameter.
Our bi-directional hypothesis is essentially about designing rejection region of null hypothesis [45]. How to design an efficient rejection region an interesting problem from the early work [46], and some techniques has been developed for selecting rejection regions [47-49]. We consider the most discriminative directions and
- F in our bi-directional hypothesis, which could improve the sensitivity for two-sample test by selecting appropriate parameters according different datasets.
Our bi- bi-directional hypothesis can be viewed generalization of previous hypotheses. that is,
By setting 3 = 1, our test has been the non-directional hypothesis [9, 11, 50] regardless of direction information, which is also referred to as two-sided/tailed hypothesis [49, 51, 52];
By setting / 2 and F (1, 1, 1)T, our test has become one-directional hypothesis [53-55], which also referred to as one-sided/tailed hypothesis [52, 56].
Notice that previous con/one-direction hypotheses fix the structures of rejection region for a given significance level a, whereas our bi-directional hypothesis could adjust rejection region according to inference direction F w.r.t. different datasets, which can be illustrated in Figure 2. Here, we consider an illustrative dataset, and our bi-directional hypothesis gives the rejection region adaptive to dataset, which could yield higher test power in two-sample test, as shown in Figure (in Section 4).
We can also present some distribution and probability information for F (cx cy,) 2 in Eqn. (7). Denote by L E ximmm Ginbro' [L'] and Pr[F (c -y.) 2 0j. We have Lemma 3. For inference direction F in Eqn. (6) and for embedding samples X' and Y', we have
L' 1'x -cy)) ~ N (H4p 13)) wowwwra -Ie) and sgn( T L' - cq.)) ~ TP(E)
with w = m'n' /mm' (m'++n)) and = 1-0 V P.2 (1: -+)) Here, ((-) is the distribution function of standard Gaussian, and TP({) denotes a distribution over {-1, +1] with probability on the selection of +1.
The selection of parameter is highly positive-relevant the probability &. This is because larger & implies larger difference between two samples in the inference direction F, and we should select larger 3 enlarge the rejection region and improve the sensitivity of dataset. Figure (in Section 4) shows such positive relevance between the optimal parameter 3 and probability & empirically.
We finally present theoretical analysis on test power and type-I error of our bi-directional hypothesis. Let f(x: 71, n2, X) the density function of noncentral F-distribution with ni and n2 degrees of freedom and non-centrality parameter X, and denote by Fny.ng.a the a-quantile of central F- distribution with n and n2 degrees of freedom for a E (0, 1). We define the following probability, from the work of [57].
f Pnyingia
q(1 X,
f(x n1, n2, dr
Theorem 4. For our bi-directional hypothesis
((',,'') the test power can be lower bounded by
q(e.v v - e, X, Ba): + +b
(- (w/ 1/2 F Ie - m&))

---

Algorithm Construction of partition tree
Input:
Two embeddding samples X and Y Output: Partition tree T Initialize: Tree 7 with only a root [0, 1]e
1: for =,....,
do
2: Randomly select one of the leaf node B of the largest |X n
3: Initialize the best splitting feature j = 0
4: for jo..... do
5: Calculate the median value in j-th dimension Tj median : E; X B)
6:
Split two samples NB and VBS into X17, Vay and X Xgg, Veg. according to Tj
7:
Calculate T(X 8/: VEy) and T(X By Vey) based on Eqn. (4)
8: Update j" if B/ EE (
Bp B T(x BJ" BB (T( B BB²))
9: end for
10: Update node B with two children By and B w.r.t the splitting feature j" and position Tj-
end for
if F L-1
> F L 13g and the test power can also be lower bounded by
ale,v (2,, - +1
-
((x,,a)))) (w/l) L - ma)
if FF 1. F L p,0 u and the type-I error rate is equal to if u4 Pe Here, mln'/(mn +n'), V= ml + n' and X wil 1 (ue m
This theorem presents lower bounds on test power, and the performance of the statistical test is maintained under the general condition. Notice that the type-I error in our hypothesis test is controlled only by the significant level regardless of different and B.
We call test MEMaBID test because of multiple Mahalanobis kernels in training and our bi- directional hypothesis in testing.
Explore Local Significant Differences for Two-sample Test
On the exploration of local significant difference, most previous studies [24-28] partition the instance space into several regions, and then exploit the difference on each region from two samples. Motivated from pólya tree method [26, 58), we first partition the embedding instance space with new splitting criterion, which is relevant to test power and data correlations. We then exploit local regions (i.e., leaves nodes of partition tree) with significant difference.
Partition of the embedding instance space
Our partition tree is constructed iteratively as follows: We initiate the tree root with embedding space (0, 1).. In each iteration, each node associated with a rectangle region, and all leaves constitute partition of embedding instance space. The following procedure is repeated
iterations (s 2 2):
Randomly select leaf node, denote by B, uniformly over leaf nodes of the largest |X n B).
Let T median ( b; E X B3 for j E [f]. We select the best splitting feature
j" arg max ( Bp Bq) x T(X By Yez)
with
X B/ X, y = Bi nr and Y B n. Here, B/ and BC are left and right children of B w.r.t. the j-th splitting feature and splitting position Tj. respectively, and ...)) is defined by Eqn. (4).
Select the splitting position T,- median C, x 8nnne n).
5

---

Table 1: Datasets
Dataset
# Tnst.
# Fent.
Dataset
# Inst.
Feat.
Dataset
# Inst.
Feat.
Dataset
a Inst.
# Feat.
dna
3,186
180
kropt
27,705
6
200,000
200
adult
1,000,000
agnos
3,468
970
diamon
53,940
9
codma
487.867
labor
1,000,000
togo21
8,885
266
cifar 10
60,000
3072
blob
1,000,000
poker
1,025,010
har
10,299
561
mnist
70,000
784
sea50
1,000,000
higgs
11,000,000
We finally get the partition with leaf nodes, associated with rectangle regions B1, B2,,.,,.. B,. Algorithm presents the detailed description on tree construction and rectangle region splitting.
We take the statistic (.::) :) as splitting criterion, relevant to test power and data correlations, and it helpful to exploit local significant difference directly. We also adopt the median splitting position with equal probabilities on partitioned regions, i.e., balanced examples for each partition region, and this could yield better performance than regular grids, as shown empirically in [59, 60].
Our partition tree is different from previous p-value histogram based on Chi-square test [61, 62], where the difference is measured by cardinalities of elements two samples over a local rectangle region, Our splitting criterion is also different from that of previous decision trees [63-66], which consider some information-the coormation--hhoreticc theoretic criterions such as entropy, Gini index, information gain, etc. In comparisons, our statistic ((,,) )) is more essential to reflect the test power for two-sample test.
For each rectangle region B,, let Cxs, and Cye, be the means of As. B; n X and Yss B, ny respectively. We make similar Schur decomposition LB, Lg, Xxx Ss. for covariance matrix E Ys, and introduce the local inference direction for each B; as follow:
FB, = sgn LF LE! Cye, E (-1,0,+1]
For different rectangle regions, we could have different or even contrary inference directions, which helpful to exploit local differences from distributional shapes of two samples.
Exploration of local significant differences
For testing embedding samples X' and Ý', denote by X/. B, X and Y/g. B, n ý with their respective means Cxg, and Co2 for each rectangle region B,, and calculate testing statistic
TB, -(XX, E.) by Eqn. (4).
We propose the new bi-directional masked p-value for each rectangle region B, as follows:
min
x2(Tn,)) B 1 8p-
p- (1- (1xx (Ta,))
for F 1- (cxg, -c6 )20
g( (88,8186))
(10)
min
(Tr,) 2-8
p. (1-x2(TTe.
for Fg. L's, (cx cgp ) <0
(2-2-9)p.
where L/g,1 is from Schur decomposition Σ Σx, 86. L's LB p. E (0, 1) is parameter
significance level, and 9 is an adaptive parameter. Here, we also consider two discriminative directions FB, and FB, on each rectangle region B,, which is different from previous masked p-value [67] without directional information.
Our bi-directional masked p-value directly reflects the significant level of local difference when there a significant difference in local B,, similarly to [67]. The smaller the bi-directional masked p-value, the more significant the local difference. On the other hand, the bi-directional masked p-value is random number with uniform distribution over (0, p.) when there is no significant difference in B,, since (TB,) follows uniform distribution in such case [68].
Based on such recognition, we resort rectangle regions as B:1 B Bis) according to their bi-directional masked p- value,, i.e.,
g( (8 B,yy Vo,,) S g( g(8
VE ..... 508 B,, Von))

---

Table 2: Comparisons of test powers (mean:Lssd) two-sample Bold denotes the highest in per row.
Dataset
Our MEMABID
ME
MMDAgg
MMD-D
C2ST-L
C2ST-S
AutoMLTST
blob
.985
823+.000
9354.0012
.63++.010
.9724.078
.946.007
.980+.029
dna
.717
5364.0059
.559.0770
.28840006
.699:.028
.505.004
.603+.085
agnos
812
.602+.033
.7794.046
.7344.006
.7424.012
.679..051
.6322.077
topo21
192.1.006
.5261.058
1.051.077
.633. 633.1.062
16791.046
5171.006
.59111.006
.858 165
81611.015
814.1.026
72881.064
.761 1.093
.738.1.06
.740.1.058
kropt
.992
8851.0027
.71 1..24
916.1.066
.9461.013
.9291.031
19.11
diamon
.66
.697..068
6766+.047
.755+.056
.747
.727=.076
.81
cifar
.93
859
8664.091
878 090
834 099
.798
082:+.086
mnist
.985
926 056
9324.068
.722t.081
9691.002
.30+20099
.633t.074
santan
1.00::0000
896:..000
1.00#.0000
.887#.021
.11#.0044
.850#.021
.54
codrna
1.00=00000
946:..085
926:t.037
.914t.076
1.00=.000
1.00:1.000
876:t.067
sea50
.913
.933..018
982+.012
.93:..018
998=
.970= .53
989#.009
adult
.9911.002
1.34
967...029
998.1.072
.7.1
.854..0088
.992,1.006
labor
.992.1012
.8071.078
18810000
.30.1.093
.756.10099
...11.031
1.00.1.000
poker
:821:.079
1.191.009
.712.1.033
.701
..43
.731 1.052
.8321.0048
higgs :779:.024
81881
13811.047
953:1.055
.6881.033
.933 0303
.9691.030
Average 9009=.026
:795:.053
.8594.039
884-4.050
.842:.052
806:+.039
.833+.043
We then take our bi-directional hypothesis h(X',, V%, with parameter (3 and and finally get the local regions with significant differences as
p. as in Eqn. (7),
BBy is and M(X %B,)) ) =
where
In(1 a.) -
arg max t (i [t]: M(X Bay Y = +15 In p.) *E[+]
(11)
Here, P, is selected as in Eqn. (10), and ar E [p=, 1) is a parameter to control the probability of mis-identifying least one rectangle region without significant difference, also called familywise error rate [69-72]. We present theoretical analysis for familywise error rate as follows:
Theorem 5. For our exploration, the familywise error rate is upper bounded by a., if 1) the p-values of local regions without differences mutually independent; and 2) the p-values local regions with differences are independent to those p-values of local regions without differences.
Our method is different from previous space partition methods of trees clusters [24-27, 60), where the splitting criterion is taken as the cardinalities of samples in each region. Duong [28] partitioned and searched local regions from the estimated density function, and Kim et al. [30] identified local regions by clustering data samples from the estimated conditional probabilities. It is not easy to make accurate estimation for density and conditional probabilities without sufficient data, particularly for multiple small regions. Other relevant studies detected local differences implicitly based on interactive rank test [73] or learned classifier [74].
4 Experiments
We conduct experiments on 16 datasets' as summarized in Table 1. Most dataset have been studied in previous wwo-sample test, and features have been scaled [o. 1] for all datasets. All experiments are performed with Python on nodes of a computational cluster with single CPU (Intel Core i9- 19-10900X 3.7GHz) and single GPU (GeForce RTX 2080 Ti), running Ubuntu with 128GB main memory.
Experimental comparisons for two-sample test
We compare our MEMBBID with the state-of-the-art approaches on two-sample test as follows:
Dataset blob is downloaded from github. httiub..oom/fngglu//WXKKK fOTTTT and other datasets downloaded from www.openml.org.
7

---

Table 3: Comparisons of density differences (mean-istd) the exploration of local significant differences, and the bold denotes the highest mean per row.
Dataset blob
Our method
FDG
KPRIM
MRS
MMDT
BTLDD
TEAM
.45
902:.0075
8792
8494.075
.09 091
.932
.877:0067
diamon
.974:.054
852:.010
.85 089
8764.066
867 104
951
.474.002
codrna
2229
93611.037
876120611
9664.005
.884
.05
88631.036
sea50
-977
995
.933=0074
.28
.985=,045
.892#.059
.9444.065
adult
.953=.048
.862+2004
8388.109
.911+.085
.927=.101
.875+.074
.880=.070
labor
:959=.022
8944.080
905
.900+0037
.911#.098
.922#.048
.9324.067
poker
.945 1.030
1.01
.8821.027
.25.1.023
927.1.057
.894.1022
.884.1.044
higgs
.946..00
932.1.011
.9181.001
.940.1.000
.927.1.02
.9261.027
.937..002
Average .59 047
9074.044
..11
.122..049
.17
.912+.050
908:2.09
ME: Mean Embeddings over multiple test locations and single Gaussian kernel [9, 10];
MMD-D: Maximum Mean Discrepancy based on a Deep kernel [39];
MMDAgg: Maximum Mean Discrepancy with Aggregating multiple Gaussian kernels [75];
C2ST-S: Train a binary classification network and test its accuracy on a hold-out set [13];
C2ST-L: Train a binary classification network with a statistic about class probabilities [14, 18];
AutoMLTST: Train a binary classifier based on AutoML method with a statistic as C2ST-L [19].
Following [39, 76]. we train subset of each available data, and test on 100 random subsets from training and testing. We repeat such process 10
the remaining dataset, and the ratio is set as 4 given in Appendix C. For our MEMaBID,
times for each dataset. More details and take 5-fold cross validation to select / E [1 0.2 : 2]. We limit the cardinality of test locations within 20 for ME and MEMaBID in [9-11], and optimization parameters of Eqn. (5) is presented in Appendix C. We take parameter settings for other methods as their respective inferences.
set a 0.05
Table 2 summarizes the average of test powers and standard deviations. It is evident that our MEMABI takes better performance than ME and MMDAgg, because they both take Gaussian kernels with isotropic scale, and ignore the distributional differences from different directions. Our method is still better than MMD-D with deep kernel, and a reason that multiple Mahalanobis kernels are more flexible than a deep kernel to capture local difference from multiple neighborhoods and directions.
From Table 2, is also observed that our MEMABID outperforms three classifier- methods C2ST-S, C2ST-L and AutoML expect for datasets labor and poker, since those methods focus merely on the prediction information from outputs of classifiers, rather than local and directional information among data samples. For datasets labor and poker, AutoML generates new features automatically from the original mixture of continuous and symbolic features, and thus achieves better performance.
We further compare the average running time (in seconds) for different methods on two-sample test, shown in Figure 3. As expected, ME takes the least running time since considers only one Gaussian kernel, yet with the smallest average of test powers in Table 2. Our MEMARID method takes smaller and comparable running time in contrast to other methods since method takes relatively smaller time on training Mahalanobis kernels without permutation test in the testing process.
Experiments on the exploration of local significant differences
We compare with the state-of-the-arr approaches on exploring local significant difference as follows:
FDG: Partition space by probability binning and compare cardinalities of two samples [24];
K-PRIM: Partition space by patient rule induction and estimate kernel density differences [28];
MRS: Partition space by pólya tree and measure difference via Binomial distributions [26];
TEAM: Partition space by data variance and measure difference via Binomial distributions [27];
BTLDD: Estimate conditional probabilities of two samples and cluster data via difference [77];
MMDT: Partition space into equal grids and test density difference via Welch's 's statistic [29].

---

MMDAgg
AutoMITST
10a
Running (seconds)
104
blob
dna
agnos
topo21
har
kropt
diamon
cifar
mnist
santan
codma
sea50
adult
labor
paker
higgs
Datasets
Figure 3: Comparisons of running time for different methods on two-sample test. Note that y-axis is in log-scale.
Mean of mide Sample X")
Mean mfnist (Sample
9
3
Fiko-miia (Sumple
θ
2
betwern Fakoernist
Difierence betwern Fakoomnie
(8.93)
(0.91 (0.90) (0) partition tree with ; -8 rectangle regions (left)
(0.91)
(0.94) (MDT
(b) partilon tree wilh 16 rectangle (right)
Figure 4: Visualization of the local significant differences with partition tree of s 8 (left) and 16 (right) rectangle regions. The bigger the values, the larger the density difference.
For fair comparisons with some methods of density estimations, we select eight large datasets: diamond, codrna, blob, sea50, adult, labor, poker and higgs, whose instance numbers 50,000 and feature numbers are smaller than 20. We randomly select 10, 000 instances during the exploration of local significant differences. For the BTLDD method, we the percentage of selected samples as 6.25% for local regions in a cluster, while for other methods, we partition the space into 64 rectangle regions, and identify 4 rectangle regions with the most local differences.
than
We take density differences [78] between two samples in local region as an evaluation measure for local significant differences, and follow the works of [79, 80] based on k-NN density estimator with = 20. We calculate the difference between two estimated density functions in an identified local region, and normalize the returned values into [0, 1].
Table 3 summarizes the average of density differences and standard deviations. As can seen, our method takes better explorations on local significant difference than FDG, MRS and TEAM, since those methods simply take cardinalities samples to measure significant difference. Our exploration is still better than K-PRIM, BTLDD and MMDT except for dataset sea50, because those methods estimate density or conditional probabilities from two samples, and it is not easy to make accurate estimation without sufficient data, especially for small regions with finite samples. MMDT could estimate density function well on dataset sea50, and achieves better exploration via Welch's ss statistic.
We further visualize the local significant differences on mnist (sample X) and Fake-mnist (sample Y) in Figure 4. Here, we take partition trees with can be seen, our method achieves the largest local differences via new splitting criterion for partition tree, and our proposed MEMBBID test and bi-directional masked p-vauues.
(left) and 16 (right) rectangle regions. As
Parameter analysis
We now present some parameter influence for our MEMABID test and the exploration of local significant difference. We only present the results on four datasets due to pages limit, the trends are similar on other datasets, and more results can be found in Appendix D.

---

16b
Tott power
THst
iedimcore bortes
Somple she
Sansle coe
Sample 50e
Somple dae
Figure The comparisons of test power vs sample size for our bi-directional hypothesis and previous one/non-
directional hypothesis.
layn
The q
0
10g
leg my
Figure 6: The relationship between the optimal adaptive parameter and probability & for our MEMABID.
8-28
25
Sancie dhs
Sarple dee
Sangle cbe
Sample dhe
Figure 7: The Iype-I error limited about a 0.05 w.r. different for our MEMARID.
CITY
Samale 9i20
Semple 5400
bample siz
Figure 8: The FWER is limited about a. 0.05 w.r.t. different s for exploring local significant difference.
Figure 5 illustrates the test power versus sample size for our bi-directional hypothesis, non- con-directional hypothesis and one- directional hypothesis. As can be seen, our bi-directional hypothesis achieves higher test power by considering the inference and its contrary direction and adaptive parameter selection. Figure 6 exploits the relationship between the optimal parameter 3 and the probability ç = Pr[ L -' (cx (cx
cp.) 2 0) for our MEMMBI method. We can easily find the positive relevance between 3 and ę: the larger the probability the larger the optimal parameter /3.
Figure 7 indicates that the lype-I error is limited about a = 0.05 for different 33 in our experiments, as shown in Theorem 4, and thus our method could effectively control the rate of falsely reject the null hypothesis, which empirically verify the trustworthiness of our ME test. Figure 8 empirically shows the familywise error rate is limited about a, 0.05 for different number of local regions s; therefore, our exploring method could control the rate of incorrectly exploiting the local regions with significant difference, and this is nicely in accordance with Theorem 5.
5 Conclusion
This work takes one more step on the exploration of local significant differences. We propose the MEMaBID test by exploiting local information from multiple Mahalanobis kernels and introducing bi-directional hypothesis for testing. We partition embedding space via new splitting criterion, and then identify local significant differences based on our bi-directional masked p-value and MEMABID test. We verify the effectiveness of our proposed methods both theoretically and empirically. An interesting work is to explore other local and directional information for local significant differences.
Acknowledgments and Disclosure of Funding
The authors want to thank the reviewers for their helpful comments and suggestions. This research was supported by National Key R&D Program of China (2021ZD01 12802), NSFC (61921008 1006, 62376119) and CAAl-Huawei MindSpore Open Fund. W. Gao is corresponding author of this paper.
10

---

References
[1] S. Shell, S.-M. Park, A. R. Radjabi, R. Schickel, E. O. Kistner, D. A. Jewell, C. Feig, Lengyel, and M. E. Peter. Let-7 expression defines two differentiation stages of cancer. National Academy of Sciences,
2007.
[2] G. Boracchi, D. Carrera, C. Cervellera, and D. Macciò. Quanttree: Histograms for change detection in multivariate data streams. In Proceedings of the 35th International Conference on Machine Learning, pages 638-647, Stockholm, Sweden, 2018
[3] C.-L. Li, W.-C Chang, Y. Cheng, Y.-M. Yang, and B. Póezos. MMD GAN: Towards deeper understanding of moment matching network. In Advances Neural Information Processing Systems 30, pages 2203-2213, Curran Associates, Dutchess, NY, 2017.
[4] D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A.J. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In Pro- ceedings of the 5th International Conference Learning Representations, Toulon, France, 2017.
[5] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
[6] W. Zaremba, A. Gretton,, and M. B. Blaschko. B-test: A non-parametrie, low variance kernel two-sample test. In Advances in Neural Information Processing Systems 26, pages 755-763. Curran Associates, Dutchess, NY, 2013.
[7] G. Wynne and A. B. Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research, 23(73):1-51, 2022.
[8] Shekhar, Kim, and Ramdas. A permutation- kernel two-sample test. In Advances in Neural Information Processing Systems 35, pages 18168-11 181800 Curran Associates, Dutchess, NY, 2022.
[9] K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. In Advances in Neural Information Processing Systems 28, pages 1981-1989. Curran Associates, Dutchess, NY, 2015.
[10] W. Jitkrittum, Z. Szabó, K. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems 29, pages 181-189. Curran Associates, Dutchess, NY, 2016.
(11) M. Scetbon and G. Varoquaux. Comparing distributions: & geometry improves kernel two- sample testing- In Advances in Neural Information Processing Systems 32, pages 12306-12316. Curran Associates, Dutchess, NY, 2019.
[12] P. Golland and B. Fischl. Permutation tests for classification: towards statistical significance in image-based studies. In Proceedings of the 18th International Conference on Information Processing in Medical Imaging, pages 330-341, Cumbria, England, 2003.
[13] D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
[14] X.-Y. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks. CoRR/abstract, 1909. 1298, 2019.
[15] H. Cai, B. Goggin, and Q. Jiang. Two-sample test based on classification probability. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(1):5-133 2020.
[16] I. Kim, A. Ramdas, A. Singh, and L. Wasserman. Classification accuracy as a proxy for two-sample testing. Annals of Statistics, 49(1):411-434 2021.
[17] S. Jang, S. Park, I. Lee, and O. Bastani. Sequential covariate shift detection using classifier two-sample tests. Proceedings of the 39th International Conference on Machine Learning, pages 9845-9880, Baltimore, MD, 2022.

---

[18] X. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions on Information Theory, 68(10): 6631-6662, 2022.
[19] J.-M. Kübler, V. Stimper, S. Buchholz, K. Muandet, and B. Schölkopf. AutoML Two-Sample Test. In Advances in Neural Information Processing Systems 35, pages 15929-15944 Curran Associates, Dutchess, NY, 2022.
[20] S. Hediger, Loris L. Michel, and J. Näf. On the use of random forest for two-sample testing. Computational Statistics & Data Analysis, 170: 107455-107466, 2022.
[21] K. Muandet, K. Fukumizu, B. Sriperumbudur, and B. Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends(k in Machine Learning. 10(1-2): 1 1-141, 2017.
[22] .E. Freeman, I. Kim, and A.B. Lee. Local two-sample testing: A new tool for analysing high-dimensional astronomical data. Monthly Notices of the Royal Astronomical Society, 471 (3)33733-2822, 2017.
[23] A. T. L. Lun, A. C. Richard, and J. C. Marioni. Testing for differential abundance mass cytometry data. Nature Methods, (477):077779,2011
[24] M. Roederer and R. R. Hardy. Frequency difference gating: A multivariate method for identi- fying subsets that differ between samples. Journal of the International Society for Analytical Cytology, 45(1):56-64, 2001.
[25] T. Duong, I. Koch, and M. P. Wand. Highest density difference region estimation with application to flow cytometric data. Journal of Mathematical Methods in Biosciences, 51(3):504-521, 2009.
[26] Soriano and L. Ma. Probabilistic multi-resolution scanning for two-sample differences. Journal of the Royal Statistical Society: Series B, 79(2)5447-572, 2017.
[27] J. A. Pura, X.-C. Li, C. Chan, and J.-C. Xie. Team: A multiple testing algorithm on the aggregation tree for flow cytometry analysis. Annals of Applied Statistics, 17(1):621-640, 2023.
[28] Duong. Local significant differences from nonparametric two-sample tests. Journal of Nonparametric Statistics, (5(3):635-645 2013.
[29] J. D. Dworkin, K. A. Linn, A. Solomon, T. D. Satterthwaite, A. Raznahan, R. Bakshi, and R. T. Shinohara. A local group differences test for subject-level multivariate density neuroimaging outcomes. Biostatistics, 22(3):646-6611 2021.
[30] I. Kim, Ann. B. Lee, and J. Lei. Global and local two-sample tests via regression. Electronic Journal of Statistics, 13(2):5253-5355, 2019.
[31] S. Sonnenburg, G. Rätsch, and C. Schäfer. general and efficient multiple kernel learning algorithm. In Advances in Neural Information Processing Systems 18, pages 1273-1280. MIT Press, Cambridge, MA, 2005.
[32] M. Gönen and F. Alpaydın. Multiple kernel learning algorithms. Journal of Machine Learning Research, 12:2211-2202 2011.
[33] R. De Maesschalek, D. metrics and Intelligent Laboratory Systems, 50(1):1-18, 2000.
and D. .L. Massart. The mahalanobis distance. Chemu-
[34] Q. K. Weinberger and K. L. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10(9):007-244, 2009,
[35] J. Wang, A. Kalousis, and A. Woznica. Parametric local metric learning for nearest neighbor classification. In Advances Neural Information Processing Systems 25, page 1610-1618. Curran Associates, Dutchess, NY, 2012.
[36] E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell. Distance metric learning with application to clustering with side-information. In Advances in Neural Information Processing Systems 15, pages 505-512. MIT Press, Cambridge, MA, 2002.
12

---

[37] S. Xiang. F. Nie, and C. Zhang. Learning mahalanobis distance metric for data clustering and classification. Pattern Recognition, 41(12):3600-3611 2008.
[38] A. Gretton, B. K. Sriperumbudur, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Balakrish- nan, and K. Fukumizu. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems 25, pages 1214-122 1422. Curran Associates, Dutchess, NY, 2012.
[39] F. Liu, W.-K. Xu, J. Lu, G.-Q. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for con-parametric two-sample tests. In Proceedings of the 37th International Conference on Machine Learning, pages 6316-6326, Virtual, 2020.
[40] B. S. Everitt. A monte carlo investigation of the robustness of hotelling's one-and two-sample t
2 tests. Journal of the American Statistical Association, 74(365):48 51, 1979.
[41] H. Hotelling. The generalization of student's ratio. Annals of Mathematical Statisties, 2(3): 360-378, 1931.
[42] N. Deb, B. B. Bhattacharya, and B. Sen. Efficiency lower bounds for distribution--ree hotelling- type two-sample tests based on optimal transport. CoRR/abstract, 2104.01986, 2021.
[43] Boyd, Boyd, and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[44] E. Deadman, N. J. Higham, and R. Ralha. Blocked schur algorithms for computing the matrix square root. In Proceedings of the 11th International Workshop on Applied Parallel Computing. pages 171-182, Helsinki, Finland, 2012.
[45] L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.
[46] J. Neyman and E. S. Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London: Series A, 231:289-337, 1933.
[47] Cornfield. Sequential trials, sequential analysis and the likelihood principle. The American Statistician, 20(2): 1966.
[48] C. R. Genovese, K. Roeder, and L. Wasserman. False discovery control with p-value weighting. Biometrika, 933))509-552, 2006.
[49] F. Zhang and J. Gou. Refined critical boundary with enhanced statistical power for non- directional two-sided tests in group sequential designs with multiple endpoints. Statistical Papers, 62(3): 2021.
[50] H. F. Kaiser. Directional statistical decisions. Psychological Review, 67(3):160-167) 1960.
[51] A. Cohen and H. B. Sackrowitz. Directional tests for one-sided alternatives in multivariate models. Annals of Statistics, 2666):2321-2338, 1998.
[52] O. lbe. Fundamentals of Applied Probability and Random Processes. Academic Press, 2014.
[53] L. Anselin. Lagrange multiplier test diagnostics for spatial dependence and spatial heterogeneity. Geographical analysis, 20(1):1-17, 1988.
[54] D. Follmann. A simple multivariate test for one-sided alternatives. Journal of the American Statistical Association, 91(434):854 1996.
[55] M. J. McIntosh. Calculating sample size for follmann's simple multivariate test for one-sided alternatives. American Statistician, 76(1):16-) 2022.
[56] J. M. Lachin. Applications of the wei-lachin multivariate one-sided test for multiple outcomes on possibly different scales. PloS One, 910)ee188884 2014.
[57] A. Graybill. Theory and Application of the Linear Model. Duxbury Press, 1976.
[58] I.. Ma and W.-H. Wong. Coupling optional pólya trees and the two sample problem. Journal of the American Statistical Association, 106(496):1553-1555 201
13

---

[59] G. Boracchi, C. Cervellera, and D. Macciò. Uniform histograms for change detection in multivariate data. In Proceedings of the 30th International Joint Conference on Neural Networks, pages 1732- 1739, Brisbane, Australia, 2017.
[60] A.-J Liu, Lu, and G.-Q. Zhang. Concept drift detection via equal intensity k-means space partitioning. IEEE Transactions on Cybernetics, 51(6):3198-322 -2111,, 2021.
[61] G. V. Kass. An exploratory technique for investigating large quantities of categorical data. Journal of the Royal Statistical Society: Series C, 29(2):119-127, 1980.
[62] J. R. Quinlan. Induction of decision trees. Machine Learning, 1(1):81-106, 1986.
[63] J. H. Friedman. A recursive partitioning decision rule for nonparametric classification. IEEE Transactions on Computers, 26(4):4044008 1977.
[64] R. Quinlan. Simplifying decision trees. International Journal of Man- Maa-mme Studies, 27(3): 221-234, 1987.
[65] S. Gelfand, CS Ravishankar, and E. J. Delp. An iterative growing and pruning algorithm for classification tree design. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 818 18-3,, Cambridge, MA, 1989.
[66] I.. Rokach and O. Maimon. Top-down induction of decision trees classifiers-a survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C. 3564):476-487, 2005.
[67] B. Duan, A. Ramdas, and L. A. Wasserman. Familywise enror rate control by interactive unmasking. In Proceedings of the 37th International Conference on Machine Learning, pages 1720-2729, Virtual, 2020.
[68] G. S. Mudholkar and D. Srivastava. class of robust stepwise alternatives to hotelling's 2 tests. Journal of Applied Statistics, 27(5):599 -19, 2000.
[69] Y. Hochberg and A. C. Tamhane. Multiple Comparison Procedures. John Wiley & Sons, Inc., 1987.
[70] S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979.
[71] F. Bretz, W. Maurer, W. Brannath, and M. Posch. A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine, 28(4):586-60) 604, 2009.
[72] A. C. Tamhane and J. Gou. Advances in p-value based multiple test procedures. Journal of Biapharmaceutical Statistics, 28(1): 2018.
[73] B. Duan, A. Ramdas, and L. Wasserman. Interactive rank testing by betting, In Proceedings of the 1st Conference on Causal Learning and Reasoning, pages 201-235, Eureka, Canada, 2022.
[74] W. Li, G. Dasarathy, K. N. Ramamurthy, and V. Berisha. A label efficient two-sample test. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, pages 1168-1177, Eindhoven, Netherlands, 2022.
[75] A. Schrab, Kim, B. Guedj, and A. Gretton. Efficient aggregated kernel tests using incomplete u-statistics. In Advances in Neural Information Processing Systems 35, pages 18793-18807. Curran Associates, Dutchess, NY, 2022.
[76] Liu, W. Xu, Lu, and Sutherland. Meta two-sample testing: Learning kemels for testing with limited data. In Advances in Neural Information Processing Systems 34, pages Curran Associates, Dutchess, NY, 2021.
[77] F. Cazáis and A. Lhéritier. Beyond two-sample-tests: Localizing data discrepancies in high- dimensional spaces. In Proceedings of the IEEE International Conference on Data Science and Advanced Analytics, pages 1-10, Paris, France, 2015.
14

---

[78] M. Sugiyama, T. Kanamori, T. Suzuki, M. Plessis, S. Liu, and I. Takeuchi. Censity-different estimation. In Advances Neural Information Processing Systems 25, pages 692-700. Curran Associates, Dutchess, NY, 2012.
[79] Y. P. Mack and M. Rosenblatt. Multivariate k-nearest neighbor density estimates. Journal of Multivariate Analysis, 9(1):1-15, 1979.
[80] S. Dasgupta and S. Kpotufe. Optimal rates for k-nn density and mode estimation, In Advances in Neural Information Processing Systems 27, pages 2555-2563 Curran Associates, Dutchess, NY, 2014.
[81] W. Jitkrittum, Z. Szabó, and A. Gretton. An adaptive test of independence with analytic kernel embeddings. In Proceedings of the 34th International Conference on Machine Learning, pages 1742-1755 1751, Sydney, Australia, 2017.
[82] R. A. Johnson and D. W. Wichernand. Applied Multivariate Statistical Analysis. Prentice Hall, 2002.
[83] Papoulis A and U. Pillai. Probability, Random Variables and Stochastic Processes. McGraw Hill, 2001.
[84] K. V. Mardia, T. Kent, and J. M. Bibby. Multivariate analysis. Probability and Mathematical Statistics, 1979.
[85] L. Lei and W. Fithian. Adapt: An interactive procedure for multiple testing with side information. Journal of the Royal Statistical Society: Series B, 80(4):649 679, 2018
[86] S. Rabanser, S. Günnemann, and Z. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In Advances in Neural Information Processing Systems 32, pages 1396-1408. Curran Associates, Dutchess, NY, 2019.
[87] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. In Proceedings of the 4th International Conference Learning Representations, San Juan, PR, 2016.
[88] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations, San Diego, CA, 2015.
[89] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. De M. Raison, A. Tejani,
S. Chilamkurthy. B. Steiner, L. Fang, Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8004 8035. Curran Associates, Dutchess, NY, 2019.
15

---

A Detailed Proofs for Our Theoretical Results
A.1 Proof of Lemma
We begin with some useful definitions and lemmas as follows:
Definition 6, (Random Metric (9)) We say that p random metric with values R, indexed with pairs from the set of probability measures M, i.e., {p(P, Q): P. Q M). if it satisfies the
conditions for metric with qualification 'almost surely'. Formally, for every P, Q, U E M, random variables p(P, Q). p(I,, U) and p(Q, U) satisfy
1. (PP, Q) 2 0 a.s.
2. IP = Q then p(P, Q = 0 ass, i Q then p(P, Q) + 0 a.s.
3. (F. Q - p(Q,P) a.s.
4. PP, Q) S ((P, U) p(U, Q) a.s.
Lemma 7. [11] Let μ be absolutely continuous Lebesgue measure Rd. A non-zero analytic
function I can be zero at most in the set of measure w.r.l. μ
Lemma 8. [9] If K is a bounded and analytic kernel on R2d x Rd, then it analytic for overyffucttin in the RKHS associated with this kernel.
Definition 9. For test locations V = f(21 12,,00.-.1 ... _ue and Mahalanobis kernels f1, Ki,, K. Rig given
by Eqn. (I), define
Pu an (P, Q) Σ ([yyP) ( (v,))
(12)
where [ayP] E (ve, w,)l and 1 (0,) Ey-olts 0,)].
We show that the above distance metric for two probability measures is random metric as follows: Lemma 10. If u1, .2,... 0e are drawn i.i.d. from absolutely continuous distribution G, then Premaae" see (ss is random metric for bounded kernels {K1, "2 Kee)
Proof. For j E [8]. we first introduce function
Pug (P, Q (In,P] - I1,))
and it sufficient prove that Prg (P. ? is random metric for each j [f] from Eqn. (12).
It is well-known that Mahalanobis kernels in Eqn. (1) are characteristic and analytic from [81], and the corresponding mapping
P and Py Q Qu
injective for Kij, where ny and Q denote the images of measures PP and Q, respectively. Hence, the image of (1j is a subset analytic functions for analytic and bounded ky, according to Lemma 8.
If P Q, then we have
f, = u.P 12 -0 and Pr. - - 1u, (1,)) =
(I,FF)
We now prove that if IP + Q then ʃj + almost surely, by applying Lemma 7 to analytic function 1= u Q with distribution G. For injective map (lj, there exists at least one point such that fa(a) # 0. and there exists ball around with non-zero f, from the continuity of f,. Hence, is almost everywhere nonzero based on Lemma 7, and this follows that
Pa, (P. Q (Iu,1P] (v,) [u, (v,))² > 0
a.s. for IP P
Hence, Pys random metric from Definition from the symmetry and triangle inequality of Pas-
Proof of Lemma 1. We first have
He Eape [é] = (())) (22)) ... (u)P)(o:))
Egrali [ (v1)) (v2), (vx)
14
16

---

and further rewrite Eqn. (12)
Pariarr pe (P, Q2) = Σ
Σ p4. 10.))03) m3.) )
= plp. p
j=1
From Lemma 10,
that Ppi.//2." (I, 0 if and only MPP=00 Q, and this implies that
Pg Pg and only if IP = Q. This completes the proof of Lemma 1.
A.2 Proof of Theorem 2
begin with useful lemma from [11] as follows.
Lemma 11. For symmetric and positive definite matrix E, function h(E) = - and well-defined on the positive definite space.
1/2 --1/2 is continuous
Based on this lemma, we present the detailed proof Theorem 2 as follows:
Proof of Theorem' For embedding testing sample X' { .. 3 and 8 (091.... -9 recall the pooled covariance matrix in Eqn. (3)
(m' 1) + (n' 1)E8 eld
Σ E
m² + -2
and we also define
Ex
P = Exintm Sp
We first prove that testing statistic T(X', Y') is almost surely asymptotically distributed x² with E degrees of freedom under the condition μ4 = 1g for small e.
For i.i.d. samples X' and Y', Ca is independent Cg. Up M then Cx - cy follows a multivariate normal distribution with mean μ = M4 Hg 0 and covariance matrix (m' n') E')Ep, ///m///mrr m'n',, from [41, 82] and the Slutsky's 's theorem [83], that
4 N(O, (m' n'' nn)ppp.//m'n))
where 4 denotes convergence in distribution.
The matrix L is symmetric and invertible in the Schur decomposition Ex.g. LL. Our statistic can be formalized
T(8,8) = m'n't T E 2/ 2/(m' / + n')
= m'n' & L' L'' //11((m²) + n')
(Vm'n' L' 1111 Vml ni 2
and by applying the Slutsky's 's theorem, we have
Vm'n'L' 2/ Vml + _ NN(N
This follows that
T(X, m'n' m' + n
j=1
where (L' is the j-th dimension value of L'-1z, x, and /m'nt 'n' (m' + (L')1))) follows the standard normal distribution. This proves the x² distribution for T(X', Y¹) from the sum of squares of standard (i.id.) normal random variables.
We then prove that x2 (T (*)) 0 as m'n' /mmm + n²) 0o if up # ug. and hence our test rejects Hg almost surely.
17

---

From Lemma 11, we have
lim m m'n//(m +n')
L'-1
lim -013 m m'n'/(mm'+ (mn//mmmmm==- (om'+nn)+o> )+ox
8 =
1-112
p.
In similar manner, 2 cx Cy converges to 14g M4 in probability. For Me # May have
| - M&)) >00
Then, ||'-12112 is continuous function with entries 2 and L'-1, and it is convergent to some positive constant. We have (m' + =x))/m''n'xxx. - 0, and
m'n' mn + n
1 X2.a P (11-
m' + nt' m'n Xēa) 1.
Pr
This follows that
m'n'
x (T(X' 83) = Xe
ml n || > +0
and hence (T((X, 0 m' (m' + n') if Mp + ua
A.3 Proof of Lemma
Recall the pooled covariance matrix , and Pp. = Ex-n Ginlpe [ g. in the proof of Theorem 2, and & (cx: cv.) follows multivariate normal distribution with mean peş Mg and covariance matrix (m' + nn' n')= / We first observe
/m'n'.
L = Exiebm &
[L'] = "
=
and this follows that
m' + n' m'n' Ie
1'-1 N [
1-1 (14 Me))
This is because E[ = L (pp - -13) and the covariance matrix is given by
m' + n' m'n' m' + n' m'n'
Cov(L'-1z,1 22, = 1
Lrd
m' + n' 1/300 m'n'
Write B v ml'll / n') L' diag F)), and have
m'n' L'- diag diagi))) (V
m'n'
B B
L' L'' diigddagFFF))
m' + n
ml + n'
m'n'
m'n' m'n'
m' n. diag (iLg(F)(L' L' diag
diag( F) F) x) diag(F)
n
m'n'' m'' + n diag(F) F) 1 8 8, m' + n
==
diag --1 X.
m''n' Σ
m' +m² 8,
by using the symmetry of E-1 x.
It is easy to get
E B
Im'nn/(mn + n') 7.
diag( F) - p()
and covariance matrix is given by
m'n'
m' + n''
m'n'
Cov Cov(B, B) = V m' + n'
-1 diag F) 2.0
m'n' Egp 2)) m' - n² diag( [ -1
L
P.0
m'n'
m' + nl men' L b V ml nd diag F) L
m'n'
ml n diag( F L
P.0
I
18

---

This yields that
~ m''' m'm' + +1) (1) uo); 1t)
p.
and all random variables in B are mutually independent. Define
B B/1 BB/e and S = ( (B--B)²
==1
1=1
and B is normally distributed with mean Vm'n' (m' + n²) F (1p MQ)/ and variance 1/E.
P.
is easy to see that
F L' 1'-1 (exi -x.))= = v(m' + n²) n1)m/'n'1 B,
which yields that
m' + m'n
Pr [sgn( F L' (exr - cq.)) =1] = Pr
1 >
= Pr [ 0]
(13)
We further have
Pr [sgn( F L' - 1] +
(-V
m'n' (m' + n')e
F L P.0
(14 ms)
where (-) is the cumulative distribution function of standard Gaussian distribution. For continuous normal distribution, we have
Pr [sgn(F F L' ('x'(ex- - -g.)) = 0] Pr B 0 = 0,
and this follows that
Pr (F L'-1 (cx: (c.)) = --1 Pr
(-V
m'n' (m' + +nn)e"
F Leb(ue me)
Hence, the sgn(F" (T L'-1 (0gi-ccp11 follows two-point distribution TP(E) with parameter
g $=1-0 (-V
m'n' F L us2) (m' + n'}e
This completes the proof.
A.4 Proof of Theorem
Recall the pooled covariance matrix E and Ep,ù F pi (xxy in the proof of Theorem and & (cx* cp.)) follows a multivariate normal distribution with mean μlş - Pt and covariance matrix (m' + n'' n')=p, 2
/m'n'. /m'nn' We have L Σe.0
We begin with useful lemma and corollary follows. Lemma 12. FT L P,0 F
P.c ug null hypothesis Ho: péş u&g we have
m'n' n 8.
m'n' nd
Pr
z I
F 1'-1
0
Pr
z E &
8
2 2
(14)
Proof. Recall = /'nn n²) L diag (F) and FT 1'-1g
V + n()/m'n'1
B in
the proof of Lemma 3, and Eqn. (14) is equivalent to
Pr B
Pr
B B B
(15)
Recall that
B-1
Σ B1/l and 22 = Σ
(B: - B)²
19

---

and from Eqn. (15), we have
Pr [ 2 (c - // 2 0] 2 Pr [B2 2 (c- ///
From the independence of S and B, is sufficient to prove that, for every 2 0,
Pr B 8 B 2 0] Pr B3 2 6
(16)
It's easy to see that B is normally distributed with mean Vm'n' (m' + n²) F 1p. (14 - 1k)/l
and variance 1/e. We define
m'n! F L (ppp 13),
a = VE8 and b == V (m' + n')e
and from Eqn. (16), we have
((b)
- a)
( 1
where the equality holds from Corollary 13.
0, i.e, u1 ps.
This completes the proof.
F
L L-1
P.
Mp
L F
us have
Pr
m' nn
m'n'
8 p.
2
F L'
=00
<
Pr
m'n' m' + n
E =20
8 p.
Lemma 14.
F
L-1 P.
F I P.c u
then the test power of our bi-directional hypothesis can M
be lower bounded by
q(f, v - E, A, Box) + +
(2/10 FT ( 1 (1p
m3))
ml + n' - and X = w/ L ( ma)1l where w = m'nn/(mm + n'),
Proof. From Lemma 12, we first have, for F L
P.
ug
F L20
m'n'
Pr(h= F -'--===- 2 2
Pr
IJ 15-1 x. 2 Xi,8a
F p' 2
m' + n!
m'n'
Pr
Eー 8 8 Xe
(17)
m' +
by substituting X2. into Eqn. (14). From Me + 1b. we also have, from the work of [84],
m' n' - 1 T(X'8'') m' + n' 1 x m'n' T
(m' n' - -2)e m' + +nn x 2~ ~ F(E, m + nn' (,),
(m' n'' 19-2) 2)
Lp. (1= m²)/(m² + n²).. Given the significance level 8a, we have where \ = m'n' mn'n'((4)) (Mp 1&)
m' + n¹' - 1 T(X 8') > -I = =(lemm +n' - -1,,,,,Ba) (18)
Pr (m' + n' 2 and this follows that
m'n'
Pr
2 2 X2.00 z 5-1 X1,8.
= q(L,, m' + - -1- -12,,,,
m' + +n
Recall B Vm'n' fn'n'/(mm + n'' L' dial'daa())) F)z and F in the proof of Lemma 3, and that B I BBE is normally distributed with mean / fmtnn/(mm + n')F' L 1 m6)/L and variance 1/f. We have
V(m' + n')/m'n'1 B
Pr 0j = Pr 2 0J =1 +b (-V
m/n' (m' + n')
Lib(Hs - (o)
(19)
20

---

since B is normally distributed with mean vm'n'/ + +n')F' L P.c. (up -ma)/e and variance 1/6. Combining with Eqns. (17)-(19), we have
m'n'
Pr
2
--
2
88
F
12-12200
m' + n
81,
m'n'
Pr
T
Σ 8
Xe
Pr Pr(F 1'-1221200 0] F L'-12 2 2
m' + n'
m'n'
Pr
Σー1 X1
22 2 Xe,Ba
Pr(FT 1'
m' + n
(1-0(- (m' + n'ye
m'n'
F 1 (Mp (2a))
ale, ml + n' -1 e, Ba)
(20)
For F L-1
2,0 up > F L-1 H we substitute (2, into Eqn. (14), and it holds that
m'n'
Pr
- x. 2 Xi 0 nF -'-1 1 < 0
m' + n
Pr
B² z (x1,12-9)a 9)a - S < 0
Pr B B v (Xi -52//E
m'n'
=
4
xu - 82 V (ml + n'ye
Lg (14 ug)
m'n'
L (p4.- (pG meg)
+
V 12,(2-81a (m' + +')e
F
(21)
Combining with Eqns. (20)-(21) we give lower bound for the test power of bi-directional hypothesis
Pr 1/- = 1]
Pr h
= 1n F L'-12> 0 Pr h 10 F L < 0]
m'n'
Pr
z TI=l
x 22 22 nF nF L''-1z
m' n'
m'n'
+ Pr
--1
22 (2-
OF L' z < 0
m' n2"
> q(e, m' + +n' -1 1 Ba) ( 1 1-8((1)
m'n'
F I (7g(1s--6)) Hol)
(m' + +n'ee
+Φ (-VARce-on -V (m' + n')l
m'n'
F L mo)
Lemma 15. FT F. < FT <FTI L then the test power of our bi-directional hypothesis can be lower bounded by
q(e, v e, A, (2 3)a) +1 +1 + 1/22 (w/E)
- (Hp-)))
-
where w = m'n' /mm' + n'), v == m + - and X = w|l [| (1p - 130
Proof. From Corollary 13, we have, for F L
< F --1
1.
m'n'
Pr[h = 1| F L' z 0 = Pr
& p. 2= X2,(2-p)a B)a
2
FTL''lz <<
m' + n
m'n'
Pr
-- x. 2 x²,00 (2-
m' + n2'
-
21

---

Recall that
m'n'
Pr
2 _E-
8 8 22 xe) = gle, n'' - 1- e, 12--22888 B)a)
m' + nd
in the proof of Lemma 14, and this follows that
m'n'
Pr
D 8
m)) nF L'-1= <0
mn + n²
8
m'n'
Pr
"xx,, 8',8' 2 X2,(2 (2.(2-B)a
F L'L''1 2 < <0
L'===00
m' + n'
m'n'
Pr
-T
81,8 22 2
(2
Pr[F 1' <
m' + nnn
(-V m'n'
F L (up (1 m2m
= q (L, + -1- 3)a) 1
(22)
(m' +n')e
Vmm + n')/m'n'1 B in the proof of Lemma 3. and that B 1 BBB is normally distributed with mean
Recall B Vm'n' (mm' + (rr))''ldiag ()) and FL
/ménn/(ma n'jF' P.0 (14p m&)/f and variance 1/6. We have, if F L L F L 2
m'n'
Pr
z Σ Σー1 8 Xt F ---=21 220
2 2
2
ml + +n
Pr B22 2 (2, / 0]
- = Pr B2 v (xi,ea
1 1- $ V X22 -S2 V (m' n'ye
m'n'
F. 1 mb)
Vx220a (m' + n'ye
m'n'
F
L (mu - mb)
(23)
Combining with Eqns. (22)-(23), we give a lower bound for the test power of bi- di-directional hypothesis
Pr h = 1] = Pr h = 1 2--200001 Pr [ LLF L L'-1z < 0]
m'n'
= Pr
Σ --- "x'
22 X2.00 n F 1'--z21 0
m' n!
m'n'
+ Pr
m' + n
z
=T Æーー z
8118"1"
()
2
3)a
n FFLL 2 < 0
m'n'
q(e, m' n' e, (2 3)a)
(-V
F¹ 1e,2 ms)
(m' + n')e
+1 Φ
VVxisa
m'n' (m' + n'ye +''je
F Ied
ma²)
This completes the proof.
Lemma 16. For our bi-directional hypothesis, the rype-l ervor rate is equal to a if ps = Pa-
Proof. We first consider the case F LTL1112 0. By substituting X2 into Eqn. (14), we have
Pr[h- = T - 2 0]
Pr
m'n'
T x-1 2 2 X2,30 X
F L'
0)
m' +n²
m'n'
Pr
TT-1 "
2 2 X
m' + n!
3a.
22

---

Recall B vm'n' Domn//(nn (m' + n') diag ()) and F'
"vmm' + n'mmm'n'' B
in the proof of Lemma 3, and that B B/B is normally distributed with mean v Vm'n'//mi + +n'FF L 1.0 (up ma)/e and variance 1/C. We have
Pr[F 1'-12 2 0J = Pr 2 0J = 1/2,
and we have
P m'n'
Xe n OF L' 2 0)
z D-1 x2
2
2
ml + n
Pr m'n' m' + n
F L' 0 Pr|F L' 2 0] Ba X
Σ
2
(24)
8
since B is normally distributed with mean
x < O. we similarly substitute c c= 33a into Eqn. (14), and it follows that For the case F L'
m'n'
Pr
2 8,
2 2 (2, 33a OF L' <
m' + nd
m'n'
=
Pr
m' + nd
I-- 81, 8: 2 2
X2-(2-9ja
m'n'
T "x 8,,8' ;,² 2 (2,(2-9)a 9)a OF LF 0
Pr
m' + né
m'n!
= Pr
= 8. 2 X2,12-03a
ml + +
m'n'
Pr
z 21,8,1 ,,, 22 (2,(2-18a
Pr[F L'-1 x 20
m' + n'
m'n'
m'n'
= Pr
T--1
X1, p. 22 2 (2-
- Pr
z I
x. x2. 22x
m' + nd
2
m² +
m'n'
=
Pr
z T --- X
3 2 X2, X2,(2-M00 M]0
= (1 - 3/2)a.
(25)
2
m' + n'
From Eqns (24)-(25), we have
Pr l
1]
Pr =
11h=10P
L'
1 -'-1 <
m'n
Pr
2
x2 0 F L' 220
m' +n
m'n'
0 OF I' < 0)
+ Pr
m' +n'
h-1 2
X1
9)a
3a
+ (1 - -B/2)a = a.
2
This completes the proof.
Theorem follows from Lemmas 14-16.
A.5
Proof of Theorem 5
Recall that B(1) ...)...., B;,) B..,, are rectangle regions of non-increasing order w.r.t. g(s ). For each rectangle region Big), we could define its local null hypothesis
Ho, MP
Môs,. with MPB
E EarnPay a' and 1 =E EggrCo ID'l
From Theorem 2, the testing statistic T(X
T(X Bug BBy follows the x2 distribution with freedom of &
degrees under the local null hypothesis Ho.(s) Denote by (T( E),) the p-value, and we have
23

---

Lemma 17. [68] The p-value (X 8,6) (3))
8 follows a uniform distribution L/(0, 1] under the M
local mull hypothesis Ho, MB
We take an interactive multi-step testing procedure identify the index of rectangle regions of local significant differences. Define the candidate rejection set Rt) = = { for 1 S t S with R() = 0, and exclude one null hypothesis Ho.te) at the 1-th step. We could generate a sequence follows:
(0 1i R(s) 2 R(s 1) 2 R(s 2) 2 R(0) =
(26)
and it holds that Ho,c0 R(i) \ R(i 1). Recall the local bi-directional hypothesis h() h(X YB,s) for i [s]
Denote by p. the parameter of significant level for the local two-sample test and masked p-value. From Theorem we have (r(h() 1] = p. under the null local hypothesis Ho.0))- We further present some useful lemmas as follows.
Lemma 18. [67] We have ([ = foror E [s] and hay. ...).... h(s) are mutually independent, if MPr () for every E [s] and the p-values are uniformly distributed.
Denote by B the set of rectangle regions that the local two samples Bts and Pb are actually drawn from identical distribution, and define
Ho Ho B BB
Lemma 19. [67] If there is some rectangle region such that Pracs * Moness and if the p-value follows a uniform distribution under null hypothesis Ho, then we have
E hha) (has (har) 1 {I (H), E (Ho)) Hag0 E Hu] p- for i [s]
We also define the weighted mirror-comserratiivnesss, motivated from [85, 67], as follows. Definition 20. We say that a density function f(-) satisfies the weighted mirror-comserrativeness it holds that, for some given p E (0, 1].
(aw) S f(1 (1 wp)a/p) for every E [1,2], (0.
Here, we introduce additional parameter to incorporate two directions for method, which is different from the previous mirror-conservativeer [85, 67). We could also present two sufficient conditions for weighted nirror-conservaiiveness from [85, 67]: i) the non-decrease of and ii) the convexity of cumulative density function of p-value.
Lemma 21. If the density function of the p-value satisfies the weighted mirror-conservaiivensss under local null hypothesis Ho, and if there is some rectangle region such that Mis,,, * P then we have, for every E [s).
E B.Mes (hap)kkmit {I(hças HHa) Ho, E Ho, (100) spa,
(27)
where g(v) g( o(8b,n "yy))
Proof. We first prove
Elhay 960) = a] S p. for Ho. E Ho.
(28)
from our bi-directional hypothesis and masked paluu with 3 E [1,2) in Eqn. (7), and it is sufficient to consider the following two cases:
24

---

For ((s))
18 (cX'ss)
(3 2 0, we have
E h(s) g(t) a, FB... BB (CX'es))
}>
p.f(3a)
p.I(Ba) + (1 - -pp)s((1 1 1-8p= ())
p-
P.
1-0p= a) f(Ba)
p. P+,
p.
where the last inequality holds from the weighted minror-conserrvtiiveesss with p-value's uniform distribution 24 1].
For F
(5)))
L' (cx
-C8
< O. we similarly have
E
hi0 (900) = a, FB,,) FB.,, LBB)) (C8
CR <0
p=f 3)a)
p.f((2 - + (1 - f(( 1-(2- a)
p-
P.
p. + (1 (1-))))) 1- A1p. a)/s((2 3)aa
S Ps.
We define the information available for choosing Ho,(3) as filtration (sequence of nested o-fields)
Fig 9()) (T Y D)1k+11))
and also define the filtration
Fly = (1hrs))e=i4 (I(Ho, E Hol Ho))
This follows that
E (hs)) (haa) M {I(hqs) Hol Ho, E Ho. 1g
E hha Fli Ho) E Ho, (0
E E (eg Fo,,Fii Ho, Ho, [o
Fliy, E Ho, 1om
where the last equality holds from the law of total expectation. We have
F hhs Fea)
Fley Ho E Ho, [p
Σ
Fly, Ho,c, E Ho, (9m)) Jiml]
HoJER((j)H
x Xr Ho,0) = Foo, Flay Ha, E Ho. lim
= Σ
E Tha Fill Pr Ho..0 = Ho, Fly Ho,0 E Ho, (owe )im1
where the last equation holds from the fact that (F), Ho,(3) Ho, (im)) is subset of F(a)- We further have, since hig is independent of other information in F(g)
E han Fuy
Flyy Ho Ho, (9 1i=1
Σ
E [hys [w)] Pr Ho,0) Ho, Fio), Flyy Ho E Ho. (g Pim
Ho Ho
s P+ Σ
Pr Ho Ho, Fia. Ho,0) E Ho. (0m)
HR)ER(
P+.
25

---

which completes the proof.
We say that a random variable follows Bernoulli distribution with parameter p. denoted by Z ~ (ern(p). if
Pr|Z 1] =p
and
Pr|Z =1
We also say that random variable Z follows negative binomial distribution with parameters r and p. denoted by Z NB,r,,) if
Priz k] k|= k + ( ( p)'p"
is necessary to introduce definition as follows:
Definition 22. We say that a random variable is stochastically dominated by distribution G, denoted by
Z Z=G,
if for random variable X G, it holds that
Pr[Z 2 x] S Pr(X2x) for E (-00, +00).
We further introduce some useful lemmas follows:
Lemma 23. [67] Let "., 7sg be i.i.d random variables with Z3 ~ Bern(pa) (P) for some p. > 0, and write N. Ej= Zj and G. oN,, (2)))=44)) for E [s]. We have
N z NNB((
where the stopping index is parameterized by some constant (( 1), defined by
F max {o (0<tts:tt-W N. or t t=
We further introduce weighted version of Lemma 23 as follows:
Lemma 24. [67] Let (Wj be a sequence of weights, drawn from a Bernoulli distribution, s.f.
Ej=1 W3 = for fixed constant S 8; and Zj ({ W Walk=stt, 1) Bern(p.).
Write No -ド WzZj, and we have
N Z NB(u
where the stopping index " is parameterized by some constant v(> 1), defined by
O<t<s:
Σ W9 No < or or t=1
We now introduce a different version of Lemma 24 by considering different parameter for Bernoulli distribution as follows:
Lemma 25. [67] Let Zj (K.,, Wilies411 Wg = 1) follow a Bernouli distribution with param- eter p(ZZ,, W W for E [s], respectively. We have
N = NB (v. p( (z, Well NBB(p.).)
PFZ(, Wa P. for every E [s].
We now present the detailed proof of Theorem 5 as follows.
Proof of Theorem 5. We first consider Pip Môs for every [s). From Lemma 18, (hay 1 are i.i.d. random variables with hap Bern(p.). Recall that the stopping rule in our testing, i.e., Eqn. (11), which is equivalent to
1 - (1 p.)* [2(1)++1 1-(0)1
au
26

---

where ar is a parameter to control familywise error rate and [(t) = {i [t] h(XB (B,)) = 1}. The stopping rule can be rewritten
[Z(+) with
V [ln[1 - a. a../Im(1 - pp)]
(29)
Let Z3 = huy) and N₂ = 2j=1 Zz. We define the stopping index 7 as follows
-
F max {o ( N,
1 111
From Lemma 23, the number of rejections at the stopping index given by
I(F) =
bリ - NNBopp.)
If (Peea)
for every E [s]. then the number of false rejections is
I(7) Hol [Z(>) (U,
and hence the familywise error rate (FWER) is upper bounded by Pr [[Z(F) Hol 2 1| S - (1 p.)" S a.,
where the last inequality follows from Eqn. (29).
We now consider that there is some rectangle region with MPg;;
* Moses In such case, we provide an upper bound for familywise error rate without the information of masked p-values, and prove that the number of false rejections stochastically dominated by (B.
Let Zj = hig) and W = I(Ho, E Mo)) We define the stopping index -" as follows:
T max 0 O<tSs: 8: :
[ho = o0 Ha,0) Hol Σ j=1
WG(1 21) <v <10
j=1
where v is given in Eqn. (29). It is easy to see that
Z, ((7,, Walk=s+1: W, = 1) Bern(p.)
from Lemma 19. Denote by (76), and we have Σi= W, and From Lemma 24, have the number of false rejections
Σ
(ho = 1 n Ho,u) E Hol = Σ W,2 No ABe,p. j=1
(30)
Recall that - [Z(()] < o is our stopping rule on the exploration of local significant differences. Denote by 7 the stopping index in exploration, and have
/
Σ Ihe = on Ha E Hol
Σ [hú = 0] = T
Ihú = =1] = I(TT) < v.
Since No is non-decreasing with respect to f, it is easy to obtain
T/ S "" and
N
< N
and we have the number of false rejections
[[TT] Hol
Thu 10 1 Ho] = Now < No
3NB((e,p)) p.).
(31)
27

---

We upper bound the familywise error rate without considering the masked p-values as follows:
Pr[ n Hul 2 1] & Pr [[((") Hol 2 1| S - (1-p.)" S a.
We finally take masked p-values (g li= into consideration. From Lemma 21, it is easy to observe
Zg la ({ We W= Wj =) Bern(p( Wx)
where
p(t (e,
[me)) (han)) ) (T(hau) Ho))i=s411 Ho,ss Ho, (a()) $ p.
This follows that
NB (v.p (( Wa Wx) 3 NBBuv,))
and we further have, from Lemma 25,
[[(TT) Hol
Σ
Wyz No NB (v.p (fZ,, Wall=)+1)) = NB(u.p.).
j=1
We finally upper bound the familywise error rate by considering the masked p-values {a l as
Pr
[I( n n 2 1 Him]
P ((I(TT)) Hol 2 1 (i
1 (1 a..
This completes the proof.
B Optimization for Test Locations and Mahalanobis Kernels
We take gradient method [43] for the optimization of Eqn. (5) in the work of [10]. Specifically, calculate gradients, and update test locations and Mahalanobis kernels iteratively. In the following of this section, present the calculation of some crucial gradients in optimization.
For test location Uy with j E [f], we have
Ve, ve, ((8,)) OT(X, OT(X,8)
OT(X,8)
(32)
001n1
00s,2
where, for i E [4)
OT(X,8) 0T 8) des
OT( Y) dey Tr
,T(X, 0E
(33)
Doga
dcx 0eg
dey 00g
0Exx
Oeg
where Tr|-] denotes the trace.
further have
8T(,
2mm2 8, x,y (ex cm) +n)
dcx
dcg
m
dz,
with dæ,
(0.... (0......., ax (ăr v,) Dog
0, .0) .))² T
0oj.i
m
Dogs
where all elements are zeros except for the j- element. We also have
Ony (Ir: Uz) toys
(exp (-(x, - v,) ( M3
- u;)
Ovy,
ky(ār, (M) - -2;)) (7;))](0 (O. (0.... .1, 1......0 0) /73
where the i-th element is 1. We similarly calculate 0T(X, 8) 000.
28

---

For the third term in Eqn. (33), we have
OT(XY))
E ccylex - cpp)
-
E-1
28,8'
xE
lá. cx) cx)(E, -xx)
0( ci [ cΦ)
+ Σ
m + 12 -
dos
where
0( cx (x) cx
Orgi
0(2, -e -g))(,
0
0
:
0 -xg (d, Beaa-exx ba Dog,
0 - cx (at
-e 10
a(a. -x Ma. 33
:
0 og (d, 0ng.
2x 2
0
Here, (á, - 0x (2r, cx 33,0 denotes the element in j-th row and t-th column, and we have (0, -x-cx)((a -cx)],e
1-11 m
Kic (ar, UU)
m Σ sppr
Ni (x Uz))
Kig (2 v2)
Σ
m
ke (2 Ue)
m
m
m
spr
and this follows that
A(z, cx)( - -xx)
Orey m -1 Kg( Uz)
Σ Kig (1or vz) (
1 0kg (mpp Uy)
m
Ok, (2r (g) Oo,,
m
O0,
efpir
We similarly have 0(2, cx)( ex)
Önjj
= 2 m K, (2 Uz)
1
Σ #fr
Ky 10i)
m
v.))
dKy (Ier vy) Zmj.i
For gamma parameter ?/ with j [e]. we have
V,, )( = 0T(X, f) dcx
OT 8) dey
(TT(X OF 0E
+ Tr
(34)
dcx Oy
dey
Oy
σΣ
Ory
We further have
dcx
1 Dæ,
(0, (0..... dz,
Ōky aky(ār; U,) ,.... ....,))² Onj
with
m
0y Byj
r=1
where
0k,(2,, 0;)
Oexp (- TT)T TTMM(;; 009)/273)
O7j
Ogj (e (3) ux) MM, v,)n
29

---

We similarly calculate
1
cx)(tr -xx)
Ö(gr cx)(Ur - cx) er)
07j
07g
07g
r=
where
0( - cx) - -cx)
0%
0(0, ex) [(rr
-(p0--0))11 cglils
0
07
0(
cx (d, cx 1/11
0(0. ex -xxx-x)) -xx 17.
0 cy Ma.
0%
02. 2x (d, -ex)
0
Ons
and this follows that 0(2, ex)(a, -cx)
Ons
m
Ki (2r Uz)
Σ 1x6r
m
Ki sxs Vt)
m 1 0x, (2r, Uj)
m Σ
dr, (x Uz)
m
m
m
O7j
m.
07
esp
We similarly have
0(2 - cx) - cx)
87,
=2
m
Kg((rr Uz)
m Σ (2 Uz)
1
m
axy(ar. Uz)
Σ
anz(wss 1z)
m
m
On
Oni
For Mahalanobis matrix Mj with j E [C], we have
0T(&88
0T1&,8)
0M,a..
aM
M (( =
(35)
aT(&
oT(&
JM,
where denote by M Mj,abb the element in a-th and b-th column in My. We further have
0
0T(8,8)
dcx
9T(X,8) dey
OT(X, 2 3x
+
Tr
J,,
dcx
aMjap
dey
DMiab
00x 0Mj.bb
with
dcx
1
Σ
and
dæ,
0 (o....,
@xylār, UÚ)
OM,aab :. ))
3M
m
aMj.ab
and
Digy(1, vz))
2 Dexp (-(x, U,) T,)T (2, U,))/27?
aMaab
aMjab
Ki (Mr; Uz)
O(xr - -1,) TT,,-,,,,.. - -0y) 0M,
=
273
Tr
0M,
OM,
Ri (2 Uz)
Tr [xx,- - 0,)(2r TT)T Ja.b]
2n}
30

---

where Jnh is the single-entry matrix (1 at (a. and zero elsewhere). We similarly calculate dcy/O0 j0,b and have
x
O ex)( exr
0(ū. ey ey) -c)T
j.,bb
m + n
iM
J.,.o,bb
where
o( - (,, -
aMg.ab
0 cx) - -ggis
0
0Mjuubb
010,
cx M2, -x
0 ex Me, cx
0(2 ox Ma. ex
JM3.u.b
UM junb
BM,
012 -cx (tr
0
O.H.a.h
Here, we have
0(0, cx) -xx]e cx
jMjabb
171 (i,
Σ #7
K, (2,, Uz)
m - (2
(Br, Uj)
Σ
(2 Uj)
OMiabb
0Mjab
esr
We similarly have 0(z - cx (á, - ex)T (MMaa, and
aa -xx)(a, -c
aM
-1 (2rr B3)
1 m m
Σ (a By))
(m- (m--)) (35, D2)
m
(1s 0z))
m
m
OMj.abb
m
aM,a,bb
Project Mahalanobis matrix onto positive definite
We can not guarantee the positive-definiteness of Mahalanobis matrices during the optimization process via gradient ascend. Motivated from [43], we project Mahalanobis matrix onto positive definite cone as follows:
Present the spectral (eigenvalue) decomposition of Mahalanobis matrix M
M
ApPPP.
where A2, Aa are their eigenvalues with corresponding eigenvectors Pi Pi.P2." P.. Pa- Project the Mahalanobis matrix M onto positive definite cone
M
max {Ai, 8) Vana for small positive constant 8.
C
Datasets and Parameter Setting
Datasets
partition datasets into several disjoint subsets, and then randomly draw data elements from each subset based the sample fraction, i.c., the proportion of samples be selected from each
31

---

subset. We construct two different samples by using two different sample fractions in above stratified sampling process, and construct two samples drawn from one identical distribution by adapting a same sample fraction, as done in [60].
We provide the details of constructing two samples for each dataset as follows:
blob is constructed the mixture of nine Gaussian modes. We first write
d1 0]. Ug [0, 1], ug [0,2] 2]
2 = [1. 0]. us = [1, 1], us = [1,2],
u7 [2, (2,0), ug [2, 1],
Ug [2, 2].
and
- 0.02 0.002 x (i-1) for i < 5
A,
0
for 5
0.02 + 0.002 x (i 6) for 5.
To construct different distributions IP and Q. we adapt different covariance structures and
9
1
0.03 A;
1) 0.03
A:
P N (us, 0.03 12) and
Ui
i=1
To construct identical distribution for two samples, i.e., P = Q, we have
P P= V (uz, 0.03 x I2) and Q Q Σ N (u; 0.03 12)
We set the sample size for training to 900 and for testing to 224.
dna is a categorical dataset with 3 classes. For constructing two different samples, we set the sample fraction for one sample to [0.30, 0.35, 0.35] and for the other sample to [0.45, 0.25, 0.3]; To construct two samples with one identical distribution, we set the same sample fraction [0.30, 0.35, 0.35] for two samples. We set the sample size for training to 1000 and for testing to 250.
agnos (agnostic) is a categorical dataset with 2 classes For constructing two different samples, we the sample fraction for one sample [0.35, 0.65) and for the other sample to [0.65, 0.35]: To construct two samples with one identical distribution, we set the same sample fraction [0.35, 0.65) for two samples. We set the sample size for training to 1000 and for testing to 250.
topo21 is a regression dataset with continuous target variables. Based on the sorted target variables, we divide the data into 4 equal parts. For constructing two different samples, we set the sample fraction for one sample to [0.1, 0.3, 0.2, and for the other sample to (0.5, 0.2, 0. 0.2): To construct two samples with one identical distribution, we set the same sample fraction (0.1, 0.3, 0.2, 0.4] for two samples. We set the sample size for training to 2200 and for testing to 550.
har is a categorical dataset with 6 classes. For constructing two different samples, we set the sample fraction for one sample to [0. 0.20, 0.10, 0.20, 0.20, 0.10) and for the other sample to [0.15, 0.15, 0.20, 0.15, 0.20, 0.15]: To construct two samples with one identical distribution, we set the same sample fraction (0.10, 0.20. 0.10, 0.20. 0.20, 0.10] for two samples. We set the sample size for training to 2200 and for testing to 550.
kropt is categorical dataset with 18 classes, where we only consider categories 13, 14, 15 and 16 which have the majority of the data.. For constructing two different samples, we set the sample fraction for one sample to 00.15, 0.2, 0.3, 0.35] and for the other sample to [0.35, 0.35, 0.15, O. 15]: To construct two samples with one identical distribution, we set the same sample fraction [0.25, 0.25, 0.25, 0.25) for two samples. We set the sample size for training to 2000 and for testing to 500.
diamon (diamonds) a regression dataset with continuous target variables. Based on the sorted target variables, we divide the data into 4 equal parts. For constructing two different samples, we set the sample fraction for one sample to (0.35, 0.2, 0.2, 0.25) and for the other sample to [0.2, 0.3, 0.3, 0. To construct two samples with one identical distribution, we set the same sample fraction [0.25, 0.25, 0.25, 0.25] for two samples. We set the sample size for training to 2000 and for testing to 500.
32

---

Table 4: Optimization parameters of MEMaBI test for different datasets.
Dataset
# Test Locations
Learning Rate
Optimization Epoch
blob
17
0.007
1000
dna 15
0,0004
1000
agnos
15
0.001
200
topo21
15
0.001
1000
har
15
0.001
100
kropt
0.002
1000
diamon
0.0013
1000
cifarl0 2
0.003
200
mnist
0.001
500
santan 10
0.001
1000
codrna
0.01
1000
sea50
0.01
1000
adult
0.001
1000
labor
0.001
1000
poker 3
0.002
200
higgs
15
0.001
1000
Original cifar10 (samples P) is compared adversarial-cifarl (samples Q) following [19], where adversarial-cffar is constructed by [86] for distribution shift detection. To construct two samples with one identical distribution, we select randomly between original cifar10 and adversarial-cifarl and drawn two samples from the same dataset. The image is scaled 32 32 and we set the sample size for training to 200 and for testing 50.
mnist contains 70000 handwritten digit images, compare true mnist data (samples IP) to Fake-mnic data (samples Q) following [39]. where the Fake-mnis data is drawn from a pre-trained deep convolutional generative adversarial network [87] and the image is scaled to 32 x 32. To construct two samples with one identical distribution, we select randomly between original mnist and Fake-mnin and drawn two samples from the same dataset. We set the sample size for training to 600 and for testing to 150.
santan (snntandercustomerrstiiifcciiiie ing two different samples, set the sample fraction for sample J0.8, 0.2] and for the other sample to [0.25, 0.75]; ; To construct two samples with one identical distribution, we set the same sample fraction [0.5, 0.5] for two samples. We set the sample size for training to 1000 and for testing to 250.
is a categorical dataset with 2 classes. For construct-
codra a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for sample to [0.7, 0.3] and for the other sample to [0.35, 0.65]; To construct two samples with one identical distribution, we set same sample fraction [0.5, 0.5] for two samples. We set the sample size for training to 1000 and for testing to 250.
sea50 is categorical dataset with 2 classes. For constructing two different samples, set the sample fraction for one sample to [0.7, 0.3] and for the other sample to [0.3, 0.7]: To construct two samples with one identical distribution, for two samples. We set the sample size for training to 5000 and for testing to 1250.
set same sample fraction [0.5, 0.5]
adult is a categorical dataset with classes. For constructing two different samples, we set the sample fraction for one sample to [0.6, 0.4] and for the other sample to [0.4, 0.6] To construct two samples with one identical distribution, we set same sample fraction [0.5, 0.5] for two samples. We set the sample size for training to 5000 and for testing to 1250.
labor is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0.6, 0.4] and for the other sample to [0.45, 0.55]; To construct two samples with one identical distribution, we set same sample fraction [0.5, 0.5] for two samples. We set the sample size for training to 5000 and for testing to 1250.
poker a categorical dataset with classes. For constructing two different samples, we set the sample fraction for one sample to [0.2, 0.8) and for the other sample to [0.8, 0.2); To construct two samples with one identical distribution, we set same sample fraction (0.5, 0.5] for two samples. We set the sample size for training to 3000 and for testing to 750.
33

---

Algorithm 2 MEMBBID training
Input: Two training samples X and Y, number of test locations optimization epoch K, step size Output: [0:Ki=> {M, fra)- and F
1: Initialize (0:) { [M)) La and (rr)== by Eqn. (36)
2: for k= 1, K do
3: Calculate the embeddding training samples X and Y based on Eqn (2)
4: Calculate the statistic T(X, Y) based on Eqn. (4)
5: Calculate gradient ,. T(X, Y).V,, T(X, Y) and Va, T(X, P) via Eqns. (32), (34), (35)
6: Gradient ascend based on Adam optimization method with step size 7
7: end for
8: Calculate the embeddding training samples X and Y based on Eqn (2)
9: Calculate the pooled covariance matrix 2 Xx based on Eqn. (3)
10: Calculate the Schur decomposition of the pooled covariance matrix: LL E
11: Calculate the inference direction F based on Eqn. (6)
12: return (w: {Mi {m)-= and F
higgs is categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0.0, 1.0] and for the other sample [1.0, 0.0]; To construct two samples with one identical distribution, we set same sample fraction [1.0, 0.0] for two samples. We set the sample size for training to 16000 and for testing 4000.
In optimization, we adapt Adam optimization method from the pytorch library in python [88, 89). Table 4 presents the details of the hyperparameter settings for each dataset, including the number of test locations, learning rate.
Experimental settings
At initialization, we usually set the Mahalanobis matrices {M, to identity matrices. We further provide an alternative initialization for the Mahalanobis matrices based on the correlation information of two training samples as follows:
m-
M,
Σ S., (s,) cs (/s,)//c /mm 1) + 5I.
where S = {x Ug 3m 8z y1 - U...... Un u,}. Sj. is the i-th element of Sj, and CS = EieT Sj.// (m + n)). I, is an identity matrix of size d x d to guarantee the positive definiteness with small constant 5 > O. Here, the elements and y come from training samples X and Y, and m and n are the numbers of elements in X and Y respectively.
For initialization of test locations, we first fit Gaussian distribution for each sample and draw half of the number test locations from each distribution. This could be expensive for high dimensional dataset, and
simplify the process by directly sampling test locations from original dataset.
For initialization of the bandwidth parameters of Mahalanobis kernels, let ?j y for every E [4] and then linearly search for that maximize the statistic from candidate list with fixed test locations and Mahalanobis matrices, which can be formalized as follows:
7+ argmax,, T(x, Y) for 7 E (med2.2- 2 med2.2-3.. 3.6
med8.2)) 24), (36)
where med denotes the median of pairwise Euclidean distances of points in X and Y.
Compared methods
We now present the details for our compared methods as follows:
ME?: The mean embeddding method learns set of test locations and a single Gaussian kernel, and then measures the difference between two mean embedddings with a statistic following x" distribution [9, 10];
The code is downloaded from www.lwwwwwowvitteeoaaaneereeeeee
34

---

Algorithm 3 MEMBBID testing
Input: Two testing samples X' and Y' test locations (D:) Mahalanobis kernels parameters
(M,)== and (Ni)==1: inference direction F, significance level a and parameter /3
Output: h
1: Calculate the embeddding testing samples X and Y based on Eqn. (2) tri"
2: Calculate mean embeddings cx. - - Ø//m' and Cy. 2j=1 91/n'
3: Calculate the pooled covariance matrix Σ E based on Eqn. (3)
4: Calculate the Schur decomposition of the pooled covariance matrix: L'L' 8
5: if FF Cgr) 2 0 then
6: h = [Ixx Y)) [o]]
7: else
8: h (T P)) s (2 3)a]
9: end if
return
Algorithm 4 Exploring the local significant differences
Input: Two testing samples YY Mahalanobis kernels' parameters {M:) and fri test locations (v: partition tree T
Output: 8*
Calculate the embeddding testing samples X' and y based on Eqn. (2)
2: Get the local two samples X's and Y/g, for every E [s] based on partition Tree
3: Calculate bi-directional masked p-value 9 B, Y%, for each rectangle region B;
4: Resort rectangle regions B()) B(2)...... BBa) in non-increasing order based g(::))
5: Calculate bi-directional hypothesis (XX B, Yg,) for each rectangle region B; by Eqn. (7)
6: Calculate the index set t" based on Eqn. (11)
7: return
C2ST-S': : binary classification neural network trained and the statistic is computed as the accuracy over a hold-out set of two samples [13];
MMDAgg A solution for the fundamental kernel selection problem involves the aggrega- tion of large number of kernels with several bandwidths, where the incomplete U-statisties are used to measure the difference between two samples Schrab et al. [75]. Notice that we set the testing sample size for MMDAgg to 5 times that of the other methods, since it does not require training:
MMD-D A deep kernel approach for Maximum Mean Discrepancy (MMD), where the parameters of a neural network, two lengthscales of Gaussian kernels and a regularization parameter are optimized [39];
C A binary classification neural network is trained and the statistic is computed as the difference between outputs the logit function corresponding to two samples [14, 18];
AutoML?: A binary classifier is trained based on Automated Machine Learning techniques and the statistic is same to C2ST-L approach [19].
We implement methods for exploring local significant differences in Python, following their respective guidelines as follows:
FDG: Partition the sample space based on probability binning and then compare the car- dinalities of two samples over rectangle regions, where a normalized chi-squared value is computed for each bin to measure the local difference [24];
'The code is downloaded from github.
4 'The code is downloaded from ciilubb..omumtmiiicccbbungggenne
'The code is downloaded from gwilubb.com/fenggiigoooot fSr-TST.
6 TThe code is downloaded from wtllub..co//ycceeng/eee logit
logit_test test.
The code is downloaded from github. httpla/c.oom/kkubbeeetttttLLLLS
35

---

Step Initialization
Step II: Split x-axis median position
Splitting regions
Splitting regions
Partition free
Partition
x
Step Split mode median position
Step IV: 1) Split mode at y-axis with medium position
2) Split node ar y-asis with median position
2) Split node x-axis with median position
3) Split node at x-axis with modian position
4) Split at y-axis median position
Partition tree
Splitting regions
Partition
Splitting regions
7
y
15
8
11
12
Figure 9: An illustration for partition tree with splitting regions.
K-PRIM: Partition the space based on patient rule induction method along with the infor- mation of kernel density estimation, and statistic following xi distribution to identify
local distributional difference [28];
MRS: Partition the sample space based on pólya tree method, which splits leaf node based the median of randomly selected feature, and then measure the difference between local two samples based on Binomial distribution [26]:
TEAM: Partition the sample space on the median of the feature with largest sample variance,
and then measure differences between local two samples with Binomial distribution [27];
BTLDD: Estimate the conditional probabilities of two samples based on a regression model and then cluster those elements with significant different conditional probabilities [77];
MMDT: Partition the sample space into multiple equal grids based quantile values of the features, and then estimate the kernel densities two samples and measure local differences based on Welch's two-sample t-test statistic [29].
For the exploration of local significant differences, we take density differences [78] between two samples in a local region as an evaluation measure for local significant differences, and follow the works of [79, 80] based on k-NN density estimator with k = 20. Here, denote by X and Y the available data for density evaluation, we have estimated density functions as follows:
20
20
fx(x) == N va (x( and (x := N va (z)d
where = E [0, 1]d, N denotes sample size for density estimation and va is the volume of unit ball in d Denote by rx (z)d and ry (z)d the distances from to its 20-th nearest neighbors in X and Y. respectively. For density estimation, we use all data diamond, and 1, 000, 000 data other datasets due to the limitation of time complexity.
For our MEMABID test, we present the detailed training procedure in Algorithm 2 and testing procedure in Algorithm 3. Figure 9 is a pictorial illustration to present the region- eegiionspliiiing method and clarify that our partition tree is constructed iteratively: We initiate tree root with embedding space, and during each iteration, we select randomly one of those leaves with the largest size of training data points, and select the feature of the largest statisties value and with the median splitting position. We present the detailed description on the exploration of local significant differences in Algorithm 4.
36

---

Our Manao
MHD
C25T5
CESTL
AUtOMETST
Running (ime (veconds))
2.7
6/91
dna
tspo23
har
diamoo
cttor
mois Datasets
codma
sea50
adulf
labor
poker
hi00s
Figure 10: Comparisons of training time for different methods on two-sample test. Note that y-axis is in log-scale.
Our MEvooe
N
ME
MMD-D
MMDAg9
ZZZ
C2ST-S
SIN
C2ST-
AutoMITSST
10
Running time (seconds)
10"
101
10"
10-1
10-1
blob
dna
agnos
topo21
har
kropt
diamon
dift
mnist
santan
codrna
sea50
adult
labor
poker
higgs
Datasets
Figure 11: Comparisons of testing time for different methods two-sample test. Note that y-axis is in log-scale.
D Experimental comparisons
Running time
For fair comparisons, run all experiments single core without parallel optimizations, and experiments are performed with Python on nodes of a computational cluster with a single CPU (Intel Core 19-109X0X 3.7GHz) and single GPU (GeForce RTX 2080 Ti), rumning Ubuntu with 128GB main memory. For these methods based on deep neural networks, such as C2ST-S, C2ST-L and MMD-D, run experiments on GPU. For other methods except for AutoMLTST, we run experiments on CPU. For AutoMLTST, we run experiments on GPU for cifar and mnist, and run experiments CPU for other datasets.
further compare the average training time and testing time (in seconds) for different methods on two-sample test, shown in Figure 10 and Figure 11. Notice that, the training time of MMDAgg is since it has no training procedure. In testing procedure, our MEMARD and ME methods calculate the rejection threshold based on the asymptotic distribution x² whereas other methods perform permutation test or wild bootstrap.
Additional experimental comparisons
Table 5 presents the comparisons of type-I error for different methods. As can seen, type-I error is limited about 0.05 for all compared methods, which shows the effectiveness of test power in experiments for different methods.
Figure 12 shows the experimental comparisons between Mahalanobis kernels and Gaussian kernels, and it is obvious that Mahalanobis kernels achieves higher test power with better performance by exploiting local regions and directional information. Figure 13 presents experimental comparisons of test power with different training sample sizes. The testing sample size set as 200 for blob, 1000 for higgs, 200 for codrna and 150 for sea50. As can be seen, our MEMARiD test takes better test powers empirically than other methods w.r. different training sample sizes by incorporating correlation and directional information of two samples.
37

---

Table 5: Comparisons of type-I emoor (mean
Dataset
Ours MESsaRD
ME
MMDAgg
MMD-D
C2ST-L
C2ST-S
AutoMLIST
blob
043:: .055
064 056
.061
0644
.037 .032
004: .006
dna
0454 .036
.524 068
.061 i .055
.022 004
.077 .055
.099 .044
.331 .034
ugnostic
0474 .023
0491 .04
.0514 .03
.02:: .004
.422 .029
.634 .045
.044 .039
topo21
0334 .019
0494 .022
.038: .034
049: .039
.019 .022
.051 .024
.084 .024
har
0464 .033
.033 026
.440:: .030
045:: .028
.022 .014
.611 .041
.052. .041
kropt
.033:: .016
.021 .006
.013: .000
.301 .029
.466 .016
.033: .012
.033: .021
diamon
.022 .024
.029 .024
.017 ,012
046 .023
.013 .010
.011 .016
.033 .031
cifar
.055 .010
.012 .011
.025: .014
.088 .668
.021 .016
.027 .017
.020 .014
mnist
0434 .038
.0134 .066
.041: .028
002: 068
.032 .041
.057 .026
.049 .039
santan
037.1 .011
092.1 043
0581
.255
.027
.042.1
codma
0133 .010
.017 .16
.075 .080
0424 .028
.0461 .047
042. .012
.008 .012
sca50
.032+ .027
.01 .012
.088
088 .059
.029
.088 .066
.050: .047
adult
0244: .016
.21 016
.099: .016
.08: .010
.033 016
.033 .026
.062: .220
labor
.077 .016
.077 010
.054
056: .029
.099 .044
.099 006
.042 .026
poker
029: .166
0133 010
.044 .041
078: .047
.29 000
.58 .031
.50 .04
higgs
0464 .011
0244 016
.033 .031
06661: .028
.68 .098
048: .017
.034 .039
ways
Hops
MA
Sumplesee sie
ste
Samp add
Figure 12: The comparisons of test power vs sample size for Mahalanobis kernel and Gaussian kernel.
102 powerr
0555
0513
20
25
20 25
20
de
200
1200
Figure 13: The comparisons of test power with different training sample sizes.
HAMOd pastle
Naric 9o00
Sanck coe
Sample dee
Sample tbe
Sample toe
Figure 14: The comparisons power vs sample size for our bi-directional masked p-value and prior Sidak.
canra
8
166
5
40 &
0
150
256
36a
450
874 750
14
35 65
10s nls 1de Wis
Figure 15: The comparisons of with different sample size and number of test locations.
Figure 14 presents experimental comparisons between our di-directional masked p-value and previous Sidak [67]. As can be seen, bi-directional masked p- pavul achieves higher test power by exploiting the significant level of local difference. Figure 15 analyzes the relationship between probability E = Pr[sgn DE (cx; - 22g )] 2 1) and sample size, by considering different numbers of test locations E {2,6 6, 10, 14, 18). As can be seen. gradually increases the sample size for different number test locations, which shows the necessity of directional information in two-sample test.
Figure 16 analyzes the relationship between the number of local regions and probability < = Pr[sgn Fg L/-1 1 (cx; cv; 2 1). The probability & increases with 8, and hence can effectively partition the space into local regions with different inference directions. Figure 17 presents
38

---

8
1931
Sample
Sample
Sample ibe
Sampre
Figure 16: The comparisons { for partition trees with [2"; 2º The value of increases increases.
MarT Co
15
Figure 17: The correlation between the optimal parameter and probability Pr|sga[F@ L' (cx Cig 1). Here, N denotes sample size.
additional experiments on the relationship between the optimal parameter 3 and the probability & = Prr[F L (cx' - cg.) 0J, as shown in Figure 6.
39


Reviews:
[{"review_id": "LrFb6rjJNf", "reviewer": "Reviewer_DcFi", "summary": "This paper investigates the exploration of local significant differences in the context of two-sample tests. The authors propose a new test called ME_{MaBiD}, which utilizes multiple Mahalanobis kernels and introduces a bi-directional hypothesis to exploit local information. The embedding space is partitioned into rectangle regions based on a new splitting criterion, and local significant differences are identified using the bi-directional masked p-value and the ME_{MaBiD} test. The paper provides theoretical guarantees for the ME_{MaBiD} test, including lower bounds on test power and control of familywise error rate. Extensive experiments validate the effectiveness and efficiency of the proposed methods for two-sample testing and the exploration of local significant differences.", "questions": "In addition to what has been asked in the Weaknesses part. the type-I error rate in Theorem 4, and M_1, \\ldots, M_\\ell in Eq 5 are not defined. For the Construction of partition tree step one, is the leaf node randomly selected, or just select the largest? The paper seems to go with both. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Novel Approach: The paper introduces a new test, ME_{MaBiD}, which offers a novel approach to exploring local significant differences in two-sample tests. The utilization of multiple Mahalanobis kernels and the introduction of a bi-directional hypothesis provide a unique perspective for analyzing local information.\n\nRigorous Theoretical Analysis: The paper provides theoretical guarantees for the MEMaBiD test, including the asymptotic distribution of the proposed statistic with Mahalanobis kernels, lower bounds on test power, and control of familywise error rate. This demonstrates a strong foundation for the proposed method and enhances its credibility.\n\nExploration of Local Significant Differences: The paper addresses the important aspect of identifying local significant differences, going beyond traditional two-sample testing. By partitioning the embedding space into rectangle regions and introducing the bi-directional masked p-value, the paper offers a comprehensive framework for discovering and understanding local variations.\n\nExtensive Experimental Validation: The paper conducts extensive experiments to validate the effectiveness and efficiency of the proposed methods. The superior performance of the ME_{MaBiD} test on various datasets for two-sample testing and the exploration of local significant differences demonstrates the practical value and applicability of the proposed approach.\n\n------------------------after rebuttal------------------------\nI'm leaning towards improving my evaluation due to the authors' commitment to enhancing clarity, which directly addresses my concern. Their responses have clarified the unique contributions of their work, particularly in exploring local significant differences, while their explanations on theoretical foundations underscore the method's credibility. ", "weaknesses": "In general, I think the weakness lies mainly in clarity. It fails to give a formal definition of the input and output of the problem and defined further what is counted as local or not. Without such a formal definition, it would take more time to go into the details of the paper. Another issue with clarity is that the paper does not provide enough reasoning on why the proposed techniques are working towards the goal of the paper. For example, in Lines 67-68, why do Mahalanobis kernels exploit more information than the Gaussian kernels? In Eq.7 why the two directions are necessary, but they are not used in the exploration of local significance differences? For Lemma 1, would the learned \\v_j satisfy the condition of drawn i.i.d. from some distribution? Why the alternative hypotheses H1 and H2 are important and practical? These factors are important to accurately evaluate the contribution of the paper but are not sufficiently discussed. Without such discussion,  the paper's main contributions seem to replace the Gaussian kernel with the Mahalanobis kernel and one direction test by two-directions, which is novel but may not be significant enough considering the motivation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Q1] I think the weakness lies mainly in clarity. It fails to give a formal definition of the input and output of the problem and defined further what is counted as local or not.\n\n[A1] We will present a formal definition of the problem, and clarify that the input includes two i.i.d. samples from two distributions, and the output for two-sample test is a boolean value to show whether two samples are from an identical distribution; and the output for local explorations includes some local regions with significant differences. We will also clarify that local explorations imply some local regions (i.e., leaves of partition tree) with significant differences, as in (Soriano and Ma, 2017).\n\n&nbsp;\n\n[Q2] in Lines 67-68, why do Mahalanobis kernels exploit more information than the Gaussian kernels? \n\n[A2] We will clarify that multiple Mahanalobis kernels flexibly exploit local differences from different neighborhoods and feature maps, and adapt to variations of distributional smoothness and shape, whereas Gaussian kernels focuses on  an isotropic scale in all directions, as shown in Figure 1.\n\n&nbsp;\n\n[Q3] In Eq.7 why the two directions are necessary, but they are not used in the exploration of local significance differences?\n\n[A3] We will clarify that two directions could improve test power by incorporating directional information between mean embeddings of two samples, as shown in Figure 4, and it is more adaptable to different datasets than previous one-directional test (McIntosh, Amer. Stat. 2022) and p-value test without direction (Scetbon & Varoquaux, NeurIPS2019; Wynne & Duncan, JMLR2022), by selecting parameters between two directions. We will also clarify that two directions are used for bi-directional masked p-value and local hypothesis test in the exploration of local significance differences.\n\n&nbsp;\n\n[Q4] For Lemma 1, would the learned \\v_j satisfy the condition of drawn i.i.d. from some distribution? Why the alternative hypotheses H1 and H2 are important and practical?\n\n[A4] We will clarify that the i.i.d. condition for v_j has been a classical assumption in two-sample test (Chwialkowski et al., NeurIPS2015; Sectbon et al., NeurIPS2019), and the alternative hypotheses H1 and H2 are helpful to test the differences between two samples from two directions, and improve the lower bound of test power theoretically (Theorem 4).\n\n&nbsp;\n\n[Q5] … the type-I error rate in Theorem 4, and M_1, \\ldots, M_\\ell in Eq 5 are not defined. For the Construction of partition tree step one, is the leaf node randomly selected, or just select the largest?\n\n[A5] We will clarify that type-I error rate denotes the probability of falsely rejecting the true null hypothesis, and M_1, \\ldots, M_\\ell are positive-definite Mahalanobis kernels defined by Eqn. (1). We also clarify that we select randomly one of the largest leaves in the construction of partition tree.\n\n&nbsp;\n\nWe will improve the clarity of this work with some defintions and discussions according to your suggestions. Thank you."}, {"review_id": "KrJEF127kc", "reviewer": "Reviewer_42S6", "summary": "In recent years, the focus on two-sample testing has significantly increased due to its wide range of practical applications. This study advances this exploration by highlighting local significant differences in two-sample testing. The authors introduce the ME, a robust test for two-sample testing that leverages local information through multiple Mahalanobis kernels and proposes a bi-directional hypothesis for testing. To pinpoint local significant differences, the embedding space is initially partitioned into several rectangular regions using a novel splitting criterion tied to test power and data correlation. Following this, local significant differences are identified using the bi-directional masked-value in conjunction with the ME test. The authors provide theoretical guarantees for their ME test, including lower bounds on test power, and effectively control the familywise error rate when investigating local significant differences. The effectiveness of the proposed methods in two-sample testing and the examination of local significant differences are then validated through extensive experiments.", "questions": "See the weakness above.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on an important problem. Two-sample testing techniques have an important role in detecting distribution changes. T\n\n2. The proposed method focuses on the local difference between two distributions, which is a promising research direction based on existing two-sample testing techniques. Recent testing techniques focus on learning good representations of data via maxing test power (Liu et al., 2020), which is another way to look at different areas with different weights but more flexible (bringing some issues). This paper proposes an effective way to implement this kind of technique.\n\n3. The experiment is solid, verifying that the proposed method is good and effective.", "weaknesses": "1. There should be more discussion in the introduction. To show the difference between the deep-kernel-based method and your proposed method. I can understand the merit of the proposed method, but more discussions are needed in this paper.\n\n2. The motivation to use Mahalanobis kernels is still not clear. Can we have more choices? What is your selection standard for kernels?\n\n3. Procedures of permutation test should be introduced in preliminary. it is better for readers to understand the procedures of two-sample testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Q1] To show the difference between the deep-kernel-based method and your proposed method, I can understand the merit of the proposed method, but more discussions are needed in this paper.\n\n[A1] We will add more discussions that Liu et al. (ICML2020) presented a deep kernel to adapt for variations in distribution smoothness and shape, with solid theoretical guarantees, and it is especially applicable to high dimensions and complex data, while our work takes a different way to exploit local distribution variations based on multiple Mahalanobis kernels from different neighborhoods and feature maps.\n\n&nbsp;\n\n[Q2] The motivation to use Mahalanobis kernels is still not clear. Can we have more choices? What is your selection standard for kernels?\n\n[A2] We will clarify that multiple Mahanalobis kernels flexibly exploit local differences from different neighborhoods and feature maps, and its analytic property (Theorem 2) ensures that the difference between two distributions can be detected almost surely at a finite number of randomly chosen locations (Chwialkowski et al., NeurIPS2015). There are different choices for kernels, such as Gaussian kernel, Laplacian kernel, deep kernel, etc., while our selection standard includes effectiveness, computational costs, kernel analyticity property, etc.\n\n&nbsp;\n\nWe will introduce the permutation test (Sutherland et al., ICLR2017) in preliminary for readers, and improve this work according to your suggestions. Thank you."}, {"review_id": "9rpD2H9MQh", "reviewer": "Reviewer_CMa2", "summary": "The paper introduces a two-sample test that explores local variations in distribution. The authors provide a motivation for this specific formulation by highlighting two practical scenarios in cosmology and biology. They proceed with a literature review and conclude that none of the existing approaches is fully satisfactory. Guided by the problem's requirements and their intuition they incorporate space partitions, Mahalanobis kernels, and bi-directional hypothesis testing in developing the new test. The proposed algorithm is experimentally compared with existing methods, and it is found to empirically outperform them.\n", "questions": "I suggest authors improve the quality of the writeup.  ", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is quite novel in both its motivation and the algorithm's construction. It is commendable that authors justify further specialization of two-sample tests ( \"it is necessary to take one more step to find and understand local significant differences, rather than only two-sample test\" ), by invoking two compelling practical problems. \n\nI am impressed by the unorthodox design of this test, which incorporates methods from various sub-fields to create an algorithm that addresses the requirements and and seems to perform well in practice.", "weaknesses": "It is unclear which elements of the design (bi-directional p-values, tree partition, kernel design) contribute to overall performance of the algorithm. \n\nClarity of the writeup goes down starting from the \"Construction of partition tree\" section. A few examples:\n\n* It is not explained why authors need masked p-value.\n* Why is it important to guarantee equal probabilities in partitioned rectangle regions?\n* Theorem 5 makes assumptions that I don't understand \".. if p-values are mutually independent for regions without difference, and are independent to regions with difference\". Does your test control FWER?\n\nFinally,  Bi-directional hypothesis for testing looks like designing non-centered rejection regions for the L^{-1}(c_{\\hat x} - c_{\\hat y}) (you don't have to that L2 norm of the embedding)\n\n\nedit: I have read the rebuttal, and now I have less confidence that the clarity of the paper will improve. I am also more worried that the lack of clarity hides technical flaws. Moving  to 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Q1] It is unclear which elements of the design (bi-directional p-values, tree partition, kernel design) contribute to overall performance of the algorithm.\n\n[A1] We will clarify that the bi-directional p-values is proposed to improve test power by exploiting directional information between mean embeddings of two samples, as shown in Figure 4, and it is more adaptable to difference datasets than previous one-directional p-value (McIntosh, Amer. Stat. 2022) and non-directional p-value (Scetbon & Varoquaux, NeurIPS2019; Wynne & Duncan, JMLR2022), by selecting proper parameters.\n\nWe will also clarify that we take partition tree to explore local significant difference following (Soriano and Ma, 2017; Pura et al., 2023), but with a new splitting criterion relevant to test power and data correlation; therefore, our method takes better performance on the exploration of local significant difference empirically, as shown in Table 4.\n\nWe finally clarify that kernel has been a standard technique for two-sample test such as Gaussian Kernel (Gretton et al., NeurIPS2012; Jitkrittum et al., NeurIPS2016) and deep kernel (Liu et al., ICML2020), and our work takes multiple Mahanalobis kernels to exploit flexibly local differences from different neighborhoods and feature maps.\n\nHence, the bi-directional p-values and kernel are helpful to improve the performance of two-sample test, and tree partition and kernel are helpful to improve the performance of exploration of local significant difference.\n\n&nbsp;\n\n[Q2] Clarity of the writeup … why authors need masked p-value. Why is it important to guarantee equal probabilities in partitioned rectangle regions?\n\n[A2] We will clarify that masked p-value is used to select the region of the largest local difference, and it is helpful to improve the test power of local hypothesis test. We will also clarify that equal probabilities could yield partition regions with balanced data points, and such partition takes better performance than regular grids, as shown empirically in (Boracchi et al., 2017; Liu et al., 2021).\n\n&nbsp;\n\n[Q3] Theorem 5 makes assumptions that I don't understand \". if p-values are mutually independent for regions without difference, and are independent to regions with difference\". Does your test control FWER?\n\n[A3] We will clarify that the assumptions (Theorem 5) follow previous studies (Lei and Fithian, 2018; Duan et al., ICML2020), that is, 1) the p-values of local regions without differences are mutually independent; and 2) the p-values of local regions with differences are independent to those p-values of local regions without differences. We will also clarify that our test could upper bound the FWER with a given significant level under those assumptions, as shown in Figure 6 empirically.\n\n&nbsp;\n\n[Q4] Bi-directional hypothesis for testing looks like designing non-centered rejection regions for the L^{-1}(c_{\\hat x} - c_{\\hat y}) (you don't have to that L2 norm of the embedding)\n\n[A4] We will clarify that Bi-directional hypothesis essentially designs non-centered rejection regions in direction L^{-1}(c_{\\hat x} - c_{\\hat y}), and the L2 norm gives an alternative expression for optimization statistics T(\\hat{X},\\hat{Y}), and maximizing the L2 norm is essential to maximize the test power for two-sample test w.r.t. fixed sizes of two samples.\n\n&nbsp;\n\nWe will improve the clarity of this work according to your suggestions. Thank you."}, {"review_id": "074RGpfJxh", "reviewer": "Reviewer_wXPw", "summary": "This paper proposes a bidirectional two-sample test that can address both positive and negative gaps from one sample to the other. The test is proposed based on Hotelling's statistic, and the direction information is encoded via the mean difference. By searching test locations and an adequate Mahalanobis kernel, and more importantly, by detecting the local inference, the test exhibits a higher testing power than the baseline. ", "questions": "Do you split the dataset into a training set and a test set, optimize the kernel parameters and test locations and split the region base on the training set, and then perform the two-sample test based on the test set? If so, how does the size of the training set impact the testing power, and do you have experimental results for different sizes of training sets? ", "limitations": "The following papers propose two-sample tests by detecting local differences inexplicitly. \n\n1. Duan, Boyan, Aaditya Ramdas, and Larry Wasserman. \"Interactive rank testing by betting.\" Conference on Causal Learning and Reasoning. PMLR. \\\n2. Li, Weizhi, et al. \"A label efficient two-sample test.\" Uncertainty in Artificial Intelligence. PMLR, 2022.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\\\nThe proposed test is novel. \n\n**quality**\\\nThe authors spent great effort delivering theorems and performing thorough experiments to validate the test. \n\n**clarity**\\\nThe writing and the structure of the paper are adequately clear, though it can be improved in some parts. \n\n**significance**\\\nThe proposed test has extensive applications in experimental design and observation studies, especially for the case where the global difference is small but the local difference is large. \n", "weaknesses": "**Tradeoff between local difference and overall sample complexity**\\\nThe testing power typically depends on the problem's difficulty and the sample complexity. An easier problem and higher sample complexity lead to a higher testing power for a two-sample test. One important idea of the proposed test is to shift the testing focus to the local region instead of global, and hence potentially change the original testing problem to an easier one. The price to pay is the sample complexity used to compute the statistic for a region is certainly smaller than the sample complexity used for a global test. The paper seems not to address the tradeoff between the decreasing problem's difficulty and the decreasing sample complexity used to compute the statistic. \n\n**Clarity**\\\nThe authors could use pictorial illustrations for presenting the region-splitting method; right now it is not super clear to me how the method work. In addition, the paper could include a problem formulation section to present what's the null and alternative hypothesis here; the introduction of the local test sort of blurs the typical setting for the two-sample test. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Q1] Tradeoff between local difference and overall sample complexity … seems not to address the tradeoff between the decreasing problem's difficulty and the decreasing sample complexity used to compute the statistic.\n\n[A1] We will clarify that we take the number of Mahalanobis kernels and number of leaves of partition tree to balance problem's difficulty and sample complexity, as in (Soriano and Ma, 2017; Pura et al., 2023). For a difficult problem, we require more Mahalanobis kernels and more leaves of partition tree, which increases overall sample complexity. In experiments, we take cross validation for different datasets to select proper parameters on such tradeoff.\n\n&nbsp;\n\n[Q2] The authors could use pictorial illustrations for presenting the region-splitting method; right now it is not super clear to me how the method work.\n\n[A2] We will add a pictorial illustration to present the region-splitting method (Figure 1 in the uploaded PDF file), and clarify that our partition tree is constructed iteratively: We initiate tree root with embedding space, and during each iteration, we select randomly one of those leaves with the largest size of training data points, and select the feature of the largest statistics value and with the median splitting position. Algorithms 3 and 4 present the detailed descriptions for the region-splitting method in Appendix E.\n\n&nbsp;\n\n[Q3] Do you split the dataset into a training set and a test set …how does the size of the training set impact the testing power, and do you have experimental results for different sizes of training sets?\n\n[A3] We will clarify that we split every dataset into a training set (80%) and a test set (20%) as done in (Kübler et al., NeurIPS2020), and the details are given in Appendix B. We will add a figure (Figure 2 in the uploaded PDF file) to show the experimental results for different sizes on four datasets, and the trends are similar for other datasets. As can be seen, our method takes better test powers empirically.\n\n&nbsp;\n\n[Q4] The following papers propose two-sample tests by detecting local differences inexplicitly … 1. Duan, Boyan, Aaditya Ramdas, and Larry Wasserman … 2.Li, Weizhi, et al. …\n\n[A4] We will add two relevant papers (Duan et al. 2022; Li et al. 2022) by detecting local differences inexplicitly, and clarify that Duan et al. (2022) introduced the interactive rank test by betting, while Li et al. (2022) considered a different setting where sample features are easily measured yet sample labels are unknown and costly to obtain.\n\n&nbsp;\n\nWe will add a problem formulation section to introduce local test, null and alternative hypothesis, etc. We will improve this work according to your suggestions, Thank you."}]

Author rebuttal:
We want to express our gratitudes to reviewers for their insightful comments, and we will improve the paper accordingly. 

We present the detailed responses for each review (below), and upload a PDF file with two figures, as suggested by a reviewer.

Meta-review: The paper studied two-sample test and focused on local significant differences to increase the test power. The paper has strong motivation and contributions. After the rebuttal and author-reviewer discussions, there were three positive (5, 6, 7) and one negative (2) reviewers. The rating of the negative reviewer was originally 3 before the rebuttal and then 2 after the rebuttal, which looks like an outlier to me. The major concern from this reviewer was clarity, which I think might be an issue but would not be a major issue. All the four reviewers recognized its novelty and impact. Thus, we should accept the paper for publication.

The only negative reviewer was especially unhappy about the name bi-directional hypothesis, but it is just a natural extension of the existing names non-directional hypothesis and one-directional hypothesis following previous research papers in the same research area. The authors should carefully introduce the research background: when a concept has two or more names (for example, one-directional hypothesis, one-sided hypothesis, and one-tailed hypothesis), all related names must be covered; then, you should explain why you adopt the naming convention of a specific paper but not that of another paper.
Decision: Accept (poster)