Anchor-based Maximum Discrepancy for Relative Similarity Testing
Authors: Zhijian Zhou
Liuhua Peng
Xunye Tian
Feng Liu
Venue: NeurIPS 2025 poster ()
Keywords: relative similarity testing
kernel-based hypothesis testing
deep kernel
Primary area: other

Abstract: The relative similarity testing aims to determine which of the distributions, $P$ or $Q$, is closer to an anchor distribution $U$. Existing kernel-based approaches often test the relative similarity with a fixed kernel in a manually specified alternative hypothesis, e.g., $Q$ is closer to $U$ than $P$. Although kernel selection is known to be important to kernel-based testing methods, the manually specified hypothesis poses a significant challenge for kernel selection in relative similarity testing: Once the hypothesis is specified first, we can always find a kernel such that the hypothesis is rejected. This challenge makes relative similarity testing ill-defined when we want to select a good kernel after the hypothesis is specified. In this paper, we cope with this challenge via learning a proper hypothesis and a kernel simultaneously, instead of learning a kernel after manually specifying the hypothesis. We propose an anchor-based maximum discrepancy (AMD), which defines the relative similarity as the maximum discrepancy between the distances of $(U, P)$ and $(U, Q)$ in a space of deep kernels. Based on AMD, our testing incorporates two phases. In Phase I, we estimate the AMD over the deep kernel space and infer the potential hypothesis. In Phase II, we assess the statistical significance of the potential hypothesis, where we propose a unified testing framework to derive thresholds for tests over different possible hypotheses from Phase I. Lastly, we validate our method theoretically and demonstrate its effectiveness via extensive experiments on benchmark datasets. Codes are publicly available at: https://github.com/tmlr-group/AMD.

Anchor-based Maximum Discrepancy for Relative Similarity Testing
Zhijian Zhou, Liuhua Peng, Xunye Tian, Feng Liu"
The University of Melbourne, Australia (zhijianzhou. ml, xunyetian.m ml, fengliu.ml| @ gmmai..com liuhua.peng @niielbb.du..a
Abstract
The relative similarity testing aims to determine which of the distributions, IP or Q is closer to anchor distribution U. Existing kernel-based approaches often test the relative similarity with a fixed kernel in a manually specified alternative hypothesis, e.g., Q is closer U than P. Although kernel selection is known be important to kernel-based testing methods, the manually specified hypothesis poses significant challenge for kernel selection in relative similarity testing: Once the hypothesis is specified first, we can always find a kernel such that the hypothesis is rejected. This challenge makes relative similarity testing ill-defined when we want to select good kernel after the hypothesis is specified. In this paper, we cope with this challenge via learning proper hypothesis and kernel sinultaneously, instead of learning kernel after manually specifying the hypothesis. We propose an anchor-bate based maximum discrepancy (AMD), which defines the relative similarity as the maximum discrepancy between the distances of (U, PP) and (U, Q) in space of deep kernels. Based on AMD, our testing incorporates two phases. In Phase I, we estimate the AMD over the deep kernel space and infer the potential hypothesis. Phase II, we assess statistical significance of the potential hypothesis, where we propose a unified testing framework to derive thresholds for tests over different possible hypotheses from Phase I. Lastly, we validate our method theoretically and demonstrate its effectiveness via extensive experiments on benchmark datasets. Codes are publicly available at: https: //ti:////tthbbb cwwccoommmm///e ttrrtmmlreae group/ AMD.
Introduction
Assessing differences between distributions is an important research area in machine learning. especially for tasks that often involve various distributions. c.g., different training and test data distributions in many real-world scenarios [1]. For such tasks, two-sample testing has been developed as principled method to assess whether two distributions are identical [2]. However, it becomes less effective when the task requires comparing three or more distributions. For example, in scenarios where multiple models are available for given test dataset, selecting the model trained on data most similar the test data is essential for optimal performance. Additionally, in studies with generative data, including those arising from generative models [3], adversarial attacks [4], data augmentation [5] and machine generated text detection [6], key challenge to assess relative similarity between different generative data distributions and the original data distribution.
Fortunately, relative similarity testing provides framework assess which of the distributions, IP or Q is closer to the anchor distribution U; in practice, these distributions are typically unknown, and we can only observe samples from them. Existing methods evaluate the relative similarity between the pairs (U, P) and (U, by comparing the discrepancy between their distances, d(U, IP) and d(U, Q).
*Correspondence to Feng Liu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A popular distance is the integral probability metric (IPM) [7. 8), which defines the distance between two distributions as the maximum discrepancy between their expectations over a specified class of functions, offering strong theoretical guarantees and adaptability across various scenarios through the choice of function classes. The kernel-based maximum mean discrepancy (MMD) an instance of IPM 19, 10], providing an effective measure for assessing distributional discrepancy [2, 11, 12], and is well-suited for applications in relative similarity testing [13, 14J. Some methods use kernelized Stein discrepancy, another instance of IPM, to evaluate similarity in cases where U is known [15-18].
Prior to conducting hypothesis testing, it is typically necessary to formulate the null and alternative hypotheses. Previous methods for relative similarity testing follow this procedure by manually specifying a relationship in the alternative hypothesis and performing the test accordingly. For example, given the specified relationship P( > d(U, Q), they define the hypotheses as follows
H¹: : P( S d(U, Q) and H1 d(U,P) > d(U, Q);
and then the difference between distances, i.e., d(U, P) d(U, Q): is estimated and compared against positive threshold to test the prespecified relative similarity relationship [13]. Consequently, instead of testing which of or Q is closer to U. these methods evaluate whether Q is closer to U than P.
Influence of Manually Specified Hypothesis in Oriented Test. This type of oriented test focuses on evidence supporting d(U, IP) > d(U, Q), and potentially neglects cases where d(U, S d(U, Q). We consider a scenario where d(U, P) S d(U, Q) and analyze the probability of correctly identifying this relative similarity relationship. It is evident that e is equal to 1 1- a in the oracle hypothesis setting, which corresponds to the probability of accepting the null hypothesis Hig given the significance level a. This implics that there is a probability of a of neglecting the case P(U, P) d(U, Q), regardless of the amount of data the kernel used for testing. Thus, it is crucial to construct an alternative hypothesis that reflects the potential true relative similarity relationship. which is unknown in practice. Without prior knowledge, randomly selecting an alternative hypothesis results in a success rate of merely 0.5.
IP) S d(U, Q)
Kernel Selection Challenge for Existing Relative Similarity Tests. More importantly, in kernel- based test, it is critical to select a kernel to effectively capture the evidence that supports the alternative hypothesis [19], and the relative similarity relationship may vary with the choice of kernel from the same family, such as Gaussian kernels. For instance, with a Gaussian kernel KA., the relationship d(U,P) > d(U, @) may hold, while with Gaussian kernel KB, d(U,,) < d(U, Q) holds instead. Given this, after we specify a relative similarity relationship in alternative hypothesis as in Hi, the method then tends to select kernel that best supports d(U, P) > d(U, Q). This deviates from the primary goal of testing which distribution, P or Q. is closer to the anchor distribution U. A median heuristics selects the Gaussian kernel bandwidth by averaging median distances between samples from U and IP, and those from U and Q [13], but it has no guarantees of effective performance in various scenarios [20, 19] with complex data (e.g., images) [21, 22]. Moreover, it restricts the use of more expressive kernels, such as deep kernel [21] and Mahalanobis kernel [23].
Our Solution. In this paper, we cope with the above challenges via learning proper hypothesis and a kernel simultaneously, instead of learning a kernel after mamually specifying the hypothesis. We propose an anchor-based maximum discrepancy (AMD), which defines the relative similarity as the maximum discrepancy between the distances of (U, and (U, Q) in a space of deep kernels (motivated by the definition of IPM). This approach inherently incorporates kernel selection and avoids a selection towards a prespecified relative similarity relationship, e.g., d(U, > d(U, Q). Based on the AMD metric, our relative similarity testing incorporates two phases
In Phase (Section 3.2), we estimate the AMD by selecting the kernel that maximizes the discrepancy between the distances of (U, and (U, Q) from a kernel space. Given this estimation, we infer the potential relative similarity relationship, that is. which distribution, IP or Q. is closer to IU, instead of manually specifying an interested relationship. Here, to ensure a sufficiently rich function space and enhance adaptability across various tasks, we use deep kernels built upon neural networks [21] and propose an optimization algorithm to estimate the AMD with augmented samples, which improves generalization and reduces overfitting for small sample sizes [24-26].
In Phase II (Section 3.3), to assess the statistical significance of the relative similarity relationship identified in Phase 1, we perform the AMD test with the selected kernel from Phase I. In this procedure, we consider the tests on two possible relative similarity relationships, i.e., IP is closer U or Q is closer to U. For this, we propose a unified testing framework to derive the testing thresholds for both cases using the wild bootstrap method.
2

---

Theoretically, our AMD serves as valid metric, and its estimation with data augmentation is proven to exhibit consistency. Besides, we analyze the advantage of our AMD test test power, while ensuring that the type-I error is bounded at the significance level. Empirically, we validate the AMD test against state-of-the-art methods on benchmark datasets, and further conduct ablation studies to evaluate the contribution of each component. We also illustrate practical applications of our AMD test in comparing model performance across different datasets relative to a reference dataset.
2 Preliminaries
Integral Probability Metric. Let P and Q denote two Borel probability measures over a space X S Rd. For a class of functions, the integral probability metric (IPM) [7] is defined as the distance between the distributions P and Q,
[F) sup FEF | -ape-sse f-0
This framework is particularly relevant in two-sample test, which aims assess the equality between two unknown distributions. Numerous approaches have been proposed to find distance measures of this form, aimed at distinguishing between various distributions. Among them, a widely used approach is the kernel-based Maximum Mean Discrepancy (MMD), which corresponds to an IPM defined over the function class H., i.e., the Reproducing Kernel Hilbert space (RKHS) of a kernel X' x X' + R. The MMD serves as an effective distance for measuring the similarity between two distributions, satisfying (MD(P, Q: K)
if and only if P Q for characteristic kernels [9].
Relative Similarity Testing with MMD. Using the squared MMD as similarity measure and given the anchor Borel probability measure U over X' C Rd, Bounliphone et al. [13] test whether Q is closer to U than P by formulating the hypotheses as follows:
HO MMD PR)) < MMD²(U, ()) and Hi MMD? P; > MMD (U, R)
Given the alternative hypothesis, the test statistic is defined as MUD x.) (U, which can be equivalently expressed using the IPM as:
| f /Su
f f
/ /CU /
f 9
sup
sup HE
SEH.
In practice, the three distributions U, P and Q are generally unknown and what we can observe are three samples { Umm X { = pm and Y = (v.) ~ Qml. By estimating
the test statistic using samples, i.e. MMD (Z, - MMD (Z, x)), and comparing against a positive testing threshold corresponding to the given significance level a, this approach effectively focuses on testing evidence in favor of the alternative hypothesis H' Furthermore, in kernel-base testing, it is critical to select a kernel effectively capture the evidence that supports the alternative hypothesis, which leads to selecting a kernel favoured at the prespecified alternative hypothesis.
The AMD Relative Similarity Testing
As demonstrated in Section 1, although kernel selection is known to be important kernel-based testing methods, the manually specified hypothesis poses a significant challenge for kernel selection in relative similarity testing: Once the hypothesis is specified first, we can always find a kernel such that the hypothesis is rejected. This challenge makes relative similarity testing ill-defined when we want select good kernel after the hypothesis is specified. In this section, cope with the above challenges via learning a proper hypothesis and a kernel simultaneously, instead of learning a kernel after manually specifying the hypothesis.
We first introduce the relative similarity measure unchor-based maximum discrepancy (AMD) in Section 3.1; then in Section 3.2, we present the Phase I of AMD test: estimating AMD and inferring potential relative similarity relationship, i.e., learning a proper hypothesis and a kernel simultaneously. Finally, in Phase II (Section 3.3), we test the statistical significance of the relationship from Phase I.
As done in Liu et al. [21]. we assume equal size for three samples to simplify the notation, yet the results in this paper be easily extended to unequal sample sizes following approaches based on the U statisiti [27, 9].
3

---

3.1 The Anchor-based Maximum Discrepancy (AMD)
As introduced in Section 1, IPM is an effective metric measuring the discrepancy between two distributions with strong theoretical guarantees and adaptability across various scenarios through the choice of function classes. Here, we extend it to the relative similarity testing with three distributions, which measures the maximum discrepancy between the distances of (U, P) and (U, Q). Specifically, we define the anchor-based maximum discrepancy (AMD) as follows
Definition 1.
For each kernel K. the kernel family K., we define function fxx (1, MP - (12) H, and introduce the corresponding function space Fx [fr/N E KC). The AMD, i.e., d(U, P. Q; FK), is
/
P Q Ji
MP + uQ
sup f.eFK
JIx dU
sup ae
pp 122) Hn
2
where PAu = EEEw[n( =]] and μaz, lu are defined similarly with no(-, x) H. as the feature map.
In AMD, we measure the relative similarity between distributions by their characteristic kernel mean embeddings (i.e., MM, μp, and μo). These embeddings uniquely represent probability distributions and capture their distinct characteristics for comparison [28]. The reason to choose such an fx. is that the Euclidean-like distances (i.e. MMD distances) m| melll and I Moll are equal in RKHS H, if and only if μoy (μp + p is orthogonal to vector μe Mg. Here, AMD measures relative similarity as the maximum discrepancy within the kernel space K. avoiding oriented kernel selection arising from prespecified relative similarity relationship, discussed in Section 1.
We now prove that our AMD (i.c., Eqn. (I)) is valid metric follows.
Theorem 2. Let M be a set of probability measures over the space X' Rd, and let K. be a kernel space consisting of characteristic kernels. For every U,P, S.P.Q.W W e M, the three random variables ((U,P,Q; d(U,P, W; FK) and d(U, Q ; FK. satisfy:
MU,P,Q;:RL 2 0;
M(U,P,Q:Ke = d(U, Q.P:FK):
d(U,P, (: S d(U,P, FxFK) + d(U, Q WFFk).
3.2 Phase 1: Estimating AMD and Infer the Potential Relative Similarity Relationship
Based on Eqn. (1), AMD can be expressed as d(U, FKQ:FF) = supwex 1dt (U.P, Q with
μ& +
E xx(z))) <(w, x')
k y²)
((
MA
up
ma)
2
2
Hh
where ~ U. x, x' ~ 22 and y. ~ Q2 Here, by the geometric relationship between the vectors μk - (μl + /2)/22 and Hrp - M4G, we interpret the sign of (U, TP, () as an indicator of relative similarity relationship: (d Q)) indicates that TP is closer to U; while sgn(d" (U, Q))
indicates that Q is closer to U.
For unknown distributions U, P and Q with corresponding samples Z = {2,) J=1 U
1m Ipm and Y {y: 3 Qmm we estimate the measure d dd(U,, Q) use the U-statistc estimator, known to be the unbiased estimator with minimum variance [27], as follows
((z,,,,) (z)), <(2,, yj) n(2;, y:) n(x,, x,) + k(y,, 9j) (2)
2o (Z,X,Y) = Σ
2m(m 1)
Estimating the AMD is equivalent to selecting the optimal kernel that maximizes the statistic (Z, Y). which can be formulated as the following optimization objective
Ky E arg max d' (Z, X)I. KEK
The absolute value introduces challenges to optimization. A common strategy address this is to optimize the squared form, i.e. (Z, X, 2)2, which would enhances differentiability and facilitates smoother optimization [29]. However, this method has drawback: it is highly sensitive outliers or

---

data points with large values [30], especially in high- densidy regions. This can overshadow important patterns in lower-density regions, potentially leading to an incomplete understanding of data.
To mitigate this issue, the optimization is divided into two separate cases: 1) d (2,
0:
2) (Z, X, Y) < O. For each case, the optimal kemel is selected independently, and the final kernel is chosen as the one that achieves the highest absolute value of the metrie. Treating these cases individually helps to better capture the characteristios of the data in both high- and low-density regions. Yet, this approach may still result in sub-optimal solutions. This is because directly maximizing or minimizing the estimator can amplity specific data patterns excessively, increasing the risk of overfitting U. particularly when sample size is limited and when deep kernel employed as in [21]
k(x,, y) = [[1 - e)Gr (ow (w)) + c]G2(x,,)) with Gaussian kemelss G1 and Gz. and neural network o...
Augmentation-assister Estimation of AMD. We mitigate the overfitting issue via an iterative optimization process known as data augmentation [5]. However, the distribution of augmented samples normally deviates from that of the original samples, which leads to an inconsistent estimation of the AMD metric. Here, we construct augmented samples Z X and yaug under the condition
that E' d (Zmu X Xx y=me]] 0 to ensure the consistency. Specifically, at each iteration, with U TH m Zzm, which is an i.i.d copy of and as the anchor distribution, we generate Z = {2,
(m= uugg maintains statistical properties of U. Then, for X Kang [x, Ji=1 and Yaug {3 mug mm
Ji=1. we have
with x, ~ X, y.y ~ Y. (5) It is evident that two samples X and Y are drawn from the identical distribution, and are thus equally close to the anchor distribution U satisfying the condition Eld'( (ZmN, Xam y y=r) = O. We take d= (ZmRR, X y=u) as regularization term to avoid overfitting and focus the optimization on d' (7, X, Y), which is the main objective for estimating the AMD. For case dr, X, Y) > 0, the optimization with a regularization parameter S < 00 is
x; aue = 0.5z + 0.5y and y; aug 0.5x' + 0.5y'
t K
E arg max d' (Z, X, X)
d= (7 X y 2
KEK
in a similar manner, for dx Y) < 0, we perform optimization as
arg min d' (2, X) + A d' (n X y yaugya
Km
E
KEK.
With an expected value of zero, the statistic (Zaug, X Youe) actually refecct sampling variability. We square it as regularization term account for reduce the impact of fluctuations where the empirical estimator differs from its expected value, thereby mitigating the overfitting in the estimation.
(
Finally, we select the optimal kernel as follows
Kiyn arg max
(2, (2, NE (MA 3
(6)
Infer the potential relative similarity relationship. Unlike previous approaches that manually specify a relative similarity relationship in alternative hypothesis, our method allows for inferring the potential relationship with kernel Ki,- The inferred relationship is denoted by F, where
F = 1 if kith = k indicating that distribution closer to the anchor distribution U. F --=-1- -1 if pin = indicating that distribution Q is closer to the anchor distribution U.
Consistency of the Estimation. We now establish the consistency of estimating d(U, Q; FFc) via the optimization procedure with augmented samples, under Assumption (Appendix A.2).
Theorem 3. Let K. be a kernel space consisting of symmetric kernels k. satisfying 0 < <(x, y) S B for all ar y E X. Then, as m. - and O S oc, the following holds
F. d. Z, X,Y) d(U,P,Q:Fx) +0 0.
This theorem shows that, although we perform estimation with augmented data, the estimator
F d (Z, X.YY Y) almost surely converges to the true AMD value sample size approaches infinity.
'Our methods are also applicable to Laplace [12], Mahalanobis [23] and Gaussian kernels [31], etc, which be optimizaed with
5

---

3.3 Phase II: Testing the Statistical Significance of Potential Relative Similarity Relationship
Next, we assess the statistical significance the relative similarity relationship identified in Phase I. In the testing procedure, we use testing samples Z' = {x1]/= L X'' = {-
f= ~ P and Y' Y'= {y' Q which are drawn independently of Z, X, and Y. This follows the data-splitting strategy commonly used in kernel-based hypothesis testing to ensure the validity of the test [21].
Given F' from Phase 1, we consider tests for both relative similarity relationships with d' PU Q) (i.e., single kernel form of AMD with the consistently estimated kernel (C))), which is defined as
For F indicating a potential relationship that P is closer U, test on hypotheses Ho dm (U. PP, Q) S
and : (U, IP, Q) 0.
Here, given the testing threshold Ta > for a significance level a, we conclude that IP is
significantly closer to U than IP at level a if test statistic satisfies d=tnm (Z', X', ,Y") > Ta For F = 11 indicating a potential relationship that Q is closer to U, we test on hypotheses PU, (U, Q 2
Ho : dm In similar manner, given the testing threshold Ta < 0 for significance level a, we conclude that Q is significantly closer to U than Q level if test statistic satisfies de. (Z', X', Y To
and H1 : dm (U, < 0.
Then, based on the value of F, we can unify the above hypotheses as one: Homi F im (U, P, S
and
H F. dm (U,P,Q)
Given the testing threshold F Tox > O, the testing procedures can be formalized as where x",, Y": k b X', , I[F d (Z', "',,',Y") > F Fall
(7)
indicates that the inferred relative similarity relationship presented by F is statistically significant at level a; otherwise, the relationship is not statistically significant.
The critical step here is determine the testing threshold F which bounds the type-I error under the null hypothesis, i.e., Pr (b(Z', X', Y"; Kim = 1) I a. Notably, the unified null hypothesis Hq is composite, consisting of the case F- d (U,P, Q) 0 and the case where F- drd (U, PP Q < O. Since the ground-truth d' (U, IP, QQ) is unknown, we set testing threshold F as the (1 a)- quantile of the estimated distribution of F d. (Z', XX,Y") under the proxy null hypothesis H. F. d (U,F, IP, Q) = 0 (i.e., the favorrblv boundary of the composite null hypothesis) by wild bootstrap. Specifically, let B be the number of bootstraps. In the b-th iteration (b E [B), we draw i..dd. variables (E1, ... 5m) from Exponential distribution with scale parameter and define
<= +m with S = m E Ee-
Then, we calculate the b-th wild bootstrap statistic for d'in (Z', X', , Y') as follows
K(2/ TB = E(sG 1) x') + n(z),, x1)) a(2), v/)) k f vf) n(z,, x/) + K(y/ 4¹) 2m
During such process, we obtain B statistics Ti,T2.... ...,..... Ta and introduce testing threshold F with
B Σ
[F To F
Ta = arg min
> 21 21-a
(8)
B
We present theoretical analysis for type-I error follows.
Lemma 4. The type-I error of the unified testing procedure is bounded by a.
Output of our AMD relative similarity test. The output consist of two components: F from Phase 1 and (2'', X',Y', "YY": :(k from Phase 2. The term F specifies the nature of the potential relative similarity relationship: 1) F = 1 indicates IP is closer to U: 2) F -1 indicates Q is closer to U. The term b("', X", Y rit) determines whether the inferred relationship F is significant level a.
We present the details of our AMD relative similarity testing in Algorithm 1. help situate our methodology and highlight key differences, we introduce relevant works in Appendix B.

---

Next, to analyze the advantages of our method in test power, we consider a practical scenario in which a relative similarity relationship is present, indicated by d (U, P. Q) * O. However, the specific nature of this relationship, represented by (g (U, Q)) (-1, 1). remains unknown. Building on this, we denote the probability that F captures the true relative similarity relationship as Algorithm The AMD test
Training Samples Z. X and Y. Iteration Epochs 7' for training, Testing Samples Z', X' and Y', Iteration Epochs B for testing Initialization: Select - km according to the median heuristic [13]. # Phase 1: Select Kernel Km and relationship F ,T
Input:
for t ,,..., do Randomly generate augmented samples Z X aig and yaugg as specified in Eqn. (5)
Kym
Km
+ 17.
V d (Z, Y) X dmtmn (Zmuz X X Y
yaug) y=ug)a
aam V de (Z.X,Y) + A d (Zmug Xuue yumgy
Km
Kim
end for
Select Km as in Eqn. (6) and relative similarity relationship F # Phase 2: testing with kernel K' and relationship F
for b = 1..... B do
Draw _ = E21.2,.... Em) from distribution Exp(1)
Let Ç = {G) m
with Çi = 5/ Σi Ej
To m(m-1) Eitj (SiS) 1 (Z' Y1)/2
end for
a}
Ta arg min Eb=1
B
b('', X', Y";";; km
I[F - (Z', X Y') > Ta
Output: b(Z', X',Y'; (m)) and F.
Asymptotic Test Power Comparison of Testing Procedures
Pr[F d (U, Q) 0)
Comparison with the test uniformly selected alternative hypothesis. Previous methods [13, 14] require manually specifying a relative similarity relationship alternative hypothesis (e.g.. dh (U, P, Q) > 0) and perform test accordingly. Yet. the prespecified relationship may not align with the true relationship. In the practical scenario, if uniformly select alternative hypothesis, the probability that reflects the true relative similarity relationship 0.5.
Now, we present the comparison of test power with a uniformly selected alternative hypothesis
Theorem 5. If tm (U, Q) * O, the test based on the learned alternative hypothesis with F achieves higher test power than that based on a uniformly selected alternative hypothesis as follows
(V Sh
(B 0.5)
+ +O(e
ŒURR
where where ((.) is the cumulative distribution function of the standard normal distribution and JUP..Q is the same to that in Corollary 9.
In Theorems as sample size m increases, the term O " decays exponentially, and the term (vml id (U.P.O)) converges 1. To ensure that the test based on the learned alternative 0u.2.0
hypothesis with F achieves higher test power than that based on a uniformly selected alternative hypothesis, necessary to ensure a large value of B > 0.5. Fortunately, as shown in Figure the probability 3 can readily converge to 1 even with limited data.
7

---

AMD
9.8.
PAAill (leg))
Rejection Rate
0.8-
vollue (log)
Rejectinn
0.8
0.0
0.2-
0.2
0.0
0.0
-22
00
022
0.4
0.5
0.5
1.0
0.0
0.2
04
0.. Ratio
0.5
1.0
0.
02
0.4
0..6
0.0
1.0
0.0
022
0.5 0.6
0.0
1.5
Ratio
(a) CIFAR10
Ratio
(b) MNIST
Ratio V
Figure 1: The comparisons between AMD test and baselines. We set U = vP + (1 Q with P E [0, 1.. When
< 0.5. Q is closer to U, and previous approaches perform well in terms of rejection rates (i.e., test power), this aligns with the prespecified alternative hypothesis Hi ddU,F))
d(U, Q); however, when > 0.5.
their performance deteriorates as closer U. In comparison, our AMD test performs well for both v < 0.5 and > 0.5 by adjusting alternative hypothesis with F. Notably, when V = 0.5 (i.c., no relative similarity relationship exists), all methods control the rejection rate (type-I error) crror) at level a = 0.05 (black dashed line). The p-values align with the findings derived from the rejection rates, demonstrating the effectiveness of AMD test.
CIFAR1O
MNIST
1.0
Teat Power
0.8
0.6
3.0-
0.2-
20
50
100
140
180 220 260 200 50 100
200 250 J0U 1b0 400
Sample Size
Sample Size
Regul. Para. A (30')
Regul. Para. (10')
Figure 2: Probability 3 = Pr( dr (U,
Figure 3: Influence of the regularization parameter À 0]
versus sample size with parameter v = 0.3.
in AMD relative similarity testing.
Experiments
We first conduct experiments to compare AMD test with state-of-the-art tests on benchmark datasets, demonstrating its effectiveness. We also perform ablation studies to evaluate the contribution of each component in our method. Finally, we validate the proposed AMD test in practical applications. Notably, in all experiments, utilize the selected deep kernels as defined in Equ. (4). More details and results of experiments, including the results of type-I error, can be found in Appendix D.
5.1 Comparison with Baseline Tests on Benchmark Datasets
We begin by comparing AMD test with state-of-the-an art relative similarity tests (Appendix D.I):
1) MMD-D [13, 21]: 2) KLFI [14]; 3) MMD-H [13]; 4) UME [15]; 5) SHCE [32]; 6) LBI [33]. Following the setups in [13, 14], we adapt MNIST and CIFARIO as benchmark datasets, both of which comprise original and generative images. We set the sample size to 50 for CIFARIO and 160 for MNIST. We denote by IP the original images and Q the generative images, and set the U as
U = v/P + (1 - V with V {0.0, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
If V < 0.5, then the distribution Q is closer to U; whereas if v > 0.5, the distribution to U. When = 0.5, IP and Q are equally close to U, and accordingly, the test should not report significant relative similarity relationship exists (i.e., no alternative hypothesis should be accepted).
closer
In practice, when a relative similarity relationship exists, it is unknown which distribution, P or Q closer to U. However, existing methods are designed to test a prespecified alternative hypothesis. e.g., Hi d(U, IP) > d(U, Q). As shown in Figure 1, under this setting, they perform well in terms of rejection rate (i.e., test power) when < 0.5, this aligns with the assumed alternative H₁ In contrast, their performance deteriorates when 2 > 0.5. In comparison, our AMD test achieves high rejection rate for both v < 0.5 and v > 0.5 by adjusting the alternative hypothesis with F. Here, the rejection rate is larger when |0.5 - v| is larger, since the relative similarity relationship is more pronounced and easier to detect by the test. Notably, when v = 0.5 (i.e., no relative similarity relationship exists), all methods can control the rejection rate (type-I error) at level a 0.05 (the black dashed line). The p-values in Figure/ align with the findings derived from the test power, providing additional support for the effectiveness of AMD test.
8

---

Table 1: MNIST: Test power vs sample size m for AMD test, with and without data augmentation, and the bold denotes the highest mean power.
VIL
130
160
190
220
250
280
310
340
AMD
.277 = .033
.409 + .036
.604 + ,099
.761
.049
.779 1 .488
.827 + .037
_867 = .028
.969 = .012
AMD-NA
.224
.038
.353 + 041 .577 + 049 .691 0011 .723 .60 .790 033 .829 .08
.931 .033
Table 2:
CIFARIO: Test power sample size m for AMD test, with and without data augmentation, and the bold denotes the highest mean test power.
WI
20
40
60
80
100
120
140
160
AMD
.306 .038
.47
.06
.627 .04
.775
021
.868 .017
.871 .088
.942 : .009
.988 4: .007
AMD-NA
.300 + .46
.455 + 028 ,590 # .51 .744 + 027
.845 + 020
.851 + 020 .919 + 0111 .952 + .000
Ablation Studies. Figure 2 illustrates that 3 = Pr[F dth (U,,
0] approaches I even with limited data sample. This result highlights the capability of AMD test to achieve higher test power by utilizing adjusted alternative hypothesis with F, as supported by Theorems 5. Figure 3 illustrates the impact of the regularization parameter À on optimization with data augmentation. The results indicate that A can be selected within a relatively broad range, specifically [10 - 10o] for CIFARIO and [10 8 10'] for MNIST. Tables 1 and 2 present comparisons of test power versus sample size for AMD test with and without augmented data (referred to as AMD-NA). The results demonstrate that incorporating augmented data facilitates improved kernel selection, enabling the test to achieve higher test power. More ablation studies, including comparisons with test on both alternative hypotheses and previous methods combined with our Phase I, are provided in Appendix D.3.
5.2 Performing Relative Similarity Testing in Practical Applications
We present two case studies to illustrate the practical application of our AMD test.
Relative Model Performance Evaluation. In the first case study (motivated by [34]), for a pre- trained ResNet50 model that performs well on the original ImageNet, we aim to assess its performance across different variants of ImageNet. A natural metric is the margin between model accuracies of the original ImageNet and its variant, with a smaller difference indicating more similar model performance on the two datasets (Appendix D.2). For the variants ImageNet-(SK,, R. V2, A), the accuracy differences, calculated using ground-truth labels, are (0.529, 0.564, 0.751, 0.827).
Yet, obtaining ground truth labels for ImageNet variants is often challenging or costly. Given this, illustrate that relative model performance can be evaluated using our AMD test without labels. The key is to ensure that AMD captures the same relative similarity relationships as indicated by accuracy margins, effectively supporting relative similarity testing. Here, we set the original ImageNet as D, and sequentially set each of the variants (ImageNet- (R, V2, A, SK)) as P. Besides, we sequentially set each of the variants AinageNet([SK, R. V2, A)) as Q By testing which distribution, IP or Q. is closer U, Figure 4 shows that our AMD achieves higher test power than MMD-D. Specifically, MMD-D fails to evaluate the relative similarity between ImageNet-SK (i.c., PP) and ImageNet-A A (i.e., Q as it assumes Q is closer U under the alternative hypothesis.
Adversarial Perturbation Detection. In the sec- ond case study, inspired by [4, 35], we illustrate that the AMD test can be used to assess the level
nejecton Rates
02
of adversarial perturbation applied to CIFAR 10.
We employ ResNet18 8 as the base model and im- plement the PGD attack [36] on CIFAR-10, us-
0.4-
AMD
AMD
02-
MMD-D
MMD-D
ing perturbation levels in { 255 ===1- As expected, larger perturbations result in a dataset that devi- ates more from the original dataset. Given this, we denote the original CIFAR- 10 as U and the CIFAR-10 as Q. Besides, we set IP as the CIFAR-10 10 with perturbation level in (1/255) and perform testing with a sample size of 170. As shown in Figure 5, our AMD outperforms MMD-D and effectively evaluates adversarial perturbation levels. Specifically, MMD-D fails to evaluate the perturbation levels when IP is closer to IU (i.c., P is CIFAR-10 10 with perturbation level in (i/255) 255)3-)) 2=1), as it assumes Q is closer to U
4/255
4/255
0.0
0:01
B,62
0.03
0.01
0,02
0.03
Adversarial Porturbation
Adversarial Parturbation
Figure 5: Comparisons in detecting whether the adver- sarial perturbations on CIFAR- 10 exceeding 4/255.
9

---

MMD-D
I & vs SK
Vm vs imageleet R.
ImageNet- vs ImageNet-
ImageNes SK vs ImageNet
1.0
0.8-
0.6-
0.4
0.2
6.0-
250
500
750
1.00
20 Sample Size
10
100
100 175
250
25
05
Sample Size
Sample Size
Sample Size
Figure 4: Test Power comparison between AMD and MMD-D in identifying which unlabeled variant of ImageNet the ResNet50 model (pre-trained on ImageNet) performs better on.
under the alternative hypothesis. When IP is the CIFAR-I0 with perturbation 4/255, F Q and the rejection rates (type-I error) two methods are controlled at the level a 0.05.
6 Conclusion
This work introduces a new kernel-based relative similarity test by proposing the anchor-based maximum discrepancy (AMD), which defines the relative similarity the maximum discrepancy between the distances of (U,P) and (U, Q) in space of deep kernels. Building on this metric, we learn proper hypothesis and kernel simultaneously, instead of learning kernel after manually specifying the hypothesis. Specifically, in the AMD test, our testing procedure incorporates two phases. In Phase I, we estimate the AMD over the deep kernel space and infer the potential hypothesis. In Phase II, we assess the statistical significance unified testing framework to derive thresholds for tests over different possible hypotheses from Phase I. We provided theoretical guarantees for the proposed method and validated its effectiveness through extensive experiments on benchmark datasets and practical applications. Looking ahead, an interesting direction for future work is to extend relative similarity testing from three distributions to the more general case of multiple distributions.
the potential hypothesis, where we propose a
Limitation Statement
By learning a proper hypothesis and kernel simultancously, the AMD could lead to overfitting, especially with limited samples. To mitigate this, we introduce augmented data in the optimization. As shown in Figure the learned hypothesis reliably captures the true relative similarity even with small sample sizes (20 for CIFARIO 10 and 50 for MNIST). Although using augmented data increases time complexity, the experiments (Figures and 5) show that only a small number of samples are needed for convergence, i.e., the test power achieves 1. In Table 13 (Appendix D), we further compare the runtime of the AMD test with all baselines and provide a detailed analysis.
In the theoretical analysis, we impose several assumptions, including the covering number assumption for Theorem (Assumption in Appendix A.2) and the bounded-kemn assumption for Lemma and Theorem 5. These assumptions are standard and follow previous approaches [37, 13J. Although the optimization analysis in Theorem 3 reveals a gap between theoretical guarantees and practical implementation due to the inherent randomness of optimization algorithms with unknown underlying distributions, the core theoretical properties that ensure the validity of AMD in hypothesis testing do not depend on this theorem and remain robust to the outcomes of the optimization. As shown in Lemma 4, whichever deep kernel is chosen (bounded as in Eqn. (4)), type-I error guaranteed be controlled. Theorem 5 shows that AMD achieves higher test power whenever the learned hypothesis better than random guessing, i.e., when 3 > 0.5, where the asymptotic distribution of the test. samples used in the analysis is independent of the optimization on training samples [21].
Acknowledgments and Disclosure of Funding
This research was supported by The University of Melbourne's 's Research Computing Services and the Petascale Campus Initiative. ZJZ and XYT are supported by the Melbourne Research Scholarship and the ARC with grant number DE24010109 089.. LHP is supported by the ARC with grant number I.P240100101. FL is supported by the ARC with grant number DF240101089, LP240100101, DP230101540 and the NSF&CSIRO Responsible Al program with grant number 2303037.
10

---

References
[1] C. M. Bishop and N. M. Nasrabadi. Pattern Recognition and Machine Learning, volume Springer, 2006.
[2] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. J. Smola. kernel method for the co-ssmmle-ppobleem In Advances in Neural Information Processing Systems 19, pages 513-520, MIT Press, Cambridge, MA, 2006.
[3] D. Sutherland, H.-Y. Tung, H. Strathmann, De, A. Ramdas, A.J. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In Pro- ceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
[4] R. Gao, F. Liu, J. Zhang, B. Han, T. Liu, G.Niu, and M. Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In Proceedings of the 38th International Conference on Machine Learning, pages 3564-3575, Virtual, 2021.
[5] D. Van Dyk and X.-L. Meng. The art of data augmentation. Journal of Computational and Graphical Statistics, 10(1): 2001.
[6] S. Zhang, Y. Song, J. Yang, Y. Li, B. Han, and M. Tan. Detecting machine-generated texts by multi-population aware optimization for maximum mean discrepancy. arXiv, 2024.
[7] A. Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429 443, 1997.
[8] B. K. Sriperumbudur, K. Fukumizu, A. Gretton, Schölkopf, and G. RG Lanckriet. On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6, 2012.
[9] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Muchine Learning Reseurch, 13(1):723-773, 2012.
[10] K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. In Advances in Neural Information Processing Systems 28, pages 1981-1989. Curran Associates, Dutchess, NY, 2015.
[11] A. Schrab, I. Kim, B. Guedj, and A. Gretton. Efficient aggregated kernel tests using incomplete u-statistics. In Advances in Neural Information Processing Systems 35, pages 18793- 18807. Curran Associates, Dutchess, NY, 2022.
[12] F. Biggs, A. Schrab, and A. Gretton. MMD-Fuse: Learning and combining kemels for two- sample testing without data splitting. In Advances in Neural Information Prucessing Systems 36. Curran Associates, Dutchess, NY, 2023.
[13] W. Bounliphone, E. Belilovsky, M. B. Blaschko, Antonoglou, and A. Gretton. A test of relative similarity for model selection in generative models. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
[14] Gerber, Jiang, Polyanskiy, and R. Sun. Kernel-based tests for likelihood-fres hypothesis testing. In Advances in Neural Information Processing Systems 36. Curran Associates, Dutchess, NY, 2023.
[15] W. Jitkrittum, H. Kanagawa, P. Sangkloy, J. Hays, B. Schölkopf. and A. Gretton. Informative features for model comparison. In Advances in Neural Information Processing Systems 31, pages 316-827. Curran Associates, Dutchess, NY, 2018.
[16] J.-N. Lim, M. Yamada, B. Schölkopf, and W. Jitkrittum. Kernel stein tests for multiple model comparison. In Advances in Neural Information Processing Systems 32, pages 2240-2250. Curran Associates, Dutchess, NY, 2019.
[17] H. Kanagawa, W. Jitkrittum, L.. Mackey, K. Fukumizu, and A. Gretton. A kernel stein test for comparing latent variable models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(3):986-0011 2023.

---

[18] J. Baum, H. Kanagawa, and A. Gretton. A kemel stein test of goodness of fit for sequential models. In Proceedings of the 40th International Conference on Machine Learning, pages 1953, Hawaii, HO, 2023.
[19] A. Gretton, B. K. Sriperumbudur, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Balakrish- nan, and K. Fukumizu. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems 25, pages 1214-1222. Curran Associates, Dutchess, NY, 2012.
[20] B. K. Sriperumbudur, K. Fukumizu, A. Gretton, G. R. G. Lanckriet, and B. Schölkopf. Kernel choice and classifiability for RKHS embeddings of probability distributions. In Advances in Neural Information Processing Systems 22, pages 1750-1758, Dutchess, NY, 2009. Curran Associates.
[21] Liu, W.-K Xu, Lu, G.-Q. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In Proceedings of the 37th International Conference Machine Learning, pages 6316-6326 6326, Virtual, 2020.
[22] X. Guo, F. Lin, C. Yi, J. Song, D. Sun, L. Lin, Z. Zhong, Z. Wu, X. Wang, Y. Zhang, and J. Song. Deep transfer learning enables lesion tracing of circulating tumor cells. Nature Communications, 13(1):7687, 2022.
[23] Z.-J. Zhou, J. Ni, J.-H. Yao, and W. Gao. On the exploration of local significant differences for two-sample test. In Advances in Neural Information Processing Systems 36. Curran Associates, Dutchess, NY, 2023.
[24] C. Shorten and T. M. Khoshgoftaar. A survey image data augmentation for deep learning. Journal of big data, 6(1): 2019.
[25] X. Ying. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1168, page 022022, 2019.
[26] S. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. A. Mann. Data augmentation can improve robustness. In Advances in Neural Information Processing Systems 34, pages 29935-29948. Curran Associates, Dutchess, NY, 2021.
[27] R. Serfling. Approximation Theorems of Mathematical Statistics. John Wiley & Sons, 2009.
[28] B. K. Sriperumbudur, K. Fukumizu, and G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12:2389-2410, 2011.
[29] S. Boyd, P. Boyd, and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[30] D. Chicco, M. J. Warrens, and G. Jurman. The coefficient of determination r-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Computer Science, 7:e623, 2021.
[31] W. Jitkrittum, Z. Szabó, K. P. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In Advances Neural Information Processing Systems 29, pages 181-189. Curran Associates, Dutchess, NY, 2016.
[32] D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In Proceedings of the 5th International Conference on Learning Representutions, Toulon, France, 2017.
[33] X. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions Information Theory, 68(10): 6631 16662, 2022.
[34] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems 29, pages 136- (66144 144. Curran Associates, Dutchess, NY, 2016.
12

---

[35] S. Zhang, F. Liu, J. Yang, Y. Yang, Li, B. Han, and M. Tan. Detecting adversarial data by probing multiple perturbations using expected perturbation score. In Proceedings of the 40th International Conference on Machine Learning, pages 41429-4606 41451, Hawaii, HO, 2023.
[36] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.
[37] M. A. Arcones and Evarist E. Giné. Limit theorems for u-processes. Annals of Probability, pages 1494 1542, 1993.
[38] H. B. Mann and A. Wald. On stochastic limit and order relationships. Annals of Mathematicl Statistics, 14(3):217-226, 1943.
[39] É. Borel. Traité du calcul des probabilités et de ses applications, volume 3. Gautier- Villars et cie, 1927.
[40] K. L. Chung and P. Erdös. On the application of the borel- lemma. Transactions of the American Mathematical Society, 72(1):179-186, 1952.
[41] P. Janssen. Weighted bootstrapping of u-statistics. Journal of statistical planning and inference, 38(1):31-41, 1994.
[42] Golland and B. Fischl. Permutation tests for classification: towards statistical significance in image-based studies. In Proceedings of the 18th International Conference on Information Processing Medical Imaging, pages 330-341, Cumbria, England, 2003.
[43] H. Cai, B. Goggin, and Q. Jiang. Two-sample test based on classification probability. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(1):5-13 13, 2020.
[44] I. Kim, A. Ramdas, A. Singh, and L.. Wasserman. Classification accuracy as a proxy for two-sample testing. Annals of Statistics, 49(1):411-434 2021.
[45] S. Jang. S. Park, I. Lee, and O. Bastani. Sequential covariate shift detection using classifier two-sample tests. In Proceedings of the 39th International Conference Machine Learning, pages
Baltimore, MD, 2022.
[46] J.-M. Kübler, V. Stimper, S. Buchholz, K. Muandet, and B. Schölkopf. AutoML Two-Sample Test. In Advances in Neural Information Processing Systems 35, pages 15929-15941. Curran Associates, Dutchess, NY, 2022.
[47] S. Hediger, Loris L. Michel, and J. Näf. On the use of random forest for two-sample testing. Computational Statistics & Data Analysis, 170: 177455----1.,,8 2022.
[48] W. Zaremba, A. Gretton, and M. B. Blaschko. B-test: A non-parametric, low variance kernel two-sample test. Advances in Neural Information Processing Systems 26, pages 755-763. Curran Associates, Dutchess, NY, 2013.
[49] G. Wynne and A. B. Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research, 23(73):1-51, 2022.
[50] S. Shekhar, Kim, and A. Ramdas. A permutation- free kernel two-sample test. In Advances in Neural Information Processing Systems 35, pages 18168-181 1888-8180. 180. Curran Associates, Dutchess, NY, 2022.
[51] M. Scetbon and G. Varoquaux. Comparing distributions: & geometry improves kernel two- sample testing. In Advances in Neural Information Processing Systems 32, pages 12306-12316. Curran Associates, Dutchess, NY, 2019.
[52] B. Han, J. Yao, T. Liu, Li, S. Koyejo, and F. Liu. Trustworthy machine learning: From data to models. Foundations and Trends@ in Privacy and Security, 7(2-3):74-246, 2025.
[53] X. Tian, I.. Peng, Z. Zhou, M. Gong, A. Gretton, and F. Liu. A unified data representation learning for non-parametric two-sample testing. In Proceedings of the 41th Conference on Uncertainty in Artificial Intelligence, pages 4165-184, Rio de Janciro, Brazil, 2025.
13

---

[54] F. Liu, W. Xu, J. Lu, and D. J. Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. In Advances Neural Information Processing Systems 34, pages Curran Associates, Dutchess, NY, 2021.
[55] E. Mammen, Bootstrap, wild bootstrap, and asymptotic normality. Probability Theory and Related Fields, 93(4):439-44) 455, 1992.
[56] B. Huang, Y. Liu, and L.. Peng. Weighted bootstrap for two-sample U-statistics. Journal of Statistical Planning and Inference, 226:86 99, 2023.
[57] A. Schrab, I. Kim, M.e Albert, B. Laurent, Guedj, and A. Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research, 24: 19::111994:8,, 2023,
[58] X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han. Detecting out-of tit-of-distrbbttion data through in-distribution class prior. In Proceedings of the 40th International Conference on Machine Learning, pages 15067-15088, Hawaii, HO, 2023.
[59] Z. Zhong, J. Hou, Z. Yao, L. Dong, F. Liu, J. Yue, T. Wu, J. Zheng, G. Ouyang, C. Yang, and Song. Domain generalization enables general cancer cell annotation single-cell and spatial transcriptomics. Nature Communications, 15(1):1929, 2024.
[60] H. Chi, F. Liu, W. Yang, L. Lan, T. Liu, B. Han, W. K. Cheung, and T. Kwok. TOHAN: A one-step approach towards few-shot hypothesis adaptation. In Advances in Neural Information Processing Systems 34, pages 20970-20982, Dutchess, NY, 2021. Curran Associates.
[61] H. Chi, H. Li, Yang, Liu, L. Lan, X. Ren, Liu, and B. Han. Unveiling causal reasoning in large language models: Reality mirage? In Advances in Neural Information Processing Systems 38, pages (6640-96670 Vancouver, Canada, 2024.
14

---

Appendix
A Detailed Proofs for Our Theoretical Results
A.1 Detailed Proofs of Theorem 2
Proof. By the definition, it is evident that (; 2 0. Furthermore, since A E IC is characteristic, the mappings
Pn Ez-z (n(:,))) P p E and + μr Ez-elK(:.))
are injective. Then, for any kernel kM chosen from the maximal class of kernels associated with d( Fic), be shown that
+
μ1
MU
uQ
2
um +
up
PaU
pit
2
Haa
d(U,Q,P;FK)
Next, we prove that
ddU, Q:Fk).
In similar manner, denote by R²* the optimal kernel chosen from the maximal class of kernels associated with d(U, W; Fi)) and let x" denote the optimal kernel chosen from the maximal class of kernels associated with d(U, Q W; Fk)). The above inequality can be formalized
Hr + 0m
pp
Ha-
(
He + paw
110
MU
mm
MQQ
2
By using kernel k², it is easy see that
Pr + Mai
117
muu
11
2
2
and
04Q
pap
160
2
2
Ha.
Hence, is sufficient to prove that
1b0 2
+
D1Q
+
<
Mu
+
110
2
2
H.
H.
15

---

To facilitate further analysis, we reformulate the AMD as follows:
K my
Ms +
Mr
MQ)
2
Hie
|
1 li....
(un MPP) Pe..
(RALT, Mat
+
2
2
| A...
1 11pt..
|| |1 + (me pe)
||p0l11
(paU, maa)
+
2
2
Mou pall mallia.
MJ
M ler.
2
2
In similar manner, we have
up + paw
powl M
|
PR....
mw
2
2
2
Ha-
and
<
+
wa |1
MAU - k 1
Mn
He
Here, without loss of generality, we assume Then, we have
||
|| - 11r1 S $11000 - mel 24.. S
pep +
11wu
1im.. 14
|| - pep 11.11
AAU
MpP
us
2
2
Hn.
and
Up
+ Maw
Mig
waw u1Q Mm +
M4U
Maw
+
mm
2
||
Mel
11
||m08
14p
1i2e.
+
2
2
2
2
|
Mel
11
|
mmwl
I I IV
2
2
||Ac
Mell
|
prrl Pdo
2
2
d(U. P. Fk).
For other cases where the order of the terms differs, similar analysis applies, leading to analogous conclusions. This completes the proof.
A.2 Detailed Proofs of Theorem 3
We start by introducing the concept of the U-statistic, fundamental tool in statisties.
Definition 6. [37, 27] Let h((w1, W2...., wr; R) be symmetrie function of r arguments. Suppose we have a random sample wi, W2...., wn from distribution W. The r-th order U-statistic is defined follows
Un (h)
(
Σ
h(w; Wip ... wi
Here, (Tr) the number of ways to choose r distinct indices from m, i.e., the binomial coefficient, and the summation is taken over all possible r-tuples from the sample.
16

---

Let H be function space of h(-) for U-statistic and denote by pseudometric em : x Ht + R20 as follows
(m1, h2) = Um (1h hal) for We define the e-covering number of (Ht em) as follows
h1, H..
(10)
[e, H. fem,) min n Bh1, b2, hr, hn H s.t. sup min em (h, h4) - {E
hEX isn
For U'statistic Un (h), we define its mean value over W with fixed function EUUm (h)). and we introduce the following theorem.
Theorem 7. [37. Corollary 3.2) If H is measurable class, then the conditions ElUm < co and log (e, em)/m + 0 in probability imply 1lum (h) (m)llllll 3 0, ie..
lim 11 NEW
sup
1n EIU (h)] 0,
where " indicates convergence almost surely.
Recall the definition of d" X) from Eqn. (2) with corresponding samples Z {= Um X = { ~ Ipm and Y = {y: m ~ Qm, can be equivalently expressed as a second-order U-statistic with w, = (zi: 3i, y/) and w = (Zj. By Mj) defined in Definition 6, and is given by
(Z, 2m(m 1)
h h((z;, wi, y:), (z,, U;); (;);)
(
where we define the function
h ((2z,, 3z. Mi), (z), Fy By.
= x(z;,,,) + x(=,, x;) k(Zz, Uz) x(z,, - k(x;, 2zj) + +(ys UJ),
(11)
which is symmetric the kernel is symmetric.
Given the kernel space K consisting of bounded kernels, we denote by H the class of functions M(;;)) induced by bounded kernel K, as defined in Eqn. (11). Meanwhile, the corresponding pseudometric defined in Eqn. (10) can be written as
Cm k1); h(;;))
Σ
[k((z,, yi), (3 T.s m)) - h((2;, De yi), (2), Bz 4));
2m((m 1)
for h(-3k1,, ((:K))) .t.
Building this, we propose the following assumption.
Assumption 1. H measurable and satisfies the condition log N(e, H, em fmm/m 2x 0.
Under the assumption, the Theorem 7 can be applied to d' (Z, X, Y) such that
lim m m= M(-N))E
sup
-2, - d", = 0,
(12)
where E T 22 (2
We now present the proofs of Theorem 3 under the Assumption 1 and the condition in Eqn.(12).
Proof. Recall that the kernel is selected as follows:
K arg 2de (z,
WE (reT wi
Here, with regularization parameter 0 A co, the kernel K" is determined by:
+++ E arg max d², X, X Zx (Zang xaug yau K
REK
17

---

Similarly, the kernel Km is obtained as:
E arg min (Z, A- d' (ZuR, X y yougg? KEK.
Furthermore, the relative similarity relationship F associated with the selected kernel is defined as follows
{
for Kim
15
F
for Km
Kim
The above terms Km Km Km and are all empirical estimations derived from the samples Z, X, and Y. To facilitate the proof, we need to define their counterparts under the true distributions U, P.
and Q. We start by recalling the definition of AMD Eqn.(1) follows:
((U,PQQ;F)
sup KEK.
PC
uQ
p&p
2
Hi.
sup ldd (U, Q)
KEK
sup NE(N=-1 ,x*
ld (U.P..
In similar manner, we define
arg max KE (x-++Nn--7
(d,
where
arg max d= (U,P,Q) KEK
and
R* E arg min d" dU,, (U,P,Q).
KEK
Hence, we can write that
d(U,P. Q: FR) d' (U,P.Q).
where
for n² K* for n²' x*
F
Building this, to establish the convergence
F. a. (7, d(U,P. FR) 0,
is equivalent to show that
(Z, - d" (U,P,Q) 20 0,
which can be simplified as follows
max {1 a (Z,X,Y), -m (2,X,
max {1 (U.P,Q). -U. 25, 0.
(U.P.Q)
Here, using the Continuous Mapping Theorem [38] and the continuity of max function, it suffices to prove that
" (7. (YX d (U.P.Q) 2. 0 and T (Z, X, d= (U,P.Q) as 0.
Building on this, we first prove that, 0o, the following convergence holds:
de (Z,XYY) d== (U, n.s.
Based on the Borel-Cantelli Lemma [39, 40], is sufficient to prove that
Σ Pr
+ (Z,X,Y) d' (U, Q) 2 8) <0,
(13)
m20
18

---

for any 8 > 0. It is evident that
d
(Z,X,Y) d= (U.P.Q)
S (ran (2, (U -FF
A + B.
and we have that
Pr (2 (2 dt (U, (U. 26) Pr +r 35/4).
(14)
Based on Eqn. (12), we have that, there exists a large enough sample size ml such that
A c (Z.X,Y) dr (
8
(15)
which indicates that for large enough sample size ml.
Pr (A 2 =0
(16)
Notably, by the definition of k** is evident that d= + (U, Q 2 dr IP, Q)
and have
B dm (U, d (U,P,Q) - (U,P,Q (U d' (U,
Based on Eqn. (12), we have that, there exists large enough sample size m2 such that
(U,P.Q) < deth! (Z,X.Y)+ 4
(17)
Based on the definition of n we have
X. x Jm (Zme, yag yauzza gxtit (Z. x- d (zme,
(2ue X yaugya
Hence, substituting this relationship into Eqn. (17), we can refine the bound as follows
d= (U,P,Q) Q)
Laae,Xaau y yugg + 6
X Y 2 + + - < Th (2,X,Y) X g (amsy Xaug
d (Z,X,Y) - X J (mmz
4
yung) 2 + 8
(18)
4
for a large enough sample size m22
Meanwhile, for large enough sample size m3 and a constant À >>> 0, the regularization term satisfies the following bound
6 4
A d (n
yrun)z smm
(19)
derived from Eqn. (12) and the fact that Elde (m X y y= = 0 for K. based on Eqn. (5).
Then, combining the results of Eqns. (15), (18) and (19), the following holds
8
d' (U,IP,Q < d (U, Q) 4 4
dل d Q) + 35/4
for large sample size m4 max max(ml, m2, m3). This indicates that
Pr (B 36/ / 4) 0,
(20)
for a large enough sample size m4.
19

---

Consequently, based on Eqns. (14), (16) and (20), we have that
Pr ahe (2, X,X - d=' (U, () 26) 26)=0, =0,
for large enough sample size mA, which implies that
Σ
Pr (Z,X,Y) dt (U.P. 20) m4 < 20.
Hence, we complete the proof of
dn -t (U, a.s. 0.
In a similar manner, we now prove that, as m - 00, the following convergence holds:
dmm (2, d (U,P.Q) -i, 0.
Here, it is also sufficient to demonstrate that
Σ m20
Pr
2 (Z,X,Y) d= -T "U,, 2 20) <0,
(21)
for all 8 > O. is evident that
dm (Z,X,Y) - d= (U.P.Q)
S jotn (2,X, - (U.P.Q + & (U,P,Q " (U, P,Q)
"
D.
and we have that
Pr ( a (2, d= "0 (U.P.Q) 26) Pr P/ Pr 35/4)
(22)
Based on Eqn. (12), we have that, there exists a large enough sample size m5 such that
C J 7, X,, th (U,P.Q) <
8
(23)
which indicates that for large enough sample size m5.
Pr (C 6/4)=
(24)
Notably, by the definition of K** is evident that d (U,P,Q) d T Q) and we have
B d (U,P,Q) d==" (U,P,Q) th (U,P,Q) d=' d (U,P.Q).
Based on the definition of Ktt we have
Th (2, XYY) - A Jh (7" X yranzz < g (2, - - " (Z x yaug?a
Hence, substituting this relationship into Eqn. (23), we refine the bound as follows
Si (U,P.Q) Q
8
F (Z,X,Y) - x- mz m aa
X Ti (Z a L m
J (2,X,Y) X dni (auug Xaug yrug) 8
(25)
for large enough sample size m5.
20

---

Based on Eqn. (12), we have that, there exists a large enough sample size m6 such that
d" (U,P.Q) d (Z, 8
(26)
Meanwhile, for large enough sample size m77 and a constant X > 0, the regularization term satisfies the following bound
8
À di (zmER X mun y ymmzz
(27)
4
derived from Eqn. (12) and the fact that Eld (ZmHe X y=m)) ] = for A E K. based on Equ. (5).
Then, combining the results of Eqns. (23), (25) and (27), the following holds
dm (U, 8 8
(U,P,Q)
dmi (U,P, +
for a large sample size m8 = max(m5, m6, m7]. This indicates that
Pr (D 2 36/4) = 0,
(28)
for large enough sample size m8.
Consequently, based on Eqns. (22), (24) and (28), we have that
(1 X,) d== (U, 2 (0) 25)=0; =0; Pr
for a large enough sample size m8, which implies that
Σ Pr
d
(2,X,Y) d (U,P.Q) 26) < m8 < 00.
m20
Hence, we complete the proof of
dm (2,X,Y) d
d (U.P.Q) 0.s 0.
This completes the proof of Theorem 3.
A.3 Detailed Proofs of Lemma 4
Recall that we test on the unified hypotheses as follows
H F d' (U. 0 and
H F. (U. > 0.
We determine the testing threshold F Ta the (1- a) quaatile of the estimated distribution of 0 (i.e., the least-
F Z (Z', X', Y') under the proxy null hypothesis Hう F du, favorable boundary of the composite null hypothesis) by wild bootstrap. Specifically, let B be the number of bootstraps. the b-th iteration (b E [B), we draw i.i.d. variables & (Ei,.., Sm) from Exponential distribution with scale parameter and define
<= (c with S. ==m- m E
Σe.
E
Then, we calculate the b-th wild bootstrap statistic for dnhn (Z', X', Y') as follows
TB E(Gs, 1) x(z(,=/) + n(Z/,, mx)) k(22,, y¹) - K(2/, yi) n(al, x,) + k(y v/) 2m((m - 1)
$41
During such process, we obtain B statistics Ti, T2, ..., T' and introduce testing threshold Ta with
B [[ To F. 2 21-
Ta arg min
21

---

Given the testing threshold F Ta > 0, the testing procedures can be formalized as
b X" m I d: (Z', x'," YY) > F F.a)
where (Z', X',Y;, km, indicates that the inferred relative similarity relationship presented by F is statistically significant at level a; otherwise, the relationship is not statistically significant.
Building on this, we introduce a useful Theorem as follows.
Theorem 8. [41, Theorem 11 If h is non non-degenerate and h2 h2w w'; x) < oo with w = (z, x. y).
w/ (z', x', y²) and w, w' ~ U P x Q tthn as m oC, the following holds
sup Pr (VmF To S t) Pr (vm (F F (Z' x'' - F. d (U.P. 2)) st) +0. EE
We now present the proof's of Lemma 4 as follows.
Proof. By applying the results of Theorem and substituting t/ym with F Tas we obtain the
following asymptotic behavior as 0o:
Pr (F T& Ta) Pr F. d (Z'X',Y') - F dm (U,P.Q) S Ta) FF
+0. (29)
By the definition of Ta, the following holds,
(r F.) 21
(30)
Then, by applying Eqns. (29) and (30), the following holds asymptotically
Pr F -
(Z', X Y") F mm (U, F. .a)) 2 a.
which can be expressed as
Pr F J (Z', X' Y') F Ta + F. d (U.P, Q
2 a
Under the composite null hypothesis H F. (U, Q < 0, the ground truth F-d (U,P, is unknown, but we have that
Pr F Jm (Z' X' (")
d (U,P,, 2 Pr F J (Z' X' S
F
Hence, by considering the proxy null hypothesis H.: F tm (U, Q) 0, follows that
Pr ((7', X Y
F at ( X Y") > F Ta] =1)
Pr F d (Z', X', Y') > F F.o
< Pr
F i 1'',''' YY) < F Ta
< 1 (1 -a)
S a
This completes the proof.
A.4 Detailed Proofs of Theorem 5
In the proof of this theorem, analyze practical scenario where relative similarity relationship is known to exist, i.e., Idd (U, Q) > 0. However, the exact sign of this relationship is not known. In such cases, if we arbitrarily assume a particular direction of the relationship, for example, d PU, Q the likelihood of this assumption being correct is 0.5.
Furthermore, in our AMD test, we denote the probability that F from Phase captures the true relative similarity relationship follows
8 Pr[F tm (U, Q) 0]
22

---

Recall that, to test relative similarity, we introduce the test statistic d' (Z','', Y") (Eqn. (2)), which is an unbiased estimator of dd (U, IP, Q) and can be expressed U statistii as follows:
dti (Z' (Z',X',Y)) 2m(( - 1)
1
Σ i
h((2//, v/), (0))))))))) (2/g a U3); Hmm)
where we define the function
h((21, 2//, v/)s (z 3 vf)) Kmn)
km (2), mn5) Kin 2() Km (z), v/)) k (2 v1) Him 2.) (m(1/1 v,)
which is symmetric the kernel nm is symmetric.
The estimator has the following asymptotic behavior:
Corollary 9. [27, Section 5] If h is non degeneeeee and h² (w, w' x) < oo with w = (z, x. y).
w/ (z', xt, y') and w, w' U F Q the following holds
m d m (Z', P YY) d' (U,P.Q) 4 N(
d.. 4 EE Ew [1? w'; km))]]
E. ie [k(w, w'; >>,,
where 4 denotes convergence in distribution.
We now present the proofs of Theorem 5 as follows.
Proof. In our AMD testing procedure, outlined in Eqn. (8), F. is defined the (1 -1-aa)-qannile of the null distribution of F (Z', X', Y"). This null distribution is estimated using the wild bootstrap under the proxy null hypothesis Hº- F d (U, IP, Q
As discussed above, in our AMD test, 3 denotes the probability that J' from Phase 1 captures the true relative similarity relationship. Conditional on this event and based Corollary 9, the test power of our method is given by:
Pr F In L X',' F. Ta n F (U, 20
(U.P.Q)
- -
3Φ V
OU.P.Q Conversely, under the condition that F iincorrectly captures the relative similarity relationship. the test power given by:
Pr (Z, > F n F F.d (U.P.Q)
dd
(U,
Q1 F. TT
(1 ② Vm
OUF.&G
In summary, we have the test power for our AMD test as follows
Pr F Lot (Z', X', F -Ta)
p
in
U.
Q - F F.
(vm ld UU.P, Q) F
BΦ
V
+ (1 8)
ŒUBEOQ
ØUPP.Q
For methods that manually specify an interested relative similarity relationship in alternative hypothe- sis (c.g., d (U, F. Q) 0), it tests on
H : d (U,P,Q) and H d (U, P. Q) >
and the likelihood of this alternative hypothesis being correct is 0.5.
To test the composite null hypothesis H d' (U, < 0, we can set the testing threshold
Th > 0 as the (1 -q-uuantile of the asymptotic distribution of d' (Z', Xx, Y') under the condition
23

---

d' (U,),) 0 (i.e., the ravorrlae boundary of the composite null hypothesis). Actually, this testing threshold can be replace by F Ta in practice. From Theorem the testing threshold Y") under the proxy null hypothesis H6 : F (U, Q = O. which symmetric at according to Corollary 9 and matches the asymptotic distribution of di (Z', X', x',Y" Y") under the condition d (U, Q 0. Hence, F Ta is also consistent estimator of the (1 a---uantile of the distribution of dm (Z', ,,,Y))
F Ta converges to the true (1 qu-antile of the asymptotic distribution of d ,Z','', Xx,,
Y') under the condition d (U, Q O.
Given the testing threshold F Ta and conditional on the event that specified alternative hypothesis is correct, test power given by:
H1 i (U, P, Q)
d (U,P. Q) -
0.50
Vm
OUPP.Q
from Corollary 9.
Conversely, under the condition that the specified alternative hypothesis Hi : (U, 0s false, the test power is given by:
Vm d UU
Q1)
F-
0.50
OUP..Q
Hence, the test power randomly specifying an alternative hypothesis expressed follows:
p' = Pr in
F.To
=
0.54
d
(U,PP. Q) - F F. + 0.50
(Vm=
(Vm
UU,P. Q) F-
To
ŒUP..Q
OU.P&
Finally, as we can see, the difference in test power between adjusting alternative hypothesis with F versus using a randomly selected alternative hypothesis given by
dd Vmm
(U, Q)I F Ta
( (Vm-
-ld ld (U, Q)1 - F- To
P p¹ =
(3 0.5)
+ +0.5 -3
OU,P,Q d (U, Q)I
TU,F,Q
F Ta
(3 0.5)
ØUPP.Q
where Ta > This completes the proof.
A.5
Detailed Proofs of Theorem 10
In our AMD test, we denote the probability that F' from Phase captures the relative similarity relationship as follows
P
0].
Recall that, to test relative similarity, we introduce the test statistic d=t.. (Z', X', Y") (Eqn. (2)), which an unbiased estimator of d! IP, Q) and can be expressed U-statistic follows:
Tm 7 Y
1
Σ 144
h((2), v(), (z/,, a/j, 2/ min)
2m(m 1)
where we define the function
h((2/, o/ w(), (z's x'y, 1/) k
= ki, (2/, a + Hi, (2)j, xx)) - Ki, (2/, v/)) Ki (2),8,,81 vn) kin (2', 2tt)) + nim (in( v3)
which is symmetric the kernel ni,, is symmetric. The estimator has the following asymptotic behavior:

---

Corollary 9. 127, Section 5.5.1] If h is non-degenerate and h² (w, w²;k) < oo with w (z, 2, y).
w' (z/, x', yy) and w, w' U Q, the following holds
Vm d (7', X .Y) -U, 4 NO oi..))) i.o 4 (Ew En [h" (w, w/; kmm)
E (h(w,, w'; km))),
where 4 denotes convergence in distribution.
We now present the proofs of Theorem 10 follows.
Proof. In our AMD testing procedure, as outlined in Eqn. (8), F.Ta is defined as the - (1-x)- quantill of the null distribution of F d (7', X", Y*). This null distribution is estimated using the wild bootstrap under the proxy null hypothesis Hi F d PP, ? 0.
As discussed above, our AMD test, 3 denotes the probability that F' from Phase I captures the true relative similarity relationship. Conditional on this event and based Corollary 9, the test power of our method is given by:
Pr (F In 7 X (', F. Ta O F de PU, (.PP,QQ)))
3Φ (U.
F
OU.P.Q
Conversely, under the condition that F d'incorrcclly captures the relative similarity relationship, the test power given by:
Pr F gr (Z','')
F
F
d
(U,
UUP.QQ 4
<0)
UU,P, Q1 -
F.
(1 B) Vm
ŒU.P&Q
In summary, we have the test power for our AMD test as follows
p Pr F de (Z', X', Y") F To
= 3& \ dh (U,P, Q1 F
UU, IP, Q)
F
+ +11-3)6
Vm
OUFF.Q
dU,P.Q
For methods that test both possible alternative hypotheses in multiple testing manner with reduced significance level of a/2 for each, we first test on the following hypotheses
H am (U,P, Q O and
H d 2)
The testing threshold T/22 0 is determined as the a 11--a22)quantile of the null distribution of dm (Z', X',Y") Y') under the condition that d' (U, P,,,,,,,,, Q) 0 (i.e., the least-favorable boundary of the composite null hypothesis). Given the testing threshold, we reject the null hypothesis Zr... (Z', X", Y') a/2-
Second, we also test on the following hypotheses with significance level a/2,
H Tm (U, Q
and H" d (U, (U) <.
The testing threshold T 0 is determined as the a/2-quantile of the null distribution of dein (Z', X', Y') under the condition that d (U, Q) 0 (i.e., the least-favorable boundary of the composite null hypothesis). Given the testing threshold, we reject the null hypothesis Z (Z', X', Y') < T²/2-
Notably, in both cases, the testing thresholds actually has the same absolute value, i.e., T -//2=//2=- --""/
This arises from the fact that the null distribution of d'in (Z', X', YYY under the condition that
25

---

dU, Q = is symmetric at zero according to Corollary 9. Moreover, the null distribution also
matches the asymptotic distribution of F- F-ddim (Z', X", Y") under the condition F. (U, P. !
with F E { (-1,+ +1). Consequently, the absolute value of testing thresholds, i.e., 7a/2 -T o// used
in both tests can be estimated via the same wild bootstrap procedure as described in Section 3.3, with F E (-1,+1] +1} and a reduced significance level of a/2, yielding the threshold F- -aaa2 Ta/2 -T//2- The equivalence holds asymptotically, following from the convergence properties of F Ta/2 established in Theorem 8.
For the two alternative hypotheses Hi dt (U, P. Q) 0 and Hi d (U, Q) < 0, only one of them is true, and the corresponding test exhibits the following test power from Corollary
d U.P.P. Q1
F. Ta/2
S
OU.P,C
Conversely, for the test with the incorrect alternative hypothesis, its test power is given by:
j (U,
Q) - F Ta/2
Vm
+
dU.P.Q
Hence, the test power of testing both possible alternative hypotheses with reduce significance level is expressed as follows
(U, (.P.Q)I-F--. Ta/2 + + (vm d (U,P, Q) - F.FFFTT//
p" +b
Vm
Ta/2
OUF
dU
Finally, as we can see, the difference in test power between adjusting alternative hypothesis with versus testing on both possible alternative hypotheses is given by
p p"
do
(U, Q)I
F-
(U,P. Q) - F. F.
3&
Vm
vm
OU.P.Q
(U,P,Q)
F. Ta
chen (U,P,Q) F .T/22
+(1 - B)
Vm
+
OUPP&Q
ŒUPP
d m
U.
Q)I
F
Ta
(U...Q) P). F. Taf/2
= 38 Vm
m
+ O(e-)).
OUF
dUpP
where F TT < F. Ta/2-
Faaa2 This completes the proof.
B Relevant Work
Prior to conducting hypothesis testing, it is typically necessary to formulate the null and alternative hypotheses. Previous methods for relative similarity testing follow this procedure by manually specify- ing a relationship in the alternative hypothesis and performing the test accordingly. One approach uses test statistic based on the difference of MMD distances: MMD(U, P; x) MMD(U, Q; k). If this difference exceeds positive threshold, the test rejects null hypothesis, essentially testing whether Q is closer to UJ than PP [13]. Another approach tests on a reference distribution U (1 v) IP + vQ, where / E [0, 1]. by evaluating whether v > for some fixed threshold E (0, 1) [14]. This setup quantifies the relative closeness of P to the mixture distribution U at level determined by 6, thereby constituting an oriented test. Several subsequent methods (used as baselines in our experiments, as shown in Figure 1, with details provided in Appendix D. including MMD-D [21]. UME [15], SHCE [32], and LBI [33], extend the framework of [13] by replacing MMD with various distance metrics from the two-sample testing literature, and likewise follow oriented testing paradigm. In comparison, for our AMD test, we first infer the potential relative similarity relationship and then test the inferred relationship by proposing corresponding alternative hypothesis.
By setting U (or = Q) our test can be adapted for two-sample testing, which aims to assess the difference between two distributions IP and Q. Some relevant approaches measures differences using
26

---

classification performance [42, 32, 43, 33, 46, 47], and the fermel-based approaches measure the difference between kernel embeddings of distributions [48-52].
Various techniques have heen developed to select kernels for kernel-based hypothesis testing. Some of these rely on heuristic strategies, such as the median heuristic [9, 13], self-supervised representation learning [53], meta learning [54], or adaptively combine multiple kernels [11, 12]. Additionally, supervised methods have been explored, where kernels are selected based on result on held-out data [31, 3, 23J. Previous methods all selected kernels that best support the pre-specified alternative hypothesis, which poses challenges in kernel selection for relative similarity testing. discussed in Section 1. In comparison, our AMD test learns proper hypothesis and a kernel simultaneously, instead of learning a kernel after manually specifying the alternative hypothesis.
Wild bootstraps are widely used in hypothesis testing to approximate the null distribution [55-57]. This technique involves repeatedly re-computing the statistie with randomly assigned variables for indexes E (1.2...., m. Alternatively, one can use the (1 - a--quantele of asymptotic null distribution (e.g. Corollary as the testing threshold [13, 23J. However, is challenging to obtain an accurate asymptotic distribution with limited sample sizes.
Besides relative similarity testing, other statistical hypothesis testing methods are also widely applied in various domains of machine learning. For instance, two-sample testing could be used for out-of- distribution detection [58], adversarial image detection [35], and distribution alignment in transfer learning [22]. Independence testing could be used for domain generalization [59, 60], causal discovery [61] and trustworthy machine learning [52]-
C Comparison with the Test on Both Possible Alternative Hypotheses
Another way to perform the relative similarity testing is to test on both possible alternative hypotheses in a multiple testing manner with reduced significance level for each test. Specifically, given the significance level a, we can test on the following hypotheses with significance level a/2.
H m (U, P, Q <0 0
and
H : d (U, P. Q > 0.
The testing threshold aa/2 > 0 is determined as the (1 - quaantilee of the null distribution
of dm (Z',', X", Y') under the condition that dm (U, Q) = O (i.e., the least-favorable boundary of the composite null hypothesis). Given the testing threshold, reject the null hypothesis if d (Z', X', Y') > Ta/2-
Second, we also test on the following hypotheses with significance level a/2,
H d (U,P.
and H d (U, < 0.
The testing threshold T/22 0 is determined as the a a/2-quantile of the null distribution of dtin (Z', ('',',, Y') under the condition that (U, Q) 0 (i.e., the least-favorable boundary of the composite null hypothesis). Given the testing threshold, we reject the null hypothesis desm (Z',X' Y') < T//2"
Notably, in both cases, the testing thresholds actually has the same absolute value, i.e., T T,. --/22
This arises from the fact that the null distribution of Z=i.. (Z', X'.Y') under the condition that (U,, Q 0 is symmetric at zero according to Corollary 9. Moreover, the null distribution also matches the asymptotic distribution of F- F (Z', XX', Y') under the condition F.d (U, ( with F E {11. Consequently, the absolute value of testing thresholds, i.c., Ta/2 -Ta22 used in both tests can be estimated via the same wild bootstrap procedure as described in Section 3.3, with F E (-1,,+1) +1 and a reduced significance level of a/2, yielding the threshold F- -Taa2 = Ta/2
- The equivalence holds asymptotically, following from the convergence properties of F Ta/2 established in Theorem 8.
Now, we present the comparison of test power with the multiple testing procedure as follows:
Theorem 10. If d'.. (U,P, P. Q) # O, the test based on the learned alternative hypothesis with F achieves higher test power than that tests both possible alternative hypotheses is given by as
27

---

follows
3 (mm Id (U,P, Q1
F-
\m ld ( (U,I,, Q)1
- F. Fa/a/2
+O m)),
OUP.
OU.F.Q
where dUPF is same to that in Corollary 9 and F Ta F Ta/2-
In Theorem 10, the term Oie (m) decays to 0 exponentially as sample size increases, and
@(VT) 1 (U.P.Q) F
To
m (U.P.Q) Q) F Ta/2
OU.P.Q
OUP..Q
To ensure that the test based on the learned alternative hypothesis with F achieves higher test power than testing both alternative hypotheses with reduced significance level az/2. it is necessary ensure a large value of 3 >> 0.5. Fortunately, as shown in Figure 2, the probability 3 can readily converge to I even with limited data.
D Additional Experimental Details and Results
D.1 Details of State-of-the-Art Relative Similarity Tests
We compare our AMD test with state-of-the-art relative similarity tests, which include following methods:
MMD-D: Measure relative similarity using the MMD statistic with a selected deep kernel [13];
KIFF: Measure the relative similarity using the witness function of kernel mean embeddings of distributions, which are computed with a selected deep kernel [14].
MMD-H: Measure relative similarity using MMD with a Gaussian kernel selected from median heuristic [13];
UME: Evaluate the mean embeddings of distributions magnitudes of two distances calculated on these embeddings [15];
test locations and measure the relative
SHCE: Train a binary classifier based on neural network and use a statistic about classification accuracy [32];
LBI: Train binary classifier based on deep neural network and use statistic about class probabilities [33].
Several methods, including MMD-D [21], UME [15], SHCE [32], and LBI [33], are extended to relative similarity testing following the framework of [13], replacing MMD with various distance metrics from the two-sample testing literature.
D.2 The Definition of Accuracy Margin
We can test the accuracy margin between source dataset S and target dataset T for model f. Let [(x) represent the probability assigned by the model f to the true label. We define the accuracy margin follows
[Ees| v=)] Exer vw)ll
A smaller margin indicates similar model performance in the source and target dataset.
We present the accuracy margins between the original ImageNet and its variants in Table 3, with the values computed using the pre-trained ResNet50 model.
Table Accuracy margins between the original ImageNet and its variants.
ImageNetsk
ImageNetr
ImageNetv2
ImageNeta
Accuracy Margin
0.529
0.564
0.751
0.827
28

---

D.3 More Experiments
AMD
SHCE (F)
MMD-D (F)
KLFI (F)
UME
LBI (F)
MMD-H (F)
1.0
1.0
Rejection Rate
0.8
Rejection Rate
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Ratio V
Ratio V
(a) CIFAR10
(b) MNIST
Figure The comparisons between the AMD test the baselines combined with the learned F Phase I of the AMD procedure (denoted by the notation (F) to indicate that the hypotheses of previous baselines are now formulated based on F') are presented. We set U = vP + (1 2)Q with E [0, 1]. When < 0.5, Q is closer to U, whereas when > 0.5, P is closer U. All methods, including baselines augmented with F, 0.5 and
perform well across both regimes the performance of the baselines deteriorates for V > 0.5, as they are originally designed to test the prespocified alternative hypothesis Hi d(W, P) > d(W, Q). Notably, when v = 0.5, iee, no relative similarity relationship exists, all methods correctly control the rejection rate (type-I error) at the nominal level 0.05 (indicated by the black dashed line).
0.5. This contrasts with the results shown Figure where
Comparisons with baselines combined with the learned F from Phase I of the AMD procedure.
We compare AMD test with baselines (Appendix D.I) combined with F' (denoted by the notation
(F) indicate that the hypotheses of previous baselines
F) as: 1) MMD-D (FF [13, 21]; 2) KLFI (F) [14]: 3) MMD-H (F) [13]; 4) UME (F) [15]; 5) SHCE (F) [32]; 6) LBI (F) [33]. Following the setups in Figure 1, we present the rejection rates of these methods
now formulated based
CIFAR1O and MNIST with respect to different ratios v. In Figure 6, it is evident that our AMD test achieves higher or comparable rejection rates (i.e.. test power) than others when // 0.5 (i.e., Q is closer to the reference distribution U) and // > 0.5 (i.c., P is closer to the reference distribution U). Notably, when v = 0.5 (i.e., no relative similarity relationship exists and IP and are equally close to the reference distribution U) all methods successfully control the rejection rate (type-I error) at level 0.05 (indicated by the black dashed line).
Table 4: CIFAR10: Test power vs ratio v for two AMD tests: (1) with the learned alternative hypothesis using F. (i.c., AMD), and (2) testing both possible alternative hypotheses (denoted as AMD-B). (Part 1). The bold denotes the highest mean power.
0.0
0.1
0.2
0.3
0.4
AMD
1.00
.000
.008
.804 +: .022
.543 +: .038
.197
.027
AMD-B
1.00
.000
.967 +: .012
.716 4: .22
.399 +: .037
.137
.022
29

---

Table 5: CIFARIO: Test power ratio for AMD tests: (I) with the learned alternative hypothesis using (i.e., AMD), and (2) lesting possible alternative hypotheses (denoted as AMD-B). (Part 2). The bold denotes the highest mean test power.
0.7
0.8
1.0
AMD
.180 .21
.552
.24
.820 + .27
.978 + .005
1.00 0000
AMD-B
.113
.427
.709
.955 1: 1.00
Table 6: MNIST: Test power vs ratio v for two AMD tests: (1) with the learned alternative hypothesis using F (i.e., AMD), and (2) testing both possihle alternative hypotheses (denoted as AMD-B). (Part 1). The bold denotes the highest
power.
0.0
0.1
0.2
0.3
0.4
AMD
1.00
.000
.001
.956 + .012
.487 + .011
.122 0011
AMD-B
1.00
.000
.997 1: .002
.934 1: .016
.379 1: .065
.068 1: .013
Table 7: MNIST: Test power vs ratio for two AMD tests: (1) with the learned alternative hypothesis using (i.c., AMD), and (2) testing both possible alternative hypotheses (denoted as AMD-B). (Part 2). The bold denotes the highest mean test power.
0.6
0.7
0.8
0.9
1.0
AMD
.123 .016
.482
.091
.937 4: .0300
1.00 +: .000
1.00
0000
AMD-B
.079 .011
.390
.800
.904
.999 +:
1.00
Comparisons with testing both possible alternative hypotheses. Following the experimental setup in Figure 1, we compare the test power of two AMD tests: (1) with the learned alternative hypothesis using F (i.e., AMD), and (2) testing both possible alternative hypotheses (denoted as AMD-B). The details of AMD-B are provided Appendix C. From Tables 4, 5, and 7, we can observe that the original AMD test consistently achieves higher comparable test power to AMD-B across wide range of values, particularly when y is closer to 0.0 or 1-0, where the relative similarity relationship more pronounced and F is more likely correctly identify true alternative hypothesis in Phase 1.
Table 8: Test power comparison of different methods in machine-generatee text detection. The bold denotes the highest power.
Method
UMB
SCHE
LKI
MMD-D
MMD-H
KI.FI
AMD
Test Power
.791 .078
8741 .059
.899 .043
.917T .036
.869 .086
.897 .023
1.00 .000
Comparisons in machine-generated text detection. Following [6], we also conduct an experiment showing that our AMD performs well in detecting machine generated text compared to other baselines. Specifically, we randomly draw sample from HC3 dataset, and assess whether the sample machine- generated by testing its relative similarity to numan--wittedd and machine-generatee texts from the TQA dataset. In this experiment, we set sample size to 70. The test power results are shown in Table 8, where AMD test achieves higher test power compared to baselines.
machine-generated
30

---

Table 9: Test power comparison of different methods MNIST and CIFARIO under correct hypotheses (i.e.. for baselines, the specified alternative hypothesis is correct; for AMD, the learned correct). The bold denotes the highest mean test power.
Dataset
MMD-D
KLFI
MMD-H
UME
SHCE
LBI
AMD
MNIST
.772 .112
.897 067
.300
.785=
.793
.710
.968 .023
CIFARIO
.958 076
.936# 042
.383 181
.533= .49
.606 009
.671 129
.983 .017
Table 10: Test power comparison of different methods on MNIST and CIFARIO under misspecified hypotheses
(i.e.. for baselines, the specified alternative hypothesis is incorrect; for AMD, the learned F is incorrect).
Dataset
MMD-D
KLFI
MMD-H
UME
SHCE
LBI
AMD
MNIST
.006+.003
.01 + 004
.018
.005+.002
.047=.011
.007
.001+:001
CIFARIO
.012:t.003
.004::007
+.04
.017:.0044
.021+.004
.01 0.007
.0044.002
Test power comparison under correct and misspecified hypotheses. Tables 9 and 10 compare the test power of various relative similarity testing methods on MNIST and CIFAR 10 under correctly specified and misspecified hypotheses, respectively. Under the correct hypothesis setting (Table 9, i.e., for baselines, the specified alternative hypothesis correct: for AMD, the learned F is correct), AMD achieves the highest power both datasets (0.968 on MNIST and 0.983 on CIFARIO), outperforming all baseline methods by substantial margin. In contrast, under hypothesis misspecification (Table 10, i..,, for baselines, the specified alternative hypothesis is incorrect; for AMD, the learned F is incorrect), all methods exhibit low test power and tend to accept the null hypothesis. As a result, the relative similarity relationship is not evaluated at statistically significant level, rendering the test. ineffective under such misspecification.
Table 11: MNIST: Test power vs. sample size m two AMD tests: (1) optimization divided into two separate cases: 1) d', X, Y) 0: d' (Z, X, statistic is directly optimized, i.c., d (2, X, Y) ? and this method is denoted as AMD-SQ. The bold denotes the highest mean power. The bold denotes the highest mean test power.
(i.e., the original AMD); (2) the squared form of test
130
160
190
220
250
280
310
340
AMD
.77.1.03
.409
.604
.761
:649
.779..048
.827
-037
.867
1.008
.969:.012
AMD-SQ
.063+.021
.78
.922
014
.688
.692- 048
.787+.092 .892 064
Table 12: CIFARI0: Test power vs. sample size m for two AMD tests: (1) the optimization is divided into two
separate cases: 1) d (Z, X, Y) > 0; 2) ( X, < 0 (i.c., the original AMD); (2) the squared form of test
statistic is directly optimized, i.e. d' (Z, X, Y), and this method is denoted as AMD-SQ. The bold denotes the highest mean test power.
m
20
40
60
80
100
120
140
160
AMD
306 +.038
.477 -.06
.627 4.044 .775 =022
.68
+.017
.871
2.02
.922
.00
.968 +.007
AMD-SQ
.286-0054
.452+.046
.591-.046
.713+.028
.811
.834+008
.914#0133
.957-.013
Comparison Between Original AMD and Squared-Form Optimization (AMD-SQ). Tables and 12 present the test power of two AMD tests: (1) the optimization divided into two separate cases: 1)d² (2, X, Y) > 0: 2)d (2, X,M, Y) < 0 (i.e., the original AMD)); (2) the squared form of the test statistic is directly optimized, i..., d (Z, X, 2 and this method is denoted as AMD-SQ. The
31

---

test power results are presented based on MNIST and CIFARIO with respect to different sample sizes m. In both datasets, the original AMD method consistently outperforms AMD-SQ. The performance gap is particularly pronounced in the low-sample regime, which is because directly optimizing the squared form of the test statistic, i.e., (z, X,)2, is highly sensitive to outliers data points with large values [30], especially in high-density regions. This can overshadow important patterns in lower-density regions, potentially leading incomplete understanding of the data when the sample size is limited.
Sample Size
UME (s)
SCHE (s)
MMD-D (s)
MMD-H (s)
KLFI (S)
AMD (s)
50
792.6
158.7
93.8
24.3
104.6
174.3
100
876.2
214.4
181.1
31.0
217.9
411.8
150
931.1
240.9
224.1
37.8
274.1
530.7
200
1230.6
292.6
345.9
44.4
386.0
693.2
250
1675.0
313.7
391.5
50.8
436.0
766.3
300
2077.2
365.7
457.8
57.5
546.9
902.4
Table Runtime comparison (in seconds) different sample sizes on MNIST.
Runtime Comparison. In the Table 13, we provide runtime comparison of AMD against baseline methods across different dataset scales MNIST. AMD has higher time complexity than MMD-D, KLFI, and similar baselines because includes kernel optimization on augmented data. UME the most expensive method due to its feature selection step. In contrast, MMD-H is the fastest, uses a simple heuristic without any optimization. As shown in Tables and 2, and Figure 4, AMD achieves high statistical power even with relatively small sample size, significantly smaller than the full dataset. This indicates that the increased time complexity and any potential loss in power are generally acceptable in practice.
AMD
AMD-TS
0.10
0.30
p-velue ue (log)
p-value (10g
0.25
-08-
0.90
-05
-85
455 6
1774-444
1--34-3-3-11 6
3.7444----11
Regui. Para. (10')
CIFAR10
Regul. Pera. (10')
Regul. Para. (10')
(b)
Regul. Para. (10')
Figure 7: The Type-I error is controlled around 0.05 w.r.t different regularization parameters for AMD and AMD-TS tests.
-0.5
-0.5-
p-valie [lg)
Typo- error
p-value (lgg)
Type-
0.55
-0.7
0.05
3.7
-0.9
-3.9
240
200
100
120 220
200 130 280 220 310 340
Sample Size
(a) CIFAR10
Sample
Sample
(b) MNIST
Sample
Figure Type-I error is controlled around a = 0.05 w.r. different sample sizes for our AMD and the state-of- thc-art rolative similarity tests.
Type-I error results. To conduct the experiments on Type-I error, we set U = 0.5P 0.5Q, which indicates that the relative similarity relationship does not exist, i.e., TP and are equally close the reference distribution U. Figure 8 demonstrates that the Type-I errors of both AMD test and
32

---

tfateoof-the-n relative similarity tests are consistently controlled around a = 0.05 across various sample sizes in relative similarity testing. Similarly, Figure illustrates that the Type-I errors for our AMD and AMD-TS tests remain controlled around a 0.05 across different regularization parameters {10-8, 10-7, 10-6, 10-5, 10-1 10-3 100- 10-1 1,100, 10 102, 10 104) and datasets. These findings align well with the theoretical guarantees provided in Lemma 4. The p-values presented these Figures also align with the findings derived from the Type-I error results, providing additional support for the effectiveness of our approach.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and Section clearly outline the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings.
It fine include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiiatinn, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification:
Appendix A contains full proofs and assumptions of our theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections 5 and Appendix D detail the implementation ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case closed-source models, it may be that access the model limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification: Codes are available at:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/ public/guidest CodeSubmi ssi
wwtprbiissssooooooiiiooooio onPolicy) for more details.
While we encourage the release code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/niis. clss.ccppulll htc/public/guuii
ssi oniooiiy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Sections 5 and Appendix detail experimental settings. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: results.
We 1,000 repetitions to ensure statistical significance of the experimental
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
36

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix D details the computational resources required for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / //wrrreuripp..e/
Answer: [Yes]
Justification:
This research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special considera- tion due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The study presents fundamental research without direct real-world deployment, so societal impact is not applicable at this stage.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedhack over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper presents no associated risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used are properly credited and comply with their respective licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-pa re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
39

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htt::///eurrpppi wwwce//eeeeeececcc///cccaaacceee wrweceeeenncee.... /LLM) for what should should not be described.
40


Reviews:
[{"review_id": "C8a0RRfC7y", "reviewer": "Reviewer_N9H2", "summary": "This paper introduces Anchor-based Maximum Discrepancy (AMD), a novel kernel-based metric for relative similarity testing that jointly learns both the hypothesis and the kernel rather than fixing one a priori. AMD measures the largest discrepancy between distances of two candidate distributions to an anchor distribution over a rich space of deep kernels, and its two-phase procedure first selects the kernel maximizing this discrepancy and then assesses statistical significance via a unified wild-bootstrap test. Experiments on image datasets and real-world tasks demonstrate that AMD achieves superior performance over existing baselines.", "questions": "1. How does the computational runtime of AMD compare to the baselines at different dataset scales?\n\n2. See the weaknesses part.", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well-constructed and easy to follow, the research objective and methodology are clearly exposed.\n\n2. The problem studied in this paper is important, and the theoretical guarantees as well as the validation of AMD in practical applications are good.\n\n\nWeaknesses\n\n1. The computational overhead of Phase I may limit the scalability of method. The optimization procedure over a deep kernel space is likely to be significantly more expensive than baseline methods that use a fixed kernel or a simple heuristic.\n\n2. The experiments mainly focus on image data and a small number of datasets, making it unclear how well the method generalizes to text, sequences, or more complex structured data.", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for taking the time to review our manuscript. We  address your questions as follows.\n\n[Q1] The computational overhead of Phase I may limit the scalability of method. The optimization procedure over a deep kernel space is likely to be significantly more expensive than baseline methods that use a fixed kernel or a simple heuristic.  How does the computational runtime of AMD compare to the baselines at different dataset scales?\n\n>[A1] Thank you for raising this question. We provide a comparison of the computational runtime of AMD against the baselines at various dataset scales, using MNIST as an example. We will add the runtime analysis to our paper.\n| Sample Size | AMD (s) | UME (s) | SCHE (s) | MMD‑D (s) | MMD‑H (s) | KLFI (s) |\n|-------------|---------|---------|----------|-----------|-----------|----------|\n| 50          | 174.3   | 792.6   | 158.7    | 93.8      | 24.3      | 104.6    |\n| 100         | 411.8   | 876.2   | 214.4    | 181.1     | 31.0      | 217.9    |\n| 150         | 530.7   | 931.1   | 240.9    | 224.1     | 37.8      | 274.1    |\n| 200         | 693.2   | 1230.6  | 292.6    | 345.9     | 44.4      | 386.0    |\n| 250         | 766.3   | 1675.0  | 313.7    | 391.5     | 50.8      | 436.0    |\n| 300         | 902.4   | 2077.2  | 365.7    | 457.8     | 57.5      | 546.9    |\n\n>AMD exhibits higher time complexity compared to MMD-D, KLFI, and similar baselines, primarily due to the kernel optimization phase involving augmented data. Among all methods, UME incurs the highest computational cost, as it performs feature selection through optimization. In contrast, MMD-H demonstrates the lowest time complexity, benefiting from a heuristic kernel selection strategy that avoids any optimization.\n\n[Q2] The experiments mainly focus on image data and a small number of datasets, making it unclear how well the method generalizes to text, sequences, or more complex structured data.\n\n>[A2] Thanks for raising this question. We present an experiment over the task of machine-generated text detection. Following [Zhang et al., 2024], we randomly draw samples from the HC3 dataset and assess whether each sample is machine-generated by testing its relative similarity to human-written and machine-generated texts from the TQA dataset. In this experiment, we set the sample size to 70. The test power results are shown in the table above, where AMD achieves higher test power compared to the baselines.\n| Method        | AMD          | UME         | SCHE        | LKI         | MMD‑D       | MMD‑H       | KLFI        |\n|---------------|--------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Test Power | **1.00 ± 0.000** | 0.791 ± 0.078 | 0.874 ± 0.059 | 0.839 ± 0.043 | 0.917 ± 0.036 | 0.869 ± 0.086 | 0.897 ± 0.023 |\n\n---\n\n**We hope our response has clarified the confusion and addressed your concerns. We would greatly appreciate it if you could kindly reconsider your score. Thank you. :)**"}, {"review_id": "AKNNFuLI2k", "reviewer": "Reviewer_yaCd", "summary": "This paper proposes a new relative similarity test for the question of which distribution is closer given two alternative distributions. In particular, this paper's proposed method address the existing issues in previous methods, which is ill-defined as formulating the hypothesis first and then find the correct kernel. Instead, the proposed method formulates the hypothesis and the kernel to be used at the same time. More detailly, the paper designs a two-stage method to address this issue. the effectiveness of the method is evaluated through experiments.", "questions": "I am not familiar with this topic, thus I am unable to find a suitable question.", "limitations": "not applicable.", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "strengths:\n\n1. the problem this paper tries to address is very important. Given the previous problem is ill-defined, identifying the hypothesis and kernel at the same time is particular important to fully address this problem. \n\n2. this paper has presented solid theoretical derivation to get the conclusion. The proof technique would be useful to the community. \n\nWeakness:\n\n1. As I am not an expert in this field, I did not find the weaknesses in this paper.", "was_revised": true, "final_justification": "My view keeps the same. Thanks for the paper.", "rebuttal": "Many thanks for your supportive comments and for taking the time to review our manuscript. :)"}, {"review_id": "ChppLspaIR", "reviewer": "Reviewer_v8QC", "summary": "The paper proposes anchor-based maximum discrepancy, a method for relative similarity testing that test if one distribution is closer to a reference distribution than another. To avoid the asymmetry of hypothesis formation, the paper proposes to use a new way with a constraint that allows for a symmetric formulation.", "questions": "- What is the rationale of the constraint in equation 1.\n- What is the formula for the constraint in the kernel formulation?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "I find it not sufficiently clear at the rather beginning of the paper that requires clarification as follows:\n- Formula (1) introduce the constraint with sgn function. I could not find the rationale of this constraint, except for the convenience of its manipulation.\n- Where does the constraint disappear when used in the kernel space K?\nI can not validate the paper further than this point and wait for the authors' rebuttal.", "was_revised": true, "final_justification": "I'd raise slightly the rating after the rebuttal. I cannot fight for its acceptance given the current state but probably it can be accepted.", "rebuttal": "Many thanks for taking the time to review our manuscript. We acknowledge that the **unnecessary constraint in our original presentation of formula (1) may have caused confusion**, and we appreciate your careful reading. Our intention was to highlight the connection between the AMD statistic and integral probability metrics (IPMs), but the constraint was not essential to this relationship. We address this question in detail as follows.\n\n[Q] Concerns on the formula (1)\n- Formula (1) introduce the constraint with sgn function. I could not find the rationale of this constraint, except for the convenience of its manipulation.\n- Where does the constraint disappear when used in the kernel space K? I can not validate the paper further than this point and wait for the authors' rebuttal.\n- What is the rationale of the constraint in equation 1.\n- What is the formula for the constraint in the kernel formulation?\n\n>[A] Thank you for pointing out this issue. We apologize for the confusion created by the **unnecessary constraint** in our original presentation with formula 1. Our primary intention was to illustrate the relationship between our AMD statistic (Definition 1) and the class of integral probability metrics (IPM). In fact, this relationship follows immediately from the defining equation of AMD as follows\n$$\n\\sup_{f_k\\in \\mathcal F_{\\mathcal K}}\\left|\\int f_k dU-\\int f_kd\\frac{P+Q}{2}\\right|=\\sup_{k\\in \\mathcal K}\\left|\\langle\\mu_U-\\frac{\\mu_P+\\mu_Q}{2},\\mu_P-\\mu_Q\\rangle\\right|\\ .\n$$\nThe left part of the equation can be viewed as an IPM between the distributions $U$ and $(P+Q)/2$.\n\n>For clarity, we will revise the manuscript so that we:\n>- Introduce AMD in its native form (Definition 1), without the extraneous constraint.\n>- Directly derive its equivalence to an IPM, using the equation above, rather than presenting the IPM form first and then fitting AMD into it.\n\n>We regret any confusion this may have caused. Importantly, **the methods and analyses in the paper do not rely on the removed constraint**, so this correction will not affect the correctness or validity of our main results.\n\n>Thank you again for your careful reading and for helping us improve the clarity of our presentation. \n\n---\n\n**We hope our responses have helped clarify the confusion and addressed your concerns. We would be grateful if you would kindly consider revisiting your score. If you have any further questions or need additional information, please don’t hesitate to reach out. Thank you. :)**"}, {"review_id": "7V3nxBO4XP", "reviewer": "Reviewer_ZHeZ", "summary": "This paper introduces a new statistical method for relative similarity testing - essentially answering \"which of two datasets/distributions (P or Q) is more similar to a reference dataset (U)?\" According to the author(s), existing methods have a fundamental flaw: they require one to manually specify one's hypothesis first (like \"Q is closer to U than P\"), then find a kernel to test it. But this approach is flawed because one can always find some kernel that will reject any hypothesis chosen, making the test meaningless. This paper addresses this issue.", "questions": "1. To represent a distribution, especially a complex one, what is the typical sample size?  and how are sample selected?\n2. The method solves one problem (hypothesis specification bias) but potentially creates others (overfitting, computational cost, reduced power).  How do you balance these two sides?\n3.The proofs are technically accurate but practically incomplete.  Do you prove properties for an idealized version that may not match the actual algorithm's behavior?", "limitations": "I wish there is a summary provided.  But the authors stated that the limitations are presented throughout the paper...", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengthens:\n1. The work proposes learning both the hypothesis and the kernel simultaneously using \"anchor-based maximum discrepancy (AMD)\".\n2. Works with samples, not full distributions.\n3. Practical applications are several such as data drift detection, model comparison, and A/B testing\n4. This method removes human bias from hypothesis selection, which has been a major weakness in existing approaches.\n\nWeaknesses:\n1. The method is potentially computational expensive, and sample selection can be challenging.\n2. Simultaneously learning both hypothesis and kernel on the same data could lead to overfitting, especially with limited samples.\n3. The paper doesn't clearly address the trade-off between avoiding manual specification bias and losing the benefits of domain expertise. Sometimes human insight about what constitutes \"similarity\" is actually valuable and shouldn't be discarded.\n4. For the proofs documented in the appendices, the mathematical logic within stated assumptions is correct, but there are gaps between theory and practice:\n--- The asymptotic theory is sound but may not apply to realistic deep kernel settings\n--- Critical assumptions are never verified for the proposed method\n--- The optimization/learning aspects are divorced from the statistical theory", "was_revised": true, "final_justification": "The submission lacked the required limitations section per the checklist. During the rebuttal phase, the authors acknowledged this omission, expressed willingness to address it, and committed to providing one.", "rebuttal": "Many thanks for taking the time to review our manuscript. We answer your questions as follows.\n\n[Q1] The method is potentially computational expensive. The method solves one problem (hypothesis specification bias) but potentially creates others (overfitting, computational cost, reduced power). How do you balance these two sides?\n\n>[A1] Thank you for raising this question. We will include a more detailed discussion on this point in the revised version of our paper.\n>To address the fundamental challenge of kernel based relative similarity testing, namely removing human bias in hypothesis selection, we jointly learn both the hypothesis and the kernel. This added flexibility comes at a cost, as there is no free lunch. However, these costs are generally acceptable in practice. Specifically, for computational complexity, we provide a runtime comparison of AMD against baseline methods across different dataset scales using MNIST as an example.\n| Sample Size | AMD (s) | UME (s) | SCHE (s) | MMD‑D (s) | MMD‑H (s) | KLFI (s) |\n|-------|------|------|-------|--------|--------|------|\n| 50          | 174.3   | 792.6   | 158.7    | 93.8      | 24.3      | 104.6    |\n| 100         | 411.8   | 876.2   | 214.4    | 181.1     | 31.0      | 217.9    |\n| 150         | 530.7   | 931.1   | 240.9    | 224.1     | 37.8      | 274.1    |\n| 200         | 693.2   | 1230.6  | 292.6    | 345.9     | 44.4      | 386.0    |\n| 250         | 766.3   | 1675.0  | 313.7    | 391.5     | 50.8      | 436.0    |\n| 300         | 902.4   | 2077.2  | 365.7    | 457.8     | 57.5      | 546.9    |\n\n>As shown in the above Table, AMD has higher time complexity than MMD-D, KLFI, and similar baselines because it includes kernel optimization on augmented data. UME is the most expensive method due to its feature selection step. In contrast, MMD-H is the fastest, as it uses a simple heuristic without any optimization. As shown in Table 1 and Figure 4, AMD achieves high statistical power even with a relatively small sample size, significantly smaller than the full dataset. This indicates that the increased time complexity and any potential loss in power are generally acceptable in practice. In addition, regarding the concern of overfitting, Figure 2 shows that the learned hypothesis remains accurate and captures the true relative similarity relationship with high probability even when using a limited sample size (20 for CIFAR10 and 50 for MNIST). This suggests that the learned hypothesis is fairly robust.\n\n[Q2] To represent a distribution, especially a complex one, what is the typical sample size? and how are sample selected? ... sample selection can be challenging.\n>[A2] Thank you for raising this question. We find that there might be a misunderstanding regarding our paper: we do **not need to represent a distribution**. Our approach evaluates the relative similarity among three distributions: two candidates, P and Q, and a reference distribution U, by determining which of P or Q is closer to U. Instead of explicitly representing each distribution, which is challenging and requires many samples, we rely on statistical distances between distributions, making the method more sample efficient. In our setting, following standard practice, we apply the test directly to two i.i.d. samples drawn from P and Q without any specific data selection.\n\n\n[Q3] Simultaneously learning both hypothesis and kernel on the same data could lead to overfitting, especially with limited samples.\n>[A3] Thank you for your question. As demonstrated in Lemma 4, our method ensures control of the type I error. This theoretical guarantee is further supported by the empirical results presented in Figures 7 and 8 (Appendix D.4), which show that type I error remains controlled even when the sample size is small. Consequently, even if overfitting results in an incorrect alternative hypothesis being learned, the probability of incorrectly accepting this hypothesis corresponds to the type I error, which is bounded by the chosen significance level $\\alpha$.\n\n[Q4] The paper doesn't clearly address the trade-off between avoiding manual specification bias and losing the benefits of domain expertise. Sometimes human insight about what constitutes \"similarity\" is actually valuable and shouldn't be discarded.\n\n>[A4] Thank you for your question. In this paper, our motivation is to address the problem of manual specification bias when domain knowledge is not available. In such cases, people often randomly guess an alternative hypothesis and perform the test accordingly, which can result in bias. In AMD, we learn the hypothesis, and as mentioned in [A1], Figure 2 shows that the learned hypothesis captures the true relative similarity relationship with high probability, even when using a limited sample size (20 for CIFAR10 and 50 for MNIST). \n\n>**Our method is also applicable in cases where domain knowledge is available.** In such cases, we use the manually specified hypothesis and learn the kernel directly based on it (by performing only one optimization in Phase 1: the one in line 193 or those in line 194). Then, we proceed with Phase 2 as usual. We include experimental results to demonstrate the effectiveness of AMD under this setting.\n| Dataset  | **AMD**          | MMD‑D        | KLFI         | MMD‑H        | UME          | SHCE         | LBI          |\n|----------|------------------|--------------|--------------|--------------|--------------|--------------|--------------|\n| MNIST    | **0.968 ± 0.023** | 0.772 ± 0.112 | 0.897 ± 0.067 | 0.300 ± 0.165 | 0.785 ± 0.065 | 0.793 ± 0.082 | 0.710 ± 0.064 |\n| CIFAR10  | **0.983 ± 0.017** | 0.958 ± 0.076 | 0.936 ± 0.042 | 0.383 ± 0.181 | 0.533 ± 0.049 | 0.606 ± 0.069 | 0.671 ± 0.129 |\n\n>Given the known specified hypothesis, AMD achieves the highest power on both datasets (0.968 on MNIST and 0.983 on CIFAR10).\n\n>We will revise our paper to include these clarifications and thank you again for the valuable feedback.\n\n[Q5] For the proofs documented in the appendices, the mathematical logic within stated assumptions is correct, but there are gaps between theory and practice: --- The asymptotic theory is sound but may not apply to realistic deep kernel settings --- Critical assumptions are never verified for the proposed method --- The optimization/learning aspects are divorced from the statistical theory\n>[A5] Thank you for the insightful comments. We acknowledge that narrowing the gap between theory and practice is an important direction for future work. We attempted to further investigate the behavior of the test statistic under optimization but were unsuccessful due to the challenges posed by unknown underlying distributions and the inherent randomness of optimization algorithms.\n\n>Although AMD involves an optimization procedure and assumes almost sure convergence in Theorem 3, **the core theoretical properties that ensure the validity of AMD in hypothesis testing are independent of this assumption and remain robust to the outcomes of the optimization.**\n>- **Lemma 4 (Type-I Error Control):** This result does not depend on the specific outcome of the optimization. That is, **regardless of which deep kernel (which is bounded as shown in Eqn.(5)) is selected through optimization, the type-I error is guaranteed to be controlled** (Liu et al., 2020; Huang et al., 2023). This theoretical guarantee is further supported by the empirical results presented in Figures 7 and 8 (Appendix D.4), which show that type-I error remains controlled even when the sample size is small.\n>- **Theorem 5 (Test Power Guarantee):** This theorem shows that AMD achieves higher test power **as long as the learned hypothesis is better than random guessing, i.e., when $\\beta > 0.5$.** As discussed in [A1], Figure 2 empirically demonstrates that the learned hypothesis successfully captures the true relative similarity with high probability, even with a small sample size (20 for CIFAR10 and 50 for MNIST). Moreover, **the asymptotic distribution used in the test is independent of the optimization procedure and remains valid for any deep kernel obtained through optimization** (Bounliphone et al., 2016).\n\n>We will incorporate these clarifications into our revision to improve the theoretical exposition.\n\n>[1] Bounliphone W, Belilovsky E, Blaschko M, et al. A test of relative similarity for model selection in generative models. ICLR 2016.\n\n>[2] Huang B, Liu Y, Peng L. Weighted bootstrap for two-sample u-statistics. Journal of Statistical Planning and Inference, 2023.\n\n>[3] Liu F, Xu W, Lu J, et al. Learning deep kernels for non-parametric two-sample tests. ICML,  2020.\n\n[Q6] The proofs are technically accurate but practically incomplete. Do you prove properties for an idealized version that may not match the actual algorithm's behavior?\n>[A6] Thank you for this insightful question. Theoretical characterization of our algorithm’s behavior under unknown, complex distributions remains an open problem. Although we attempted such an analysis, the tight coupling between stochastic optimization dynamics and finite‑sample statistical behavior proved too challenging to resolve. We consider this an important and interesting problem, and plan to explore it in future work.\n\n\n[Q7] Limitations. I wish there is a summary provided. But the authors stated that the limitations are presented throughout the paper...\n>[A7] Thank you for the suggestion. We will add a concise summary of our paper’s limitations, including the potential problems of our method (overfitting, time complexity, and reduced power), specifically highlighting the covering‑number assumption in Theorem 3 and the bounded‑kernel requirement in Lemma 4 and Theorem 5.\n\n---\n\n**We hope our responses have helped clarify the confusion and addressed your concerns. We would be grateful if you would kindly consider revisiting your score. If you have any further questions or need additional information, please don’t hesitate to reach out. Thank you. :)**"}]

Author rebuttal:

Meta-review: The paper considers an interesting and challenging problem: testing whether P or Q is closer to U. The paper argues that existing two-sample test kind of statistics/tests are not effective. Motivated by IPMs, a new statistic (2) is proposed, which is very intuitive. Details of estimating the statistic are discussed, and provably consistent estimators are presented. Then details of a hypothesis test around this novel statistic are presented. 

The paper is very well written with clear motivation, methodology and theoretical results proving the correctness. Simulations also verify the theory. Two application scenarios are also presented: Relative Model Performance Evaluation and Adversarial Perturbation Detection, highlighting the efficacy of the method.

In summary, I think the paper is a clear accept with no reservations (though some scores may appear boderline).
Decision: Accept (poster)