Title: Anchor-based Maximum Discrepancy for Relative Similarity Testing
Venue: NeurIPS 2025
Accepted: yes
Keywords: relative similarity testing

kernel-based hypothesis testing

deep kernel
Abstract: The relative similarity testing aims to determine which of the distributions, $P$ or $Q$, is closer to an anchor distribution $U$. Existing kernel-based approaches often test the relative similarity with a fixed kernel in a manually specified alternative hypothesis, e.g., $Q$ is closer to $U$ than $P$. Although kernel selection is known to be important to kernel-based testing methods, the manually specified hypothesis poses a significant challenge for kernel selection in relative similarity testing: Once the hypothesis is specified first, we can always find a kernel such that the hypothesis is rejected. This challenge makes relative similarity testing ill-defined when we want to select a good kernel after the hypothesis is specified. In this paper, we cope with this challenge via learning a proper hypothesis and a kernel simultaneously, instead of learning a kernel after manually specifying the hypothesis. We propose an anchor-based maximum discrepancy (AMD), which defines the relative similarity as the maximum discrepancy between the distances of $(U, P)$ and $(U, Q)$ in a space of deep kernels. Based on AMD, our testing incorporates two phases. In Phase I, we estimate the AMD over the deep kernel space and infer the potential hypothesis. In Phase II, we assess the statistical significance of the potential hypothesis, where we propose a unified testing framework to derive thresholds for tests over different possible hypotheses from Phase I. Lastly, we validate our method theoretically and demonstrate its effectiveness via extensive experiments on benchmark datasets. Codes are publicly available at: https://github.com/tmlr-group/AMD.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new statistical method for relative similarity testing - essentially answering \"which of two datasets/distributions (P or Q) is more similar to a reference dataset (U)?\" According to the author(s), existing methods have a fundamental flaw: they require one to manually specify one's hypothesis first (like \"Q is closer to U than P\"), then find a kernel to test it. But this approach is flawed because one can always find some kernel that will reject any hypothesis chosen, making the test meaningless. This paper addresses this issue.\n\nSTRENGTHS AND WEAKNESSES:\nStrengthens:\n1. The work proposes learning both the hypothesis and the kernel simultaneously using \"anchor-based maximum discrepancy (AMD)\".\n2. Works with samples, not full distributions.\n3. Practical applications are several such as data drift detection, model comparison, and A/B testing\n4. This method removes human bias from hypothesis selection, which has been a major weakness in existing approaches.\n\nWeaknesses:\n1. The method is potentially computational expensive, and sample selection can be challenging.\n2. Simultaneously learning both hypothesis and kernel on the same data could lead to overfitting, especially with limited samples.\n3. The paper doesn't clearly address the trade-off between avoiding manual specification bias and losing the benefits of domain expertise. Sometimes human insight about what constitutes \"similarity\" is actually valuable and shouldn't be discarded.\n4. For the proofs documented in the appendices, the mathematical logic within stated assumptions is correct, but there are gaps between theory and practice:\n--- The asymptotic theory is sound but may not apply to realistic deep kernel settings\n--- Critical assumptions are never verified for the proposed method\n--- The optimization/learning aspects are divorced from the statistical theory\n\nQUESTIONS:\n1. To represent a distribution, especially a complex one, what is the typical sample size?  and how are sample selected?\n2. The method solves one problem (hypothesis specification bias) but potentially creates others (overfitting, computational cost, reduced power).  How do you balance these two sides?\n3.The proofs are technically accurate but practically incomplete.  Do you prove properties for an idealized version that may not match the actual algorithm's behavior?\n\nLIMITATIONS:\nI wish there is a summary provided.  But the authors stated that the limitations are presented throughout the paper...", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
The paper proposes anchor-based maximum discrepancy, a method for relative similarity testing that test if one distribution is closer to a reference distribution than another. To avoid the asymmetry of hypothesis formation, the paper proposes to use a new way with a constraint that allows for a symmetric formulation.

STRENGTHS AND WEAKNESSES:
I find it not sufficiently clear at the rather beginning of the paper that requires clarification as follows:
- Formula (1) introduce the constraint with sgn function. I could not find the rationale of this constraint, except for the convenience of its manipulation.
- Where does the constraint disappear when used in the kernel space K?
I can not validate the paper further than this point and wait for the authors' rebuttal.

QUESTIONS:
- What is the rationale of the constraint in equation 1.
- What is the formula for the constraint in the kernel formulation?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper proposes a new relative similarity test for the question of which distribution is closer given two alternative distributions. In particular, this paper's proposed method address the existing issues in previous methods, which is ill-defined as formulating the hypothesis first and then find the correct kernel. Instead, the proposed method formulates the hypothesis and the kernel to be used at the same time. More detailly, the paper designs a two-stage method to address this issue. the effectiveness of the method is evaluated through experiments.

STRENGTHS AND WEAKNESSES:
strengths:

1. the problem this paper tries to address is very important. Given the previous problem is ill-defined, identifying the hypothesis and kernel at the same time is particular important to fully address this problem. 

2. this paper has presented solid theoretical derivation to get the conclusion. The proof technique would be useful to the community. 

Weakness:

1. As I am not an expert in this field, I did not find the weaknesses in this paper.

QUESTIONS:
I am not familiar with this topic, thus I am unable to find a suitable question.

LIMITATIONS:
not applicable.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

comments: SUMMARY:
This paper introduces Anchor-based Maximum Discrepancy (AMD), a novel kernel-based metric for relative similarity testing that jointly learns both the hypothesis and the kernel rather than fixing one a priori. AMD measures the largest discrepancy between distances of two candidate distributions to an anchor distribution over a rich space of deep kernels, and its two-phase procedure first selects the kernel maximizing this discrepancy and then assesses statistical significance via a unified wild-bootstrap test. Experiments on image datasets and real-world tasks demonstrate that AMD achieves superior performance over existing baselines.

STRENGTHS AND WEAKNESSES:
Strengths

1. The paper is well-constructed and easy to follow, the research objective and methodology are clearly exposed.

2. The problem studied in this paper is important, and the theoretical guarantees as well as the validation of AMD in practical applications are good.


Weaknesses

1. The computational overhead of Phase I may limit the scalability of method. The optimization procedure over a deep kernel space is likely to be significantly more expensive than baseline methods that use a fixed kernel or a simple heuristic.

2. The experiments mainly focus on image data and a small number of datasets, making it unclear how well the method generalizes to text, sequences, or more complex structured data.

QUESTIONS:
1. How does the computational runtime of AMD compare to the baselines at different dataset scales?

2. See the weaknesses part.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-04