DUAL: Learning Diverse Kernels for Aggregated Two-sample and Independence Testing
Authors: Zhijian Zhou
Xunye Tian
Liuhua Peng
Chao Lei
Antonin Schrab
Danica J. Sutherland
Feng Liu
Venue: NeurIPS 2025 poster ()
Keywords: two-sample testing
independence testing
diverse kernels
aggregated U-statistic
Primary area: other

Abstract: To adapt kernel two-sample and independence testing to complex structured data, aggregation of multiple kernels is frequently employed to boost testing power compared to single-kernel tests. However, we observe a phenomenon that directly maximizing multiple kernel-based statistics may result in highly similar kernels that capture highly overlapping information, limiting the effectiveness of aggregation.
To address this, we propose an aggregated statistic that explicitly incorporates kernel diversity based on the covariance between different kernels. Moreover, we identify a fundamental challenge: a trade-off between the diversity among kernels and the test power of individual kernels, i.e., the selected kernels should be both effective and diverse. This motivates a testing framework with selection inference, which leverages information from the training phase to select kernels with strong individual performance from the learned diverse kernel pool. We provide rigorous theoretical statements and proofs to show the consistency on the test power and control of Type-I error, along with asymptotic analysis of the proposed statistics. Lastly, we conducted extensive empirical experiments demonstrating the superior performance of our proposed approach across various benchmarks for both two-sample and independence testing.

DUAL: Learning Diverse Kernels for Aggregated Two-sample and Independence Testing
Zhijian Zhou'
Xunye Tian Liuhua Peng Chao Lei
Antonin Schrab
Danica J. Sutherland Feng Liu
University of Melbourne
University of Cambridgel
UBC & Amii
{zhi anzhou mml, xunyetian. xunyetian.mly@@maal..com
liuhua l pwwgeunimelb...........o edu.au
clei1@student. wwwmelb. edu.
afls20cam.ac uc
dsuth0cs .ub..ca mingiiu.m@@gmall com
Abstract
To adapt kernel two-sample and independence testing to complex structured data, aggregation of multiple kernels is frequently employed to boost testing power compared to single-kernel tests. However, we observe a phenomenon that directly maximizing multiple kerne-bbaeed statistics may result in highly similar kernels that capture highly overlapping information, limiting the effectiveness of aggregation. To address this, we propose aggregated statistic that explicitly incorporates kernel diversity based on the covariance between different kernels. Moreover, we identify fundamental challenge: trade-off between the diversity among kernels and the test power of individual kernels, i.e., the selected kernels should be both effective and diverse. This motivates testing framework with selection inference, which leverages information from the training phase to select kernels with strong individual performance from the learned diverse kernel pool. We provide rigorous theoretical statements and proofs to show the consistency on the test power and control of Type-I error, along with asymptotic analysis of the proposed statisties. Lastly, we conducted extensive empirical experiments demonstrating the superior performance of our proposed approach across various benchmarks for both two- sample and independence testing.
Introduction
In modern machine learning, non- con-parametric hypothesis tests have become essential for comparing probability distributions and detecting statistical dependencies without imposing restrictive model assumptions. Kernel-based methods provide powerful framework for these tasks by embedding probability distributions into reproducing kernel Hilbert spaces (RKHS), enabling rigorous yet flexi- ble measures of discrepancy and dependence [1, 2]. For example, the Maximum Mean Discrepancy (MMD) is prominent kernel two-sample test metric used to determine whether two sets of observa- tions originate from the same distribution [1, 3-12]. Similarly, the Hilbert-Schmidt Independence Criterion (HSIC) is related method designed to measure statistical dependence between random variables, thus serving as a test of independence [2, 13-16]. They use kernel methods to enhance sta- tistical power and are widely adopted in machine learning fields including domain adaptation [17, 18], generative modeling [19]. adversarial learning [20], machine-geet generrreed text detection [21, 22], causal discovery [23], semi-supervised representation learning [24), continual learning [25], and more.
Related works. Kernel aggregation, combining multiple kernels into single test procedure, has proven to be highly effective in non-parametric hypothesis testing, often yielding substantial gains
Equual contribution. 'Code: https: http://ggttttb.ooc httpg/gttt...m//llrrr.e//oee/eee hatpa/booccooomolrgoooo/////o DWR.///DDD-SI--DALLL.L DUAL.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

in statistical power [7. 9, 26]. This is because the choice of single kernel is critical: even though well-chosen kernel can greatly enhance test's ability to detect departures from the null hypothesis, a poorly chosen kernel may fail to capture the relevant differences. To mitigate the risk of selecting a suboptimal kernel, common strategy to aggregate test statistics across a collection of kernels rather than relying on any single kernel. Such merte- aggregation approaches have been widely adopted in various hypothesis testing scenarios - including two-sample testing [7-9, 27, 28] and independence testing [7, 27, 29, 30] and have consistently demonstrated improved test power (i.e., the probability of correctly rejecting the null hypothesis under the alternative) and adaptivity. In these works, aggregating kernels with different bandwidths or characteristics enables the tests to capture broad range of potential data structures, often achieving higher power than neggl-keerel methods.
Motivations. Aggregating multiple statistics with di- verse kernels a powerful approach to capture complex distributional characteristics and achieve higher test power in hypothesis testing. However, our research uncovers a core limitation that, contrary to conventional wisdom, not all kernels contribute to the test power, and simply using more kernels does not always imply higher power. In other words, the inclusion of uninformative or redundant kernels can potentially reduce the effectiveness of the test. As illustrated in Figure 1, we evaluate the test power of the two-sample test using set of 20 kernels, implemented [28], and also perform the test using randomly selected subsets of 5 kernels. Our results demonstrates that the performance varies significantly across different subsets, and in some cases, the aggregation over a small subset of kernels can indeed outperform the aggregation of full set of kernels, indicating that indiscriminately ensembling various kernels may introduce redundancy that dilutes the test's power. Thus, our work highlights the importance of identifying and utilizing informative kernels rather than relying on a large. potentially redundant collection.
set bernels
1.0
subsenl
subsot3
0.8
sabwetd
Test Power
suesets
0.6
0.4
0.2
0.0
100
300
500
Sample Size
Figure 1: Comparing the test power of ag- gregating different sets kernels the two- sample testing problem on the BLOB dataset. The solid blue line shows the performance when aggregating all 20 kernels. The five dotted lines represent the test power when aggregating five different randomly selected subsets (each containing 5 kemels).
Contributions. We propose a new kernel aggregating methods in hypothesis testing, Diverse U- statistic Aggregation with Learned kernels (DUAL), which improves kernel-based nonparametric tests (e.g.. MMD and HSIC) by introducing a notion of diversity inspired by ensemble learning, where diversity among base models is crucial for performance [33-35]. In particular, DUAL computes the covariance matrix of U'-staiistics obtained from multiple kernels and leverages it to quantify the pairwise diversity among these kemnels. Building on this diversity measure, we develop a novel test statistic that integrates each kernel's U-statistic with the pairwise diversity between kernels, thereby capturing complementary information and improving test sensitivity. Furthermore. we employ post-selection inference in the testing procedure to adaptively maximize test power through informed kernel selection while rigorously controlling the Type-I error rate. As result, DUAL provides a general power enhancement for both two-sample and independence tests instatiisttttittedd as MMD-DUAL and HSIC-DUAL which harnesses the benefits of aggregating multiple kernels while mitigating the influence of uninformative or weak kernels. We provide theoretical guarantees for the proposed approach (including valid Type-I error extensive empirical validation on diverse benchmarks, demonstrating that DUAL-based tests achieve strong performance relative to existing of-the-rrr methods.
error control and improved asymptotic power) and present
2 Preliminaries
In this section, we provide background about the non- non-paramen con-arametric hypothesis testing problems that we are interested in, including both MMD two-sample testing and HSIC independence testing. To begin, we introduce the concept of the second-order U-statistic with a kernel K, which is a key statistical tool. Suppose we have a random sample W (wi, u2...., wn) } from some distribution W on separable metric space W. Let h(w1, W2; x) be a symmetric function of two arguments defined over
'In this work, we aggregate multiple test statistics into single statistic [9. 28, 31J. initially via a sum for simplicity, eventually via weighted &a norm. This differs multiple testing across different kernels [8, 29, 32).

---

the kernel K. The second-order U-statistic, with computational complexity O(n²), is defined as
U =
12 2
Σ
,( Wig
For two-sample and independence tests, the widely used kernel-based test statistics MMD and HSIC, respectively, can each be formulated within the framework of U-statistics, which we introduce below. See [36] for detailed introduction to these, and [31] for overview of MMD/HSIC testing results.
Two-sample Test with MMD. Let and Q denote two unkown Borel probability measures over an instance space X' C Rd, and draw samples X {æ, J P and Y {y, ر== Q". We aim to test the null hypothesis Ho : IP = Q against the alternative hypothesis Hi P + Q The key step in this test is to quantify the discrepancy between distribution I and Q Writing W = [w,)-= = {( y W an unbiased estimate of the squared MMD [3] with kernel is
MMD (W)
in (10,, Wi2)
I5h. <inSn
(MM (xc y). (x', yy)) = k(2 x²) + x(y, v/) - k((x, y¹) - 2'),
Independence Test with HSIC. Let Uzy be Borel probability measure defined on the space X' x V and draw an i.i.d sample (X, Y) ((),,))), (I2, (22,)))).... y2)...., (Im, }m)} ~ Usy. Denote by Uz and Uy the marginal distributions of æ and y respectively. We aim to test the null hypothesis Hu Uzy U, x Uy (independence) against the alternative hypothesis Uzy o U2 x Uy (dependence).
Let n m2/2 and W (wi { Be+n Yo FT To measure the discrepancy between Uzy and U,Un, we estimate the HSIC with kernels on X' and l on as²
HSIC (SIC((((WW)
11
Σ
PHSIC (wa, wip)
2
(x.8) hHSI ((01. x2 M1, 82). (z/, 2/y. s/ 322))
MMD ((x1, 22), (a x²2) ( ((11 y2), (3/ 822))
For consistency with MMD, we will refer to the product kernel ((x, y), , (æ', y²)) = (x x') (xxx'')( (y.yy')) as "the kernel" of HSIC, as justified by HSIC's relationship to MMD [3, Thm. 25].
Degeneracy. MMD and HSIC are both used to assess the difference between two distributions (i.e.. P. Q and Uzy: U,x U,) [1-3]. Correspondingly, in the context of two-sample and independence tests, the null hypotheses take the form Hg: the two distributions are identical, while the alternative hypotheses are formulated as H1: the two distributions differ. The UJsstaisstccs underlying MMD and HSIC exhibit a common structural property: they are first-order degenerate under the null hypothesis Ho. and typically non-degenerate under the alternative hypothesis Hl, Formally, U-statistic with function ((-;k) is said to be first-order degenerate if its conditional expectation satisfies
h (w x) E[h(w,, W2; k) wi] 0.
If Varw, (bi (wi x)) o O, the U-statistic is classified as non-degenerate. This degeneracy structure plays a critical role in determining the asymptotic distribution of the test statistics under the null and alternative hypotheses, as well in the design of our proposed test statistic.
Motivation
In this section, we will identify phenomenon consistently observed across prior aggregation methods: while individually strong kernels are necessary, aggregating redundant kernels can degrade performance, and including weak kernels adds little useful information.
Aggregating better kernels may not give higher performance. Intuitively, one might expect that an aggregation of powerful kernels would outperform each kernel on its own. However, the empirical results tell a more nuanced story. For example, in Figure 2a we observe that, for three nigh-performing
'e use the second-order HSIC estimate on [m/2] quadruples [7] for its greater convenience over the complete unbiased HSIC fourth-order U Ussttisstic (which is also computable in quadratic time) [37].

---

1.0
072
1.0
dietOe, Kall
Test Power
0.8
-e- Kzm 000/4
0.8
div(Cey nal) 6203
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
350
Sample Size
Sample Size
(a) Test Power of Different Kernels
(b) Test Power of Aggregated Kernels
Test power versus samples size BLOB dataset. (a) The performance of four different individual kernels with different bandwidths. (b) performance of aggregating first kernel xı with each of the kernels. The diversity" between K2 and K4 is the largest, and that between K) and K2 is the smallest.
Figure 2:
kernels 61, 82 and R3, kernel K2 achieves higher test power than kernel Kg. Now, consider probably the simplest aggregation method: combine the features used by two kernels, corresponding to using the kernel which is their sum. In this case, we have simply that ([((1,82) U U Even though N2 is stronger than K3, the combination {N1, K3) attains greater test power than {K1, K2) (Figure 2b).
The reason for this seemingly counter-intuitive result lies in the diversity of information that different kernels contribute. In the context of UU-statistic-baa -based tests, each kernel к. produces a test statistic U that reflects a particular view of the data. If two kernels they capture, their U-statistics will also be highly correlated. In our example, the test statistics U and U more strongly correlated with each other than U/K and UKa,, Consequently, K and K2 exhibit lower diversity: they redundantly capture similar aspects of the underlying distributional difference. Aggregating redundant kernels (as in {K1,K2}) K2]) offers little new information beyond what K1 already provides. By contrast, K1 and K3 are less correlated, so K3 contributes complementary information that кı alone misses, making the combined statistic U.(M1.ma) more informative.
highly correlated in the information
Quality kernels matters as well. However, diversity alone is not sufficient to guarantee high performance. The kernels must also be effective in detecting the effect of interest. To illustrate, consider kernel K4, which is highly diverse relative to K1. Among the pairs we consider, {K1, x4 has the greatest diversity in the sense of capturing very diffeerent statistical features. Nonetheless, in Figure 2b, the aggregated {kI, Ka} does not perform as well as {K1, k. nor {K1,K2}, The reason is that K4 is 'weak' kernel: its individual test power is too low across sample sizes (Figure 2a), which indicates that cannot provide enough useful information to detect the pattern differences. This highlights that ineffective kernel, no matter how different, can drag down the performance of an otherwise strong aggregation. Thus, high test power from an aggregated test statistic arises when the component kernels both individually powerful and mutually complementary.
4 Aggregating U-Statistic with Diversity
As introduced in the Section 3, is crucial to introduce diversity into the multiple kernel aggregation. Here, we propose the multivariate U-statistic.
Multivariate U-Statistic. Given a constant integer e>> 1, let IC = {Ki K2, me) denote a set of different kernels. To incorporate information from multiple kernels, we construct a multivariate U-statistic by aggregating the individual U-statistics defined in (1) with each kernel in K. For random sample W. the resulting multivariate U-statistic computed with complexity O(en²) as:
(W))
(U (W), U
(W)))
Given the potential redundancy in the information captured by different kernels, we investigate the diversity among multiple kernels and integrate their contributions in a manner that accounts for
"Motivated by [38, 39), the relative diversity value for kernel K; with i E (2,3, w.r.t Ki is computed as [ + [Co (W),&0 UWi (W))l V (W))/Var(UTS Var((U. (W)) Here, the term Cor(s-) denotes the Pearson correlation and the square root term serves a scaling factor when comparing the relative diversity of kernels 63. and K4 with respect to kernel K1.

---

their mutual dependencies. Specifically, the diversity between Ka and Kg for 1 S a. S can be characterized by the (co)variance J.b of U (W) and U% (W). In practice, the true (co)variance unknown and must be estimated from the observed sample W. which poses particular challenge for MMD and HSIC. Under the null hypothesis Ho, these U-statistics are first-order degenerate, where the second-order (co)variance dominates and scales as ( In contrast, under the alternative hypothesis H1, the U-statistics are non non-degenerate, and the first-order (co)variance dominates, scaling as O(n=)))))))))))))))()) Consequently, the naive rescaling of the (co)variance by n yields convergence to 0 under Ho. whereas rescaling by n² results in convergence to too under H1. This discrepancy highlights the difficulty of constructing a (co) (cr)vnrinnce estimator that consistently converges to finite, non-zero limit across both hypotheses, which is essential in enabling meaningful comparison and aggregation of sernel-based test statisties under both null and alternative hypotheses.
Fortunately, we can always apply the second-order (co)variance estimator on samples from the null hypothesis to investigate the diversity among multiple kernels, motivated by [28]. For MMD and HSIC, samples under the null hypothesis can simulated by resampling the observed data W with replacement. Building on this, diversity between Ka and K5 by computing the second-order (co)variance estimator [40-42] between - (Wrr Ho) and n U (WHHH,)) Ho) with computational complexity O(n²) as
assess the utilize the simulated null samples W H. {w |
-2
n² Ho = n²
12
Σ
h(w(, w/yy ka)h(a)) w/::kb)
(3)
Given the covariance matrix Eno with entries n² &H we integrate the contributions of multiple U-statistics (i.e., U S,,.UUWWWW (W)) in manner that accounts for mutual dependencies among kernels as
(w) = (W)) (.)
(4)
which is inspired by the Mahalanobis distance [43], as done in [28]. In this work, assume that the covariance matrix Σn, is strictly positive-defintte Notably, the dimension of the covariance matrix is decided by the number of kernels, i.e., c, with a computational complexity O(e2), independent of the sample size n; thus computational complexity remains low as n increases. The asymptotic properties of the multivariate U-statistic (2), the aggregated statistic (4), and the estimated covariance matrix Σ Ho provided in Appendix A.1.
Remark 1. In our multivariate U-statistic (4), each dimension of nU (W) is normalized to common scale using matrix of covariances of the different kernels. This normalization is crucial to ensure that kernels with varying scales magnitudes do not disproportionately influence the statistic, thereby mitigating potential biases of aggregation.
5 Two-sample and Independence Testing with Learned Diverse Kernels
In this section, we introduce the implementation pipeline of DUAL, which follows the data-splitring approach for kernel selection [6, 44 46]. Even though data-splitting will reduce the sample size in the testing procedure, learning diverse and powerful kernels can gain extra power and adapt to various datasets. We partition the dataset into training set, W +r (Wrr;))= and a testing set, Wie (w..,) For notational convenience, assume both sets contain elements.
5.1 Learning Multiple Diverse Kernels
The selection of kernels that maximize the aggregated statistic T (W) effectively minimizes the (co)variances of the U-statistics under the null hypothesis while maximizing the U-statistics under the alternative hypothesis. This enhances the power of hypothesis testing. as discussed in Section 1. Thus, given training samples Wer = Jm the kernel set I. is learned as
(5) where TXK(W (Wir) is defined as in (4) but computed based on the training samples W... We apply gradient method [48, 49], initialized with set of distinct kernels, to maximize the aggregated
{K1, R2, ..., Ke} E arg max(T;; (Wor)).
'The original method in [28] focuses specifically on MMD with a tailored (co)variance estimator, while we generalize the approach to a broader class of U-statisties by employing a unified (co)variance estimator. "One can aggregate over multiple splits, in [47], but this may not actually help compared to single split.
5

---

statistic over pre- pre-specified kernels, following previous approaches [50, 51]. The learned kernels are independent of testing samples, ensuring that their use testing does not violate the type-I error constraint.
Remark 2. When c. 1, the statistic (4) becomes the square of the signal-to-noist ratio of that single kernel. This was the objective proposed in previous work to select a single MMD 15, 6] or HSIC [15, 16] kernel; thus (5) reduces to those methods when e: =
5.2 Testing with both Diverse and Powerful Kernels
After learning kernels to optimize the diversity within our multivariate U-statistics, we would further like to select only the effective kernels. We focus on kernels that effectively capture evidence against the null hypothesis (under which test statistics have mean zero), while placing less emphasis on kernels that demonstrate limited performance in subsequent testing.
Extracting prior knowledge, In order to identify and select effective kernels within our optimized aggregation, we extract essential sign information from the training compu-o as the signum vector of the decomposed statistic T (Wer), inspired by [46]. Specifically, we decompose ΣH, LH Ho using the Schur method [52] with computational complexity O(c)), yiclding:
Tht (w,)) = (U (
= n² (.
(6)
where || ||2 denotes the & norm. L helps reduce correlations among different kernels [46]. Then, select strong kernels that capture complementary information and contribute the final aggregated statistic, we first investigate the information from training samples by computing the signum vector as
Frr sgn U. E {-1,++1],
where sgn(-) is signum function as sgn(a) = (sgn(a1). sgn((2), ... agn) for a vector a = (a), .2,-.. aaa"" TT and sgn(a;) = a:/la,| for a, O; otherwise, sgn(a,) =1
Selection inference based on extracted knowledge. Based on the learned K,, we perform the test using the testing samples Wt.- = (=1. The corresponding aggregated statistic is defined as
THe (Wi.)) n² (U (W UN n² U
which is defined analogously to Eqn. (4). but computed based on the testing samples Wee.
In similar manner, we infer the signum vector of testing samples, i.c., Fie sgn (ī ( (Wie)) (K(te)) Given the two signum vectors For and Fre, we can calculate an indicator vector to assess the alignment between the training and testing signum vectors as follows
F {EAe E +1, with P. = I[Fe,
(8)
where Fte,: and F are i-th elements of Fie and Ftr. respectively. The alignment vector F selects the components of the aggregated statistic that share the same signum value across the training and testing samples. Given the selection, we focus the components that more likely capture deviations from the null hypothesis, and define the test statistic with selection inference as
== F
- Ho
(W1e)
(9)
where O is the element-wise product.
"Prior work on choosing single MMD or HSIC kernels [5, 6, 15, 16) has emphasized the importance of maximizing the power of a test, rather than maximizing the statistic. Our statistic, however, is already studentized, which makes directly estimating the test power less practical (Theorem 5) but also removes the incentive to e.g. simply scale C' for some large C; our statistic is invariant such changes. Also see Remark 2.
'The Schur decomposition works for positive-definite matrices, and in this paper, we assume the covariance matrix E. is positive-defimite practice, to ensure positive definiteness, we can replace EH,, with En, + AI, where À > 0 is small regularization constant and I denotes the identity matrix.
Notabby, for statistic (6), we compute EH, based on *o which is resampled from Hotr. The entries of Dt, are calculated as E3n.(3) with W. Similarly, for statistic (7), Ez, is computed based on W which is resampled from Wte For notational simplicity, we omit the explicit dependence on W:Ho and #

---

Remark 3. Our approach differs from previous bi-directional hypothesis testing [46], which con- structs rejection regions along the directions of F and F to determine if the test statistic lies within these regions. In contrast to their method, which employs an additional parameter calibrated via separate validation dataset to adjust the significance levels of the rejection regions, our technique avoids such parameter tuning. Additionally, our method diverges from post-selection inference approaches [53, 54], which select significant statistics based solely on predefined kernels without incorporating insights from the training phase. Such post-selection inference methods are constrained to specific kernel classes and utilize less precise "streaming" estimators, potentially leading to lower-powered tests when using fixed kernels [55, 563.
Wild Bootstrap for Testing Threshold. To obtain the testing threshold To for significance level a, employ the wild bootstrap for T with Rademacher random variables [57, 58, 7, 59]. Specifically, let B be the iteration number of bootstraps. In the b-th iteration (b E [B). we generate i.i.d. Rademacher variables (f1....,6,) Gn), that is, Pr(c- 1) Pr(c, -1) 1/2 for i [n]. and then compute
UK. (Wne)
Σ
Gig h(ws, WigiK),
2
15h <i2En
where hww,, W2; K.) = (h(w), W2; K1 ) l(w,, W0;; ko))²
Correspondingly, we calculate the b-th alignment vector Fb, analogous to Eqn. (8), follows F = [ e +0. with E. = I[Ft. Firal and F = sgn (E K (Wre))
and we perform selection inference with Fb to derive the b-th wild bootstrap statistic as
T n² Fb
(W.e))
Taking the original test statistic in Eqn. (9) as TB+1, we estimate the testing threshold (i.e., (1 a+)- quantile of the null distribution of the test statistic with selection inference) as follows
B+1 Σ
fa = inf TER: - a B+1
T
<T]
(10)
b=1
Finally, we propose the test with the testing threshold fa and the test statistic T in Eqn. (9) as
b(X, YxK) = [[ fa):
(11)
where 6(X, Y;K) means the null hypothesis is rejected; otherwise, it is accepted.
The computational complexity of the above testing procedure is O O(Bn22++ +n²c² +nc))+ The subsequent theorem characterizes the behavior of the test statistic under both the null and alternative hypotheses.
Theorem 1. Let IC be collection of bounded characteristic kernels. Under the null hypothesis Ho. the test in (11) has type-l error bounded by a, i.e., PIH, (h Y;k) = 1) S a, even non- asymptotically, Meanwhile, under any fixed alternative hypothesis H,, and assuming Assumption 1 (in Appendix B. holds, the test has power converging to 1, i.e., lim Рги, (b(X, Y; = 1)
In Theorem 1, validate the test by proving that the type-I error is controlled level in non-asymptotic sense under Ho, and the test power approaches one asymptotically under fixed Hi The asymptotic properties of the test statistic with selection inference are provided in Appendix A.2.
Experiments
6.1 Datasets & Baselines
We evaluate our proposed methods on benchmarks for two-sample and independence testing. For two-sample testing, we use three datasets: frequently used synthetic BLOB dataset [45, 51, 6]. MNIST (versus generative adversarial model DCGAN [60]) dataset [6-8], and the ImageNet (versus Im [61]) dataset [6, 46). For independence testing, we consider the Higgs dataset (a high-dimensional physics dataset) [62], MNIST, and CIFAR-10. 10. These benchmarks encompass diverse range of data modalities and difficulties, providing a rigorous evaluation of our tests.
7

---

1.2
1.0
0.5
0.-
302
150
200
250
302
50
60
10
450
550
600
702
852
920
Sample size N
Sample size N (b)
Sample size N [c]
(a)
AutoTST
HYD
MMD FUSE
1.0
Power
0.0
0.0
0.01
100
500
1000 1200
150
200
1000 3.00
200
303 Sample
555
Sample
Sample
10)
HSKAggo
HSCO
HSCCUAL
Figure 3: Two-sample (a-c) experiments on datasct BLOB, MNIST and ImageNet; and independence (e f) experiments dataset Higgs, MNIST and CIFARIO. The power results averaged over 1,000 repetitions and the type-I error are all controlled under the significant level a 0.05, where the type-I error experiments can be found in the supplementary material.
We compare MMD-DUAL (for two-sample tests) and HSIC-DUAL (for independence tests) against both standard baselines and recent state-of- fiite-of-thh-aar the-art methods. In the two-sample case, baseline methods include the AutoMI. two-sample test (AutoTST) [63], the MMD Aggregated test (MMD-Agg) [8], the MMD two-sample test with deep kernel (MMD-D) [6], and the recently proposed multi-kernel test MMD-FUSE [9]. For independence testing, baselines include the Finite Set Independence Criterion (FSIC) [13], an aggregated HSIC test (HSIC-Agg) [7], and independence testing with optimized bandwidth (HSIC-O). All methods are calibrated to control the Type-I error at a 0.05 for a fair comparison. As shown in Figure 3, the proposed MMD-DUAL and HSIC-DUAL consistently outperform all baselines across the six benchmarks.
every case, our adaptive methods achieve the highest test power (rejection rate under the alternative), demonstrating clear advantage over both classical and contemporary methods on both two-sample (BLOB, MNIST, ImageNet) and independence (Higgs, MNIST, CIFAR-10) tasks. The detailed description of datasets, baselines, experimental settings can be found in the Appendix C.
6.2 Ablation Study
Effectiveness of Diverse Kernels and Selection Inference. To quantify the effect of learning diverse kernels and selection inference in our proposed DUAL, we conduct series of well-designed ablation study all the benchmarks. We only display the results of BLOB for MMD-DUAL and Higgs for HSIC-DUAL, and the results on other datasets can be found in Appendix C. Figure (a) and (c) report the test power (for level 0.05) for the full DUAL methods and three ablated variants of each: (i) AU+D: without selection variant without the selection inference in the testing procedure (i.e., all candidate are always aggregated, but still optimizing the diversity between kernels in the training procedure), (ii) AU+S: without introducing diversity- variant without the diverse kernel pool (using an fixed identity covariance matrix) while still performing the selection inference technique, and (iii) AU: plain aggregation baseline that uses multiple kernels in an aggregated two-sample and independence testing with neither diversity nor selection inference enhancements. We observe that both MMD-DUAL and HSIC-DUAL (full methods) consistently achieve the highest power, outperforming all ablated variants at every sample size. Removing either component leads to drop in power. Importantly, both of these variants still outperform the plain aggregation baseline. This indicates that each component component-diversiit in the kernel choices and selection inference during desting-indeeenndenly contributes to improving test power. Their combination in DUAL has cumulative effect, yielding the best performance overall.
Kernel Selection Probability by Selection Inference. We next examine how the selection inference behaves under the null and alternative through the kernel selection probabilities. Under the null hypothesis (see Figure 4 (b) and (f)), each kernel is selected with roughly 50% probability, which means the selection is approximately uniform across the pool of diverse kernels and guarantee that selection inference has no influence on the control of Type-l error. This holds consistently across sample sizes, indicating that in the absence of a signal the procedure effectively randomizes over the kernel choices without bias. In contrast, under the alternative hypothesis (see Figure (c) and
8

---

10) Power of Cifferent Methods
10
(b) Kamei Selection Procabilly (Nul)
(0) Kernil Selection Probubility (Atummatve]
1.0
250
(6) Oeservised T-Statsics
scteed
150
0.
Probablyy
122
RUME-D-DUA.
0.2
AUHO
02
0.2
NLMB
Semple
Sample Size
Semple Bize
Sample Bize
[0] Power
Methods
Kemel Protabiity (Nul)
[g] Kemel Selection Probabiity (Abermatie)
(h) Oeseriied T-Stametics
1.3
10
1.0
Thoofdd
06
Ote, I. mo.sdot
Probsbiity
T-Stelalics
152
Cee T. select
04
24
04
100
ALKS-D-DUA
KLHO
AUHB
00
AU
00
0.0
Sample S
Sample Sizw
Semple Bizw
Sample Bize
Figure 4: Ablation Study on the effectiveness of diversity and selection inference. (a-d) are ablation study for MMD-DUAL; (h) are ablation study for HSIC-DUAL (a, e) Test power for model variants: AU represents simple Aggregated UtSttatitiies: represents selection inference technique: D represents considering diversity into AU; AU+S+D refers to our proposed DUAL.
(g)), the selection probabilities become increasingly concentrate on the most informative kernels as the sample size grows. In other words, the selection inference technique correctly detects which kernel is capturing the existing dependency or distribution difference, and it selects that kernel with ever-growing frequency. For instance, on the Higgs dataset (Figure 4 (g)). the kernels (Kernel 2 and Kernel 6) that best capture the dependence structure are chosen far more often than others, with its selection probability rising well above 0.5 and eventually approaching nearly 1.0 as the sample size increases. This trend demonstrates that the procedure progressively focuses on the kernels that yield the strongest test statistic, effectively leveraging the most useful features of the data.
Testing Threshold and Observed Test Statistics. We now analyze the impact of selection inference on the test power, which mainly depends on the testing threshold and the observed test statistic (see Figure 4 (d) and (b)). From the two dotted lines, we observe that the testing threshold (derived by 1-a quantiles of the aggregated statistic under Ho or wild bootstrap) of applying selection inference is approximately 50% that of the aggregated statistic on the original full kernel set, because any given kernel included in the test statistic only about 50% of the time on average in selection inference procedure. Furthermore, from the two solid lines, we can find that the observed test statistic produced by DUAL is initially approximately half that of the no-selection counterpart for small sample sizes. This is expected: at lower sample size, the selection inference might not able to determine which kernels are signal-carrying, so its test statistic starts off lower. However, as the sample size increases, the selected kernel is almost always the most informative one, and thus the adaptive test statistic grows and eventually matches the magnitude of the no-selection test statistic. Crucially, throughout this process, the DUAL enjoys the advantage of a lower threshold, and as sample size increases, its statistic catches up to the no-selection statistic. This combination growing test statistic that converges to the no-selection level, together with a consistently reduced critical meeeshold-mamnn that DUAL achieves higher power at all sample sizes.
In summary, the ablation results show that both diversity and selection inference contribute to performance gains, and the selection inference mechanism not only identifies informative kernels under Hi but also effectively control the Type-I error under Ho. leading to a substantial improvement in test power for the full DUAL method. A more detailed analytical explanation and analysis to support the results in Figure 4 is provided in Example 1 (Appendix D).
6.3 Computational and Scalability Analysis
Time Complexity Analysis. Table summarizes the time complexity of MMD-DUAL 10 in both the training and testing phases, and compares it with previous methods, MMDAgg and MMD-FUSE.
"HSIC-DUAL exhibits the same time complexity, as both follow the formulation of the second-order U-statistic described in Section 2.
9

---

Table 1: Time complexity of MMD-DUAL, MMDAgg and MMD-FUSE in training phase (using Adam optimizer [64]) and testing phase (using wild bootstrap)
Time Complexity
MMD-DUAL
MMDAgg
MMD-FUSE
Training
+ M) O(Ben² + n²c² + +c))"l
N/A
N/A O(neeB)
Testing
O(nic(B + BY))
MMDAgg and MMD-FUSE select the kernel parameters using heuristic methods without a training procedure. For all methods, the complexity dominated by the quadratic term in the sample size, i.e., O(n²), which corresponds to the computational cost of computing the second-order U-statistic. Other factors, including the number of kernels c, the number of optimization parameters T. the number of optimization epochs M, the number of wild bootstrap iterations B, and the number of permutation tests B', are treated as constants that do not scale with n1
Scalability, Regarding the scalability of DUAL method, the time complexity are quadratic or cubic related the size of kernel pool. In that way, using a very large candidate kernel pool may lead to computational inefficiency. In practical implementations of two-sample and independence testing, the kernel set initialized using heuristic methods, following the methodology of [7-9], As shown in these studies, increasing the number of kernels beyond a moderate size does not yield noticeable improvements in test power. Specifically, as shown in Figure 6 in Section 5.7 7 of [8], increasing the number of kernels from 10 to 100 and even to 1, 000 does not improve power. This supports the choice of using small number of kernels (e.g..c = 10), as there is nothing to gain empirically from using a finer discretization for the bandwidths of kernels. In fact, the referenced study even considers aggregating 12, 000 kernels.
For high- ligh-dimensional data, kernel methods scale linearly with the input dimension, since the compu- tation of each pairwise kernel evaluation (c.g.. in Gaussian kernel) involves an inner product. This cost arises only during the construction of the kernel matrix (typically through pairwise distance computations), which can be efficiently implemented and is rarely the bottleneck practice.
7 Conclusions
In this paper, we identify that kernel selection markedly influences the performance of aggregated- kemnel two-sample and independence tests. To address the challenge of optimizing kemel aggregation for nonparametric hypothesis testing, we introduce Diverse U-statistic Aggregation with Learned Kernels (DUAL). Through analysis, we demonstrate the importance of balancing kernel diversity with individual effectiveness to enhance statistical power. Our method integrates kernel selection via covariance-informeed diversity measures, thereby mitigating the adverse effects of redundant or weak kernels. Experiments on a variety of benchmarks show that DUAL-based tests (MMD-DUAL and HSIC-DUAL) consistently outperform cate-of-the-an approaches, confirming their effectiveness. aggregation approach to advanced
Future work will extend this diversity-drives optimization techniques and will explore its effectiveness on broader classes of U-statistics across diverse tasks, e.g., goodness-of-fi testing. Our method of selection inference for the kernel collection could be generalised to also be applicable to adaptive multiple testing.
Acknowledgments and Disclosure of Funding
This rescarch was supported by The University of Melbourne's 's Research Computing Services and the Petascale Campus Initiative. ZJZ, XYT and CL. are supported by the Melbourne Research Scholarship. ZJZ and XYT are also supported by the ARC with grant number DE240101089 LHP is supported by the ARC with grant number LP240100101. AS is supported by a UKRI Turing AI World-Leading Researcher Fellowship with grant number G11 1021. DJS is supported in part by the Canada CIFAR AI Chairs program. FL is supported by the ARC with grant number DE24010 01 LP240100101, DP230101540 and the NSF&CSIRO Responsible AI program with grant number 2303037.
"This differs from the time complexity reported in the rebuttal, which was Onn2 + B). The actual complexity can be lower, the covariance matrix Ei,, needs to be computed only once in the testing procedure.
10

---

References
[1] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and J. Smola. A kernel method for the two-sample-problem, In NeurlPS, 2006.
[2] A. Gretton, K. Fukumizu, C. Teo, I.. Song, B. Schölkopf, and A. Smola. A kernel statistical test of independence. In NeurlPS, 2007.
[3] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. kernel two-sample test. Journal of Machine Learning Research, 13(1:7223-77,, 2012.
[4] D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In ICLR, 2017.
[5] D. J. Sutherland, H.-Y. Tung, H. Strathmann, De, A. Ramdas, A.J. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In ICLR, 2017.
[6] F. Liu, W.KK Xu, J. Lu, G.-Q. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non con-paraettric two-sample tests. In ICML, 2020.
[7] A. Schrab, Kim, B. Guedj, and A. Gretton. Efficient aggregated kernel tests using incomplete U-statistics. In NeurlPS, 2022.
[8] A. Schrab, I. Kim, M. Albert, B. Laurent, B. Guedj, and Arthur Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research, 24:194:1-194:8, 2023.
[9] Biggs, A. Schrab, and A. Gretton. MMD-Fuse: Learning and combining kernels for two- sample testing without data splitting. In NeurlPS, 2023.
[10] S. L. Chau, A. Schrab, A. Gretton, D. Sejdinovic, and K. Muandet. Credal two-sample tests of epistemic ignorance. In AISTATS, 2025.
[11] A. Schrab and I. Kim. Robust kernel hypothesis testing under data corruption. In AISTATS. 2025.
[12] X. Tian, I.. Peng, 7. Zhou, M. Gong, A. Gretton, and F. Liu. A unified data representation learning for non-parame non-parametric two-sample testing. UAI, 2025.
[13] W. Jitkrittum, Z. Szabó, and A. Gretton. An adaptive test of independence with analytic kernel embeddings. In ICML, 2017.
[14] S. Shekhar, I. Kim, and A. Ramdas. A permutation free kernel independence test. Journal of Machine Learning Research, 24(369):1 1-68, 2023.
[15] Y. Ren, Y. Xia, H. Zhang, J. Guan, and S. Zhou. Learning adaptive kernels for statistical independence tests. In AISTATS, 2024.
[16] N. Xu, F. Liu, and D. J. Sutherland. Learning deep kernels for non-parametric independence testing, 2024.
[17] M. Gong, K. Zhang, T. Liu, D. Tao, C. Glymour, and B. Schölkopf. Domain adaptation with conditional transferable components. In JCML, 2016.
[18] H. Chi, F. Liu, W. Yang, L. Lan, T. Liu, B. Han, W. K. Cheung, and J. T. Kwok. TOHAN: A one-step approach towards few-shot hypothesis adaptation. In NeurIPS, 2021.
[19] M. Binkowski, J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD gans, In ICLR, 2018.
[20] R. Gao, F. Liu, J. Zhang, B. Han, T. Liu, G. Niu, and M. Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In ICML, 2021.
[21] Zhang, Y. Song, Yang, Y. Li, B. Han, and M. Tan. Detecting machine-generated texts by multi-population aware optimization for maximum mean discrepancy. In ICLR, 2024.
11

---

[22] Y. Song, Z. Yuan, S. Zhang, Z. Fang. J. Yu, and F. Liu. Deep kernel relative test for machine- generated text detection. In ICLR, 2025.
[23] J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15(1):2009-2053, 2014.
[24] Y. Li, R. Pogodin, D. Sutherland, and A. Gretton. Self-supervised learning with kernel dependence maximization. In NeurlPS, 2021.
[25] Z. Wang, Z. Zhan, Gong, Y. Shao, S. Ioannidis, Y. Wang, and G. Dy. Dualhsic: Hsic- bottleneck and alignment for continual learning. In ICML, 2023.
[26] A. Schrab. Optimal Kernel Hypothesis Testing, PhD thesis, UCL (University College London), 2025.
[27] I. Kim, S. Balakrishnan, and L. Wasserman. Minimax optimality of permutation tests. Annals of Statistics, 50(1):225 251, 2022.
[28] A. Chatterjee and B. B. Bhattacharya. Boosting the power of kernel two-sample tests. Biometrika, aassae08,, 2025.
[29] M. Albert, B. Laurent, A. Marrel, and A. Meynaoui. Adaptive test of independence based on HSIC measures. Annals of Statistics, 50(2):858-87 2022.
[30] C. Scheidegger, J. Hörrmann, and P. Bühhmann. The weighted generalised covariance measure. Journal of Machine Learning Research, 23(273):1-68, 2022.
[31] A. Schrab. Unified View of Optimal Kernel Hypothesis Testing. a7X1v:2503.07084, 2025.
arXiv preprint
[32] A. Schrab, B. Guedj, and Gretton. KSD Aggregated Goodness-of-fit Test. In NeurIPS, 2022.
[33] L. Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.
[34] L. Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
[35] G. Dietterich. Ensemble methods machine learning. In International Workshop on Multiple Classifier Systems, 2000.
[36] A. Schrab. A Practical Introduction Kernel Discrepancies: MMD, HSIC & KSD. arXiv preprint arXiv:2503.048200 2025.
[37] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, (1477)1333-14344 2012.
[38] Y. Wu, L. Liu, Z. Xie, K. H. Chow, and W. Wei. Boosting ensemble accuracy by revisiting ensemble diversity metrics. In CVPR, 2021.
[39] D. Wood, T. Mu, A. M. Webb, H. W. J. Reeve, M. Lujan, and G. Brown. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24:359:1-359:49 2023.
[40] R. Serfling. Approximation Theorems of Mathematical Statistics. John Wiley & Sons, 2009.
[41] W. Hoeffding. A class of statistics with asymptotically normal distribution. Breakthroughs in statisties:
Foundations and basic theory, pages 308-334, 1992.
[42] D. J. Sutherland and N. Deka. Unbiased estimators for the variance of MMD estimators. arXiv, 2022.
[43] R. De Maesschalck, D. Jouan-Rimbaud, and D. .L. Massart. The mahalanobis distance. Chemo- metrics and Intelligent Laboratory Systems, 50(1):1-18, 2000.
[44] X. Tian and F. Liu. Take close look the optimization of deep kernels for non-parametric two-sample tests. In Australian Database Conference, 2023.
12

---

[45] A. Gretton, B. K. Sriperumbudur, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Balakrish- nan, and K. Fukumizu. Optimal kernel choice for large-scale two-sample tests. In NeurIPS, 2012.
[46] Z.-J. Zhou, J. Ni, J.-H. Yao, and W. Gao. On the exploration of local significant differences for two-sample test. In NeurlPS, 2023.
[47] C. J. DiCiccio, J. DiCiccio, J. P. Romano. Exact tests via multiple data splitting. Statistics & Probability Letters, 166: 166:108865, 2020. ISSN 0167-7152.
[48] S. Boyd, S. P. Boyd, and I.. Vandenberghe. Convex Optimizatioo. Cambridge University Press, 2004.
[49] P. Jiang, F. Liu, J. Wang, and Y. Song. Cuckoo search-designated fractal interpolation functions with winner combination for estimating missing values in time series. Applied Mathematical Modelling, 40123-24))9692--711 18, 2016.
[50] K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. NeurIPS, 2015.
[51] W. Jitkrittum, Z. Szabó, K. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In NeurlPS, 2016.
[52] E. Deadman, N. Higham, and R. Ralha. Blocked schur algorithms for computing the matrix square root. In International Workshop on Applied Parallel Computing, 2012.
[53] J. D. Lee, D. L. Sun, Y. Sun, and J. E. Taylor. Exact post-selection inference, with application the lasso. Annals of Statistics, 44(3):907, 2016.
[54] Kübler, W. Jitkrittum, B. Schölkopf, and K. Muandet. Learning kernel tests without data splitting. In NeurIPS, 2020.
[55] A. Ramdas, S. J. Reddi, B. Poczos, A. Singh, and L. Wasserman. Adaptivity and computation- statistics tradeoff' for kernel and distance based high dimensional two sample testing. arXiv, 2015.
[56] F. Liu, W. Xu, J. Lu, and D. J. Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. In NeurlPS, 2021.
[57] X. Shao. The dependent wild bootstrap. Journal of the American Statistical Association, 105 (489):218-23) 2010.
[58] K. Chwialkowski, D. Sejdinovic, and A. Gretton. A wild bootstrap for degenerate kernel tests. In NeurIPS, 2014.
[59] R. Pogodin, A. Schrab, Y. Li, D. J. Sutherland, and A. Gretton. Practical kernel tests of conditional independence. arXiv preprint arXiv:2402.13196 2024.
[60] A. Radford, L. Metz, and Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. In ICLR, 2016.
[61] B. Recht, R. Roclofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[62] P. Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5(1), 2014.
[63] J.-M. Kübler, V. Stimper, S. Buchholz, K. Muandet, and B. Schölkopf. AutoML Two-Sample Test, In NeurIPS, 2022.
[64] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4t1. 42.6000, 2014.
[65] A. Papoulis and U. Pillai. Probability, Random Variables and Stochastic Processes. McGraw Hill, 2001.
13

---

[66] J. Hemerik and J. Goeman. Exact testing with random permutations. Test, 27(4):811-82) 2018.
[67] J. P. Romano and M. Wolf. Exact and approximate stepdown methods for multiple hypothesis testing, Journal of the American Statistical Association, 100(469):94-108, 2005.
[68] H. B. Mann and A. Wald. On stochastic limit and order relationships. The Annals of Mathemat- ical Statistics, 14(3)2117226, 1943.
[69] A. Gretton, R. Herbrich, A. Smola, O. Bousquet, B. Schölkopf, and A. Hyvärinen. Kernel methods for measuring independence. Journal of Machine Learning Research, 6(12), 2005.
[70] I. Kim and A. Schrab. Differentially private permutation tests: Applications to kernel methods. arXiv, 2023.
[71] V. S. Korolyuk and Y. V. Borovskich. Theory of U-statistics. Springer Science Business Media, 2013,
[72] S. Janson. Gaussian Hilbert Spaces. Cambridge university press, 1997.
[73] R. Durrett. Probability: theory and examples. Cambridge university press, 2019.
[74] B. Conway, A course in functional analysis. Springer Science & Business Media, 1994.
[75] M. Renardy and R. C. Rogers. introduction to partial differential equations. Springer Science & Business Media, 2006.
[76] LCC Evans. Measure theory and fine properties of functions. Studies in Advanced Mathematics, 1992.
[77] W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association, 58(301):13-30, 1963.
[78] A. J. Lee. U-statistics: Theory and Practice. Routledge, 2019.
[79] Laumann, J. Kügelgen, and M. Barahona. Kernel two-sample and independence tests for nonstationary random processes. Engineering Proceedings, 5(1):31, 2021.
[80] C.-L. Li, W.-C Chang, Y. Cheng, Y.-M. Yang, and B. Póczos. MMD GAN: Towards deeper understanding of moment matching network, In Advances Neural Information Processing Systems 30, pages 2203-2213 Curran Associates, Dutchess, NY, 2017.
[81] X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han. Detecting out-of-distribution data through nn-distribution class prior. In ICML, 2023.
[82] S. Zhang, F. Liu, J. Yang, Y. Yang, C. Li, B. Han, and M. Tan. Detecting adversarial data by probing multiple perturbations using expected perturbation score. In ICML, 2023.
[83] X. Guo, F. Lin, Yi, J. Song, Sun, Lin, Z. Zhong, Z. Wu, Wang. Y. Zhang, and J. Song. Deep transfer learning enables lesion tracing of circulating tumor cells. Nature Communications, 13(1):7687, 2022.
[84] Z. Zhong, J. Hou, Z. Yao, L. Dong, F. Liu, Yue, T. Wu, J. Zheng, G. Ouyang, C. Yang, and J. Song. Domain generalization enables general cancer cell annotation in single-cell and spatial transcriptomics. Nature Communications, 15(1):1929, 2024.
[85] H. Chi, H. Li, W. Yang, F. Liu, I.. Lan, X. Ren, T. Liu, and B. Han. Unveiling causal reasoning in large language models: Reality or mirage? In NeurIPS, 2024.
[86] R. Mehta, S. Pal, V. Singh, and S. N. Ravi. Deep unlearning via randomized conditionally independent hessians. In CVPR, 2022.
[87] B. Han, J. Yao, T. Liu, B. Li, S. Koyejo, and F. Liu. Trustworthy machine learning: From data to models. Foundations and Trends@ Privacy and Security, 7(23):744226, 2025.
14

---

Appendix
A Asymptotic Theory for the Proposed Statistics
A.1 Asymptotic Behavior the Aggregated Statistic
In this section, we investigate the asymptotic behaviors of various statistics defined in Section 4. To maintain notational consistency, we compute the statistics over the sample W {w;); (which is independent of the kernel set K.), done in Section 4. However, these results remain valid for the testing samples Wte discussed in next Section A.2, since Wte is also independent of the kernel set A.
We first present the asymptotic behavior of the multivariate U-statistics with multiple kernels, i.e., n U (W), ) as follows.
Theorem 2. Let K = (Ka, K2, Fie be a set of e kernels such that Hi with E [c] is charateristic and bounded. Then, under null hypothesis Ho. for first-order degenerate U-statistic, we have
n (W) $ GK (12(h(; 12(h(; (2)((((())))))--)))))))) I2(h(;3
where I2(-) is the multiple Wiener-Itō integral (Definition 10, Appendix B.2). Furthermore, the characteristic function of Gk evaluated at = (Mi...., Me) E IR is defined as
Φ(n) = E
c "G Gx
exp -(p)
21A0
where {Nole= are eigenvalues of the Hilbert-Sooe Hilbert-Schmidt operator Hk. L2WW, W) - L2(W, W) as
= /-
000
h (wi, Koy) f( (ww))d((ww))
where = 1, (..) and h (wi, w2; KG)) 23= nihh (wi, W2; Kg).
For the covariance matrix of (W) under Ho, i.e., Eno, present its asymptotic behavior as follows.
Lemma 3. For a bounded function M(;R)) with kernel R, the estimator E.o, defined in Eqn. (3) satisfies Σno E, EHo-
Building Theorem 2 and Lemma 3, the asymptotic behavior of T (W) is established in the following corollary using Slutsky's 's theorem [65].
Corollary 4. Under the null hypothesis Hy, the statistic satisfies (W) 4 GR Gk-
Subsequently, we establish that under the alternative hypothesis H1, our ensemble statistics asymp- totically converge in distribution to a normal law.
Theorem 5. Under the alternative hypothesis H1, for non non- non-degene function h(33) with k E K. and assuming E (wh( W2 x)] oo for each K. K., the following holds
(T(U - UK (W)) 4 N(00, ΣH,);
where U (W) E (UX
[ (W) and En, is the covariance matrix of /nU entries consist of (colyariances dH with 1 & a, S c defined as OHi.hh (E [h (wo ka) ka)hi (wi; [b)] - U (W (W))
(OUK((V (W) under H,, whose
where UK (WW) E [UN (W)). Furthermore, the asymptotic distribution T (W) is given by
n 3/2 T.K (W) (W)) T " ( (
where on, = =4 =4( Mn
15

---

A.2 Asymptotic Behavior of the Test Statistic with Selection Inference
In this section, we analyze the asymptotic behavior of the test statistics under selection inference, as introduced in Section 5.2. We regard that the kernel set K. and the signum vector Fir are fixed, as they are independent of the testing sample Wie on which the statisties are computed. Accordingly, the sample size refers exclusively to the size of the testing sample. Throughout this section, we compute H. and Ho based on Wte. i.e. Hgo Eno (Wie) and LHo L (Wie). For notational convenience, we omit the explicit dependence on Wte-
We first write the test statistic T = max(n diag (ir)(FF)) UK (W according to the definition in Eqn. (9). By Lemma 3, we have that LHo 3n LHo with EHo LH based on the same Schur decomposition [52]. In the following theorem, we present the asymptotic behavior of our test under the null hypothesis Hg-
Theorem 6. Under null hypothesis Ho, both the test statistic 'T and the wild bootstrap statistic To (b E [or MMD and HSIC converge in distribution to max (diag Fr,) Gk. 0) l|.. where Gx (2(h((;m)) T2(h((3 ...: ((M(())))))
Having established the validity of our test under the null hypothesis, we now investigate the asymptotic behavior under the alternative hypothesis H1, where the vector diag Ftr L Wo (W(ee converges in distribution to normal law, as a consequence of Theorem 5 and LH, 2g LH, by Lemma 3. Corollary 7. Under alternative hypothesis H1, the following asymptotic distribution holds
Vn- diag(Fi L U (W(e)) $
N (Vū (trag(FFr) (WW), diag(FFr ) Et. Lioo diag( F(r))
B Detailed Proofs for Our Theoretical Results
B.1 The Detailed Proofs of Theorem 1
We begin with an assumption as follows.
Assumption Under alternative hypothesis H1, given the signum vector Fir, we assume that there exists at least one index E {1, ...,,...c c} such that
For, sgu (L (W))
where a; indicates the i-th coordinate of vector a U (W) = E (U& (Wie)). and ΣH, L Ho the same Schur decomposition [52] applied E. LH. LHo with Eno Es EHo by Lemma
This assumption requires that the estimated signum vector, defined as
Fer = sgn (L (D [U- (W E {-1,+1]
matches the ground-truth vector sgn (L U (W)) in at least one coordinate.
Notably, throughout the statement and proof of Theorem 1, we treat Ftr fixed, since it is derived from the training samples and is independent of the testing samples analyzed here. We present the detailed proofs of Theorem as follows.
Proof. We first prove that, under the null hypothesis Hy, the test Eqn.(11) has type-I error bounded by a, ie., РLи, (b(X, K; 1) a, which holds non -s-amptooiically.
Given the signum vector F,.. we write the test statistic with selection inference follows
T = || max(n diag( F.r ) Ho UK (Wiz).0)1
(12)
according the definition in Eqn. (9). In similar manner, write the b-th wild bootstrap statistic as
T max (n diag (Fir U. (W)))
16

---

Building on the results of [7, Appendix F.1]. the statistics U. (Wie). U..,((Wee)) ie..................... (Wi)(Wie)) along with the original statistic (W.,), are exchangeable under the null hypothesis for both the two-sample and independence testing problems. By combining the exchangeability and [66. Theorem 1]. follows that the wild bootstrap statistics TI.T2... TT. TB and the original test statistic T are likewise exchangeable under the null hypothesis. Consequently, by applying the exchangeability- based argument of [67, Lemma II. the test defined in Eqn. (11) achieves non-asymptotic control of the type-I error at level under the null hypothesis, i.e.,
Рии, (b(X,, Y; = a
Having established the control of type- error, we now proceed to analyze the asymptotic behavior of under the null hypothesis, preparatory step for proving the consistency of the test power under the alternative hypothesis. Specifically, under the null hypothesis Ho. by invoking the large-deviation bound for U-statistic (Theorem 12), the following joint convergence in probability holds
(W 3 E (U)((T)) (W)]
with
E (W)] =0
(13)
By Lemma 3, we have that EHo 3 Ho Based on the same Schur decomposition [52], we denote
by Σh. = LH HH and EH, = LH. L. The continuous- mappiinng theorem [68] then yields that LH, & LHg. Combining the two convergences in probability of U (W.e) and L for the test statistic in Eqn. (12), it follows that
T 2 |max (diag(F,, F.r) L 0, 0, 012 0
by continuous-mapping theorem. Since the wild bootstrap statistics T with b [B] and the original test statistic T are exchangeable under the null hypothesis, if follows that
T & |mma (diag Fir) L 0,0) 0.
(14)
Next, we prove the consistency of the test power under the alternative hypothesis Hi, Similarly, by invoking the large-deviation bound for U-statistic (Theorem 12), it follows that
U (Wie) 2 U (W)
with Uk (W) E U (W)]
(15)
where each dimension * is strictly positive for MMD [3, Lemma 1] and HSIC [69, Theorem 6] statistics in two-sample and independence testing problems with characteristic kernels. Similarly, combined with the results of Eqns. (12) and (15), and L f L follows that
T 2 max(diag( F., F 0)),
(16)
Based on the Assumption 1, the positive-definitenes of LH, and the strictly positive U (W), have that there exists at least one index i (1,2, ef and a constant C₁ > 0 such that
(diag( For L L UW)), = Ct.
Then, combined with Eqn. (16), follows that there exists a constant C2 2 C₁ > 0 such that
T 2 Cz.
(17)
Given Eqns, (14) and (17), follows that
T
T
D3 C2 0=
n
n
by Slutsky's Theorem [65].
By the definition of convergence in probability, for any e (0, C2),
Рти, (TT=TT
T T-7" C2 -5)
C2 <
+ +1
n
n
17

---

Taking = gives C2 = C2/2> 0,
Рги, ((T-
- / 0) 2 Pтн. (IT - - + 1,
and hence
lim Pro. (T > T n2-00
(18)
Building on the results of [7. Appendix F.1]. the wild bootstrap is equivalent to applying a subgroup of permutations. Then, for the test in (11, I), the result of Eqn. (18) guarantees that the sufficient condition for consistency of [70, Lemma 8] on permutation test is satisfied, implying that
lim Рги, (b(X, 1) =1
This completes the proof.
B.2 The Detailed Proofs of Theorem 2
The proof of Theorem 2 follows the proof of [28, Theorem 3.1], with some modifications to accom- modate our framework. We begin with some useful Definitions below.
Definition 8. 128, Definition 3.1J A Gaussian stochastic measure (W,B(W) W) is collection of random variables (Aw(A) : E B(W) defined on a probability space (S2, J, (1) such that the following holds
Zw(AA ~ (O, (A)), for all B(W).
For any finite collection of disjoint sets AI...., A, E BWW,, the random variables
(Zw(Ai), Zz) (x2) ... Zwr (Ac) are independent and
Zu(U'= Ao) = Σ Zw(A)).
8=1
Let W2) ) be the space of measurable functions f W + and
|1 / W2
[8(m, u2)) 2 (wW))) < 00.
Furthermore, we define & _ L2 2 (W2), W2) as the set of all elementary functions as
f(t,,t2)
Σ SS
a a ((t,, A,, A:a).
where AI....,,, B(W) are pairwise disjoint and 0i is the coefficient of the elementary function, which is zero two indices are equal. The multiple Weiner-Itô integral for the functions in & defined as follows.
Definition 9. [28, Definition 3.2] The m-dimeensoona Weiner-Itô stochastic integral, with respect to the Gaussian stochastic measure (Zw(A), E (WW) [W))..oo for the function f E E2 is defined as
I5() 1 s(w,, w2)d )
Σ
Oi Zw Bw (A;,) x 2 (Aig)
184
By taking limits over the set E2, we can extend the above multiple Weiner-Itô integral for elementary functions to functions in L222 B(W*), W2) as follows.
Definition 10. 128, Multiple Weiner-Itô integral for general La-functioss, Definition 3.3] The dimensional Weiner-Itö stochastic integral for function J E (2 ((W2), W2) is defined as the Le limit of the sequence (22(Je))r2, where (fe)r2 is a sequence such that Se E Es with lime-soa [[ fl| 0. This is denoted by
Iz(f) = S1g f(w,, W2 W2 dw (w, (
18

---

Given the Multiple Weiner- Itō integral for general T-2-functions. We present the detailed proofs of Theorem as follows.
Proof. Recall the definition of U (W) in Eqn. (2), we have that for r (11, 12.,.., Nc)² (e) E IR,
n TT) = Σ nyUn:(W)
-1
12
Σ
h(w,,
Wizi
2
j=1
11
Σ
nyll(wzg Wigi kg)
2
1
h hww,, (0gz
2
Sinciagn
where
h (wi, Wizi k
Σ nyh(uss WigiK})
It is easy to see that the function h (:; (:;Ke,) is a measurable and symmetric function, and h (+; KC,)) (22222 ((W*), ((W"). W2) based on bounded and characteristic kernels. Moreover, the function à (5 (s)KC)) first-order degenerate, this property is inherited from the first-order degeneracy of each compo- nent function h((;;s) 16) with & E K. Then, by [71, Corollary 4.4.2. Section 4.4], we obtain
n 17 (W) $ Σ (z - 1),
(19)
1=1
where the Z, are i.i.d. random variables drawn from (( 1) and the {Ax ) are the eigenvalues of the Silbert-Schhidt operator Hy.., L LWW, (WW) W) L LW ((W), W) defined as
(x,())(w1)) = [
700
h (wi, Wy; K,,)) fww)
(20)
By [72, Theorem 6. Section 6), we have the characteristic function ((1) as follows
exp 6. Σ 1u -1)
E' Texp (1. n.n u
4 E'
(21)
V=1
2
exp
(-1 +(n))
vI 21A,
Here, 17 E IR can be chosen arbitrarily and Lemma 11 establishes the continuity of ((n) at e R By Lévy's Continuity Theorem [73, Theorem 3.3.17], random vector Zi exists with characteristic function ((1) such that
n U (W) 4 Zx.
Next, we establish that Zx can be represented as Gx. achieve this, we leverage the linearity property of multiple stochastic integrals, which allows us write the characteristic function of Gx at
19

---

as follows
('(n)
E
exp
(in T Gk)
E
exp
n 22 (
jml
E
exp
U2 Σ ngh((Kk))
E
[exp (1l2 (k();K,))))]
(b)
exp(-
(x)(
2012
where equality (a) holds for (w., (02i Knn))
Ej=1 my ( Win Wizi ky)) and equality (b) follows by [72, Theorem 6.1, Section 6]. Here, {X₂ represents the eigenvalues of the bilinear form B: (2 B (WW, x (3W, ((W), W) + R defined as
B(J, f2) =
2 F (1
for any f1. E (2 BWW), W). The terms (f1) and (ff2) are Weiner-Itô integrals (Defini- tion 10) defined as follows
I(f) / f(w)dZ((w) for (f.
Now, applying the stochastic integral multiplication formula [72, Section 7, Theorem 7.33], we obtain the following result:
2
1 / h (wi, w2; ((w1))))wa))) + [1wa)))/s(nn)) (W
2 wa
/ W2 h (w1, ww; K2,) fi (w)) (w))
where the last equation holds by the symmetry of the core function ha(:; Kry).
This demonstrates that the bilinear form B(f1, f2) has the same eigenvalues as those in Eqn. (20)
based on h (-; K,,)) Hence, we have that {>> 1 (x1 and that
( = ② d'(n)
This proves that Zx can be represented as Gx and n (W) 4 Gk
This complete the proof.
The following lemma establishes the continuity of the characteristic function.
Lemma 11. Let (n) be as defined in Eqn. (21). Then, (00) = and ((n) is continuous at R.
Proof. When with 0 E R., the cigenvalues A, = 0 for v E 2, 3, oc). Hence, 4 = 1 based on its definition as in Eqn. (21)
(nn)
exp (dv)
E
exp 6.
Σ
2
Xo -1)
21A
v=1
20

---

where 1 are eigenvalues of the (WW, (W),, W) defined
operator Hie, 2 w) 7
HK,, (1 = h (wi, W2; KG,) f(w))dd((w))
where h (w), W2; K = Ej=110 1igh (w0, W2; (k))
Next, we prove that (n) is continuous at 0 E R.. is evident that the eigenvalues { of the Hilbert-Schmen operator Hx., satisfy the summability condition [74]
2 00. 00.
Then, applying Fubini's Theorem to the term Evsl (z) 1), we obtain
Σ du (z -11
E
= E
2Z1 - 112
Σ
Au - 1)( - -1)
1=1
11
0
Σ
X 1)31
Furthermore, we have, by the spectral theorem [75, Theorem 6.35, Section 6.2.1].
2
E
Σ
(11
-2 Σ X = 21|Hxx,
20
is evident that lim 21176x, 1|? 0 since (,)))))
Σj Mn (t,,), and thus we have
Eee (Zz 1) 4 0, as 17 + Hence, by the Dominated Convergence Theorem [76, Theorem 3, Section 1.3], we have
Σ Au 11
-1) =1.
lim ((n) = lim E
exp
7-
==1
This completes the proof that ((-) is continuous 0 R
B.3 The Detailed Proofs of Lemma 3
We begin with useful Theorem which we now present.
Theorem 12. [77, Eqn. (5.7)) If the function fa(:;e) 16) is bounded given kernel K, i.e., S
h(w1, W2 k) < b, the following holds for samples W {wi, W2,.. ..,, w,, Wm
(r() - 0 2 t) 5 2 exp (- (-2nmp222))))///111111 (--2 a)))
where 0 = [(((wi, W2; x2)
We now present the detailed proofs of Lemma 3 as follows.
Proof. Denote by EH the covariance matrix of nU (WH.) under null hypothesis Ho, consisting of (co)variances n² Ho with 1 < a < as follows, by [78, Theorem 2, Section 1.4),
-1 2
n aH = COV (nU th (WH.), H))) n
n
Cr. (22)
r
2 r
r=1
21

---

for some specific Sr- For the first-order degenerate U stattisss,,,,, we have 0 and
n² Ho = n²
11
C2
(23)
2
where (2 denotes the second-order projection term defined as
C2 = Ew, wy [h(w1, W2; Ra (w1, W2; kb)]
with wi and wa independently drawn under the null hypothesis Ho- A natural estimator for (2, based on the U-statistic, is given by
-1
n2 2
Σ
h(w/, w/: Ka N/ww w/ k6)
=
2
IShniignn
with w/, wlo Wio
By Theorem 12, for any arbitrarily small positive number e, the term 122 - {2l satisfies the inequality
Pr
(122 621 > e)
s 2 2exp(-21n/2je
/b22 )²)²)
as 70000
This proves that
22 2
- 00.
(24)
For the entries n² Ho.a, of the estimated covariance matrix H. defined as
-2
(n) -1
n² a n²
h(w, w// Ka Maa/ w/y;;,a) ki) n²
2
2
1sinciasn
we have that,
0 Ho
B Ho...
00
by combining Eqns. (23) and (24). This convergence implies that
ΣHO P EHo
as
n- 00 00.
Thus, the proof is complete.
B.4 The Detailed Proofs of Theorem 5
We begin with useful Theorem follows.
Theorem 13. [71, Theorem 4.2.3] Under the alternative hypothesis Hor non-degenerate h(;; R) where ?[h?(w, Wz! k]] < oo for each ks E K., the following holds
vn(U,x - UK (W)) 4 NO EH.)
where (K( = E (W)) E [h(wi, w2; and Σn, is the covariance matrix of VnU, (W), whose entries consist of (co)variances aa with 1 I a, cas follows
oH1.a.h 4 (E [/ (wi Ka)hi (wi) kb)] Uh (W)U**(W)
We now present the detailed proofs of Theorem 5 as follows.
Proof. The asymptotic distribution of UK (W) follows directly from Theorem 13. the following, analyze the asymptotic distribution of T TT(()))))))) (W).
22

---

By applying the Taylor series expansion of T, (W) at (W), we obtain
(W)
n²(U* (M 2n (U) (W)) T SH
+ (W) S/O ([W))
Furthermore, it evident that | (W) U 0 (1//n) from Theorem 13. Building on this, we have
3/2 (Th n² (U* (W)) {
2/2 (U, (W) - UW(W)) U (W
+ 1/2 (U, (W) -U- T Σz11 (Ux (W) U*
= 2n 1/2 (U (W) - (+)) ET (W) O(1/vn).
By applying Lemma and Theorem 13, and invoking Slutsky's theorem [65], we obtain the following asymptotic distribution
n -3/2
3/2 (T* - n² EH. ( ( (W)( (00
where o2, 4 (U*(WW)) 2H,
B.5 The Detailed Proofs of Theorem 6
The proof of Theorem builds upon the results established in [7, Proposition 1]. We present detailed proof' of Theorem 6 as follows.
Proof. Under the null hypothesis Ho. and following the analysis [7, Appendix F.I] for MMD and HSIC, we have that the corresponding test U-statistic (Wte) shares the same asymptotic distribution as its b-th wild bootstrap U -tsttiit nU...(WW..) defined as
12
UWie)
Σ
ca C, w.yk).
IS Sipsi
for all b [B] and all E K. Consequently, by defining the multiple kernel forms
(W()) (n U (Wie), n- UN* (Wie) we))...,, U (We))) (n Un. (Wte), n, U, U (Wic).... UWie)))))))
U (WW))
it follows that the multivariate U-statistics nU (W.e) and nU share the same asymptotic distribution by Cramér- Wold Theorem, as shown [31, Section 6. using the fact that the U-statistics are linear with respect to their kernel parameter. Furthermore, in the asymptotic manner, we have LH LH, established in Lemma 3 with ΣH, LHo LHo and EHO Lng Based Slutsky's Theorem [65], it follows directly from Theorem that
L (Wre)
S G
i LH U.K (Wie)
According to the definition in Eqn. (9), the test statistic and the wild bootstrap statistic can be expressed as follows
T = max(nn diag( F.r U (Wit). 0)1 Tb = max n diagi F., i Ho U (Wee), 0
is observed that the mapping x + max(æ, 0) is continuous. Consequently, T and 7 are continuous functions of G. and n _ Ho U.b.b (.K,b(WWe)) respectively.
23

---

(a) Type Error (MMD-DUAL)
Type Error (HSIC-DUAL)
0.10
0.10
0.06
Type
0.04
0.02
0.02
0.00
Sample Size
Sample Size
Figure 5: Two-sample testing: (a) Type I error checking experiments dataset BLOB, MNIST and ImageNet; and Independence testing: (b) Type error checking experiments dataset Higgs, MNIST and CIFARIO. The Type error results are averaged over 1,000 repetitions under the significant level 0.05. n, no refers set of six sample sizes associated with each dataset presented in Figure 3, where the specific sample sizes vary across datasets.
Finally, by Continuous Mapping Theorem [68], we have
T 4 || ||axx (diag Ftr L GK, 0) 0)12
7 4 |maax (diag(Fir) Gx. 0)
This completes the proof.
C Supplementary Experimental Results
In this Section, will illustrate the details the experiments conducted two-sample testing and independence testing. Moreover, we present the results of extra Type-I error check and ablation experiments on additional datasets.
C.1 Experimental Details
Initialization of Kernels. We allow 6 different categories of kernels to be aggregated together: O+L, O+G, O+M, R+L, R+G and R+M, where O denotes original features, R denotes representation of original features learned by deep neural network [6]. L denotes Laplacian kernel, denotes Gaussian kernel and M denotes Mahalanobis kernel. For the initialization of bandwidths for each categories, we can either select from scaled median heuristics [7, 9] or by grid-search [51]. For the median heuristics based approach, specifically, distances between samples, and then select bandwidths uniformly between the minimum of the 0.05 quantile and the maximum of the 0.95 quantile. With fixed kernel set size this procedure yields the same kernel set for the same data, ensuring both performance and reproducibility.
compute the 0.05 and 0.95 quantiles of the pairwise
In Table 2, we further analyze the sensitivity of the kernel set initialization. Specifically, we compare the test powers of the MMD-DUAL, MMDAgg, and AMDD-FUSE methods when the kernel sets initialized using either the heuristic approach random initialization. From
(with 10 kernels) the result, we find that across all methods, heuristic initialization yields higher and more stable test power, as indicated by lower standard deviations, compared to random initialization. Moreover, randomly initializing the kernel sets leads to a noticeable drop in performance for all methods, generally resulting in lower test power and higher standard deviations. However, MMD-DUAL differs from MMDAgg and MMD-FUSE in that it performs kernel optimization. While MMDAgg and MMD-FUSE fully rely on the initial kernel sets, MMD-DUAL can adaptively optimize over diverse kernel pool even when the initialization is not ideal. This enables MMD-DUAL to maintain relatively high and stable test power under random initialization.
Learnable parameters for kernel learning. As we use four different kinds of kernels, we will list the parameters for each kernel below:
Gaussian kernel: Ne(x, v) exp(-llas v112/o7), where bandwidth o; is the trainable
parameters.
24

---

Table 2: Test power.Lstandare deviation for dataset BLOB under heuristic kernel initialization and random kernel initialization. N is the number of pains of two samples.
Heuristic
N=50
N=100
N=150
N=200
N=250
N=300
MMD-DUAL
0.111000015
0.454
0.022
0.750 + 0.028
0.915
0.016
0.983 :: 0.005
0.998
0.001
MMDAgE
0.124 + 0.055
0.270 + 0.076
0.539 + 0.073
0.764 + 0.053
0.941 + 0.026
0.987 +: 0.008
MMD-FUSE
0.158 + 0.055
0.346 0.065
0.673 + 0.084
0.863 0.042
0.972 + 0.011
0.998 0.002
Random
MMD-DUAL
0.081 0.067
0.365 + 0.123
0,420 + 0.084
0.585 0. 0.038
0.737 :: 0.055
0.825 0.061
MMDAgE
0.058 +: 0.028
0,148 0.054
0.210 0.056
0,410 0.163
0.675 +: 0.158
0.733 0.169
MMD-FUSE
0.095 + 0.067
0.098 + 0.048
0.203 + 0.152
0.439 + 0.181
0.656 + 0.163
0.740 0.083
2. Laplacian kernel: (z, v) = exp(-1|x - vll/os), where bandwidth 0; is the trainable parameters.
3. Mahalanobis kernel: <2 y) exp( د (x - y)), where the covariance matrix Σ; is the trainable parameters.
4. Deep kernel:
(; y) = [(1 -c 1lo
( (z)
uw (v) 101l//02 (xxpp11 ul1/ ul1////
where the parameters wi in the deep neural network ow, bandwidths de. and og and the weight are all trainable.
In the reproducible code repository (it can be redirected from the provided code link), we use all four kernels. However, for simplicity use, even gaussian kernels with different bandwidths can achieve high performance.
Two-sample Testing Baselines and Experiments. In two-sample testing experiments, we compare our proposed MMD-DUAL with 4 state-of-ther thrt baselines. The implementations of AutoTST [63], MMDAgg [8], MMD-D [6] and MMD-FUSE [9] can all be found in the GutHub link displayed in their papers' Abstract. For MMDAgg and MMD-FUSE, we use the default settings of total number of twenty bandwidths (ten O+L and ten O+G). For the implementation of our proposed MMD-DUAL, we use one O+G and one O+M for BLOB dataset, use each of the O+L, O+G, O+M, R+L, R+G and R+M for MNIST dataset and use one O+G, one O+M and one R+G for ImageNet dataset. For the representation model architecture, we follow the implementation of [6] in BLOB dataset and [12] in image dataset. The learning rate is 5e- for BLOB and 5e for MNIST and ImageNet. For all the two-sample testing experiments, we conduct each experiment with ten different seeds, and for each seed, we perform the testing data selection and two-sample testing process for 100 times. In total, the results are all averaging over 1,000 repetitions.
Independence Testing Baselines and Experiments. In independence testing experiments, we compare our proposed HSIC-DUAL with 3 state-of-the-art baselines. The implementations of FSIC HSICAgg [7] and HSIC-O [15] can all be found in the GitHub link displayed in their papers' Abstract. For HSIC-Agg. we use the default settings of total number of nine kernel O+G. For the implementation of our proposed HSIC-DUAL, we use four O+G, four O+L, five O+M for Higgs dataset, use two O+G, two O+L, three R+G and three R+L for MNIST dataset and use four O+G, four O+L, four R+G and four R+L for CIFARIO dataset. The representation model architecture of original features are exactly same as the two-sample testing settings. The implementation of [6] in Higgs dataset and [12] in image dataset. In independence testing. we only apply representation encoder model on the original features, not on labels. The learning rate is 5e for all three datasets. Moreover, for all the independence testing experiments, we conduct each experiment with ten different seeds, and for each seed, we perform the testing data selection and independence testing process for 100 times. In total, the results are all averaging over 1,000 repetitions.
Two-sample and Independence Testing Datasets. All the sample sizes n in the Figure 3 represents the number of samples we use in both the training phase and testing phase. Thus, for baselines without data-splitting (e.g., MMDAgg HSICAgg, FSIC and MMD-FUSE) we use 2n samples to ensure the fairness that there are same total samples included in the whole testing experiments. For two-sample testing, we use the BLOB dataset generated in the same way as [3, 6], use the adversarial MNIST the same as [6, 8] and use the ImageNetV2 the same as [6]. For independence testing, we generate different linear perturbations on each dimensions of the original Higgs data to measure the test power [79]. For image datasets, we generate the corruption of labels with percentage of 75% (only 25% of the images have their original labels) to measure the test power, followed by [7]. For
25

---

1025
(4) Power of Methods
(8) Kemee Selection Probob By (Nu0
[0] Kemei Selection Probabiity (Mbemative)
(d) Observised T-Stalistes
1.0
1.05
0.45
0.30
Pobleslly
05
Sceud
Pibbwosity
T-Stofittes
T2"
0.85
OBL Juljeed
0.30
AUX8-0-IUUA
02
02
Sample
Sample
Sample
Sample
Power of Methoss
10 Kemal Selection Probabilly (Nul)
Karnal Selection Probebiliyy (Allumativ))
(h) Observised T-Stalities
255
Thecaldd pdle
Thassdi swad
0.0
200
aded
Probuaciity
Probabbity
0.5
150
Ott.1 _d
100
ALHS-D-0HA
AUHO AUHO
02
0.0
40
Sample Size
Bample Bize
Sample Size
Sampl Bize
Figure 6: Ablation Study on the effectiveness of diversity and selection inference on two more image datasets. (a- dd) are ablation study for MMD-DUAL MNIST; ()) are ablation study for HSIC-DUAL on MNIST. (a, Test power for model variants: AU represents simple Aggregated U-Statisties; represents selection inference technique; D represents considering diversity into AU; AU+S+D refers to our proposed DUAL.
all three datasets under the null hypothesis in two-sample testing, we draw two samples from same distribution (e.g., both from adversarial images). For all three datasets under the null hypothesis the independence testing, we independently and separately draw two samples from the original datasets.
Implementation resources. The experiments of the work are conducted on two platforms. One plat- form an Nvidia-4090 GPU PC with Pytorch framework. Another platform a perfoorrfore Computer cluster with several Nvidia-Ar 1100 GPUs with Pytorch framework. The memory of two platforms are both 64 GB. The storage of disk of two platforms are both over 4 TB.
C.2 Additional Experiments Results
Type-I error Check. In Figure 5, we conduct the Type-I error check under the null hypothesis, the Type-I error of DUAL is controlled at the desired significance level = 0.05. The choice of significance level value and the procedure of Type-I error check also follow all the previous works in two-sample testing and independence testing [1, 7, 8, 6).
Additional Ablation Results. In Figure 6, we conduct additional ablation experiments to show that, the analysis in Section 6.2 is consistent across different datasets and different testing methods, proving the effectiveness of the learned diverse kernels and selection inference technique from DUAL.
Time Complexity Results. In Table 3, we present the running time of -UAL, MMDAgg and MMD-FUSE across different datasets and sample sizes. In this experiment, all three methods initialized with the same kernel set, following the experimental setup of [8]. From the result, we observe that all methods have similar testing times. The key difference lies in the training phase: MMDAgg gg and MMD-FUSE do not involve any optimization and therefore incur training time, while MMD-DUAL includes a kernel optimization phase, resulting in a training cost.
D Analytical Support for Ablation Study Results
To facilitate comprehension and offer deeper insight, present simple example illustrating the advantages of our testing approach, which incorporates the alignment vector with wild bootstrap. This example serves as a more detailed analytical explanation and provides additional support for the ablation study results shown in Figures 4 and 6.
Example 1. For simplicity, assume that each component in the vector n (Wic)) are U
mutually independent and share the same seale. For each component (nīm! UK (Wee)). with
26

---

0.5, a Table 3: Running Time (seconds) per two-sample test trial. We record the total training time and testing time for three methods across 100 trials, then we divide each total time by 100. For gMDAgg and MMD-FUSE, there are no training time, and the number of samples used in testing will be double than that used in MMD-DUAL testing, since MMD-DUM has half- half train-test splitting process.
Barning Time
MINKTT
ImagoNet
N=50
N-100
No150
Ne-00
N-250
Ned00
N=20
Ne30
Nes0
Ne50
No60
Nw70
Ne<00
Ne500
Nir000
X-700
Nos00
No000
MMD-DUAL
0.047
0,043
0.054
1.291
1.540
11.001
18:237
MMDAus
0
0
0
0
Teating MMD PUSE
0
0
0
0
9
0
0
0018
1.274
0.9T
MMI-FUSE
0.001
0.02
0.002
0,02
0.003
0.004
0,288
6.628
0.025
0,26
0/26
0.23
0.600
0.43
0.963
L172
1.405
1.352
i E [c], we consider its wild bootstrap value (n in b-th iteration with b E [B),
follows thut
Pr P (Wis)) 20) = (Pr):((((u))) ((I =
< -0.5
based the i.i.d. Rademacher random variables (which are symmetric about 0). Given the fixed vector For E {-1,+11 +13 from training phase, the probability that the sign of the i-th component of mL Ho
UKK.bWWee matches the corresponding entry in Fúr is thus 0.5, i.e.
Pr (sgn U
Fbr,.) 0.5,
where sgn U probability that F.b = T[ = -Frral = is equal to 0.5. We denote Ta as the asymptotic value of 9a with B = oo in Eqn. (10) and denote by -N the asymptotic threshold of statistic (K (Wee)ll. i.e., the statistic without selection inference. Asymptotically, as c oc, follows that
is denoted by Flc in the wild bootstrap process. As a result, the
n² 12
0.57 NA
In contrast, under the alternative hypothesis H1, the asymptotic distributions of the statistic vectors
mL Ho UK (Wie) and n. 1 UK (Wor) deviate from O, as can be inferred from Theorem 5. Denote Ho and Für sgn [ u (Wer)) the sign vector. As a consequence, the probability that the signs of the i-th components of two vectors disagree, ie., Pr(F; = Fiee, Fir.] 0), is reduced some B < 0.5. Asymptotically, as c - co, follows that
by Fie = sgn [L UK
T= In F U.
= (1- 3)n²1 L (Wec)l2.
which results in an improvement in the test power as follows
Pr(T > Ta) - Pr 2 Ho (W
-aA)
Pr(T > Ta) -Pr T
> 2Ta
1
= Pr(T > >T)) - -Pr > 2(1 - -B)Ta)
0.
In this example, we assume mutual independence to derive Pr (F.t 1 condition that is challenging achieve but is empirically observed in practice due to the de-correlation property of I H. in practice, as illustrated in Figures 4 and 6 (second column). Furthermore, we investigate the probability B, which is larger than 0.5 and increases as sample size increases as shown in Figures 4 and 6 (third column).
Prr3]=
te.
E Potential Applications of the Proposed Multivariate U-Statistics
As noted in prior work, MMD-GAN [80] is generative model that relies on the standard MMD loss. In principle, this loss could be replaced with our proposed multivariate U-statistic (4), i.e., TK (W). This statistic aggregates information from MMD statistics computed with multiple kernels,
27

---

allowing it to capture richer information about the discrepancy between the target and generative distributions. By minimizing Т," (W). the generator may produce samples that better align with the target distribution compared to using the standard MMD alone.
Moreover, the proposed MMD-DUAL and HSIC-DUAL could potentially be applied various domains that require statistical two-sample or independence testing methods. For example, MMD- DUAL (two-sample) could be used for tasks involving the comparison of trusted reference against incoming data, such as out-of-distribttion detection [81]. adversarial image detection [82], and distri- bution alignment in transfer learning [83]. Similarly, HSIC-DUAL (independence) could be useful in domains that require mitigating or verifying statistical dependence, including domain generalization [84], causal discovery [85], machine unlearning [86], and trustworthy machine learning [87].
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and Section 1, we clearly demonstrate the contribution and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The assumptions are clearly stated in the theorem conditions, and the computa- tional complexity limitations discussed Section 6.3. In Section 5.1, we also identify a risk that would happen in the training process, even though we provide an approach to mitigate
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
29

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In Appendix B, we provide the complete proof process of the theoretical contributions of our work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 5, we provide detailed implementation pipeline to ensure that our experiment be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Here is the anonymous link for the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
conooicy)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //ni //nips. peips.cc/publl ppbllc/ggidessseeeppppppppb.paee htt.ggguidde/eeeeebmiiiiiiie SrodeeSumiissionPono
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix C, clearly demonstrated various experimental settings, includ- ing hyperparameters, model settings, training settings, evaluation settings, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct the main test power experiments for 1,000 repetitions to ensure the statistic significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix C, provide sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eeeuu.pp.eeeeee wwt.rri...ccppllleee cc/public//eiecccccccccccee /Etthieiiieee csGuidelinest
Answer: [Yes]
Justification: We guarantee that the research conducted in the paper complies with NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The purpose of this paper is to improve the diversity and power in kernel aggregated hypothesis testing, without any negative societal impacts.
Guidelines:
32

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners of the assets used in the paper, such code, data, and models, have been appropriately recognized, and the licenses and terms of use have been clearly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
33

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wcc//onfeeencce/222//11111 for what should or should not be described.
35


Reviews:
[{"review_id": "E0GeRYP5wh", "reviewer": "Reviewer_fLJY", "summary": "This paper aims to improve kernel-based two-sample and independence tests. Existing methods consider aggregating multiple kernels to increase statistical power, but often suffer from selecting non-diverse kernels, thereby limiting its effectiveness. To overcome this, the authors propose a new method, called DUAL, which is a novel test statistic that explicitly promotes kernel diversity by considering the covariance between different kernels. Furthermore, it introduces a selection inference framework that uses information from a training set to select a subset of kernels that are not only diverse but also individually powerful. Empirically, the proposed method is shown to achieve better performance on kernel two-sample tests on 3 datasets and kernel independence tests on 3 datasets, respectively.", "questions": "### Q1: run-time overhead ###\nWhile the proposed method DUAL can improve the test power, it also needs to pay the price of run-time ovehead\n- Q1.1: What's the time complexity of DUAL for the training stage (Sec 5.1)? \n- Q1.2: What's the time complexity of DUAL for the testing stage (Sec 5.2)?\n- Q1.3: How does those compared to the baselines, such as MMDAgg and MMD-FUSE?\n\nIt would be great if the authors can organize these comparison in the Table.\n\n### Q2: Sample size in Figure 3 ###\n- Q2.1: Any potential explanation why the sample size N in Fig-3(b) is so small (<100), compared to Fig-3(c) which is ~1k?\n- Q2.2: What's the feature dimensionality for MNIST and CIFAR10? Do you use raw image, or some learned embedding?\n\n### Q3: What are the trainable parameters in Section 5.1? ###\n- Q3.1: What's the learnable/trainable parameters for kernel learning/selection in Eq(5) of Section 5.1?\n- Q3.2: Its more like an open-ended question, just for the thought process. Can we learn various encoders such that the learned features are somehow orthogonal, which may help the multiple kernel aggregations?\n\n### Q4: Can DUAL help learn better generative models? ###\nThis is another open research question. I am curious if DUAL, compared to say the simple MMD loss, can help learn better generative models?", "limitations": "No potential negative societal impact", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths ###\n- S1: The paper writing is well-organized and pleasant to read\n- S2: The proposed method, DUAL, is well-motivated, novel, and technical sound\n\n### Weaknesses ###\n- W1: DUAL introduces run-time overhead for both the training and inference stage", "was_revised": true, "final_justification": "Most of my concerns and questions are addressed during the rebuttal. I remained my positive rating for this submission.", "rebuttal": "Thank you very much for your thoughtful and constructive review. We truly appreciate your time and effort. Below, we answer your questions in detail.\n\n**Weakness 1 + Question 1:**\n> DUAL introduces run-time overhead for both the training and inference stage\n\n> While the proposed method DUAL can improve the test power, it also needs to perform extra computation.  \n>- Q1.1: What’s the time complexity of DUAL for the training stage (Sec 5.1)?  \n>- Q1.2: What’s the time complexity of DUAL for the testing stage (Sec 5.2)?  \n>- Q1.3: How do these compare to the baselines, such as MMDagg and MMD‑FUSE?  \n>\n> It would be great if the authors could organize these comparisons in a table.\n\n**Response 1 + 1:**\n\nThanks for pointing out this question. We thoroughly discuss the complexity and compare it with previous methods as follows.\n\n*Training Phase (Using the Adam Optimizer)*\n\n**MMD-DUAL:** $O((n^2(c^2+c)+c^3+T)*M)=O((n^2c^2+c^3+T)*M)$, where $n$ denotes the sample size, $c$ is the number of kernels, $T$ is the number of kernel parameters, and $M$ represents the number of optimization epochs. In this expression: $n^2c^2$ corresponds to the complexity of computing the elements of the $c \\times c$ covariance matrix; $n^2c$ accounts for the complexity of evaluating MMD statistics using $c$ kernels; $c^3$ represents the complexity of matrix solves involving the estimated covariance matrix $\\hat{\\Sigma}_{H_0}$.\n\n**MMDAgg:** N/A (no training phase)\n\n**MMD-FUSE:** N/A (no training phase)\n\n*Testing Phase*\n\n**MMD-DUAL:** $O((n^2c^2+c^3)B)$, where $n^2c^2 + c^3$ corresponds to the complexity of computing the aggregated statistics, and $B$ is the number of wild bootstrap iterations.\n\n**MMDAgg:** $O(n^2c(B+B'))$, where $n^2c$ corresponds to the time complexity of calculating MMD statistics using $c$ kernels, $(B+B')$ is the total number of wild bootstrap iterations and permutations, typically with $B=B'$.\n\n**MMD-FUSE:** $O(n^2cB)$, where $n^2c$ corresponds to the time complexity of calculating MMD statistics using $c$ kernels, $B$ is the number of wildbootstrap iterations.\n\n| ***Time Complexity*** | MMD-DUAL| MMDAgg| MMD-FUSE | \n|--|--|--|--|\n|Training| $O((n^2c^2+c^3+T)*M)$ | N/A | N/A |\n|Testing | $O((n^2c^2+c^3)B)$ | $O(n^2c(B+B'))$ | $O(n^2cB)$ |\n\nThe time complexity primarily depends on the sample size factor $n^2$. Other parameters, such as the kernel size $c$, the number of optimization parameters $T$, the number of optimization epochs $M$, and the number of wild bootstrap iterations $B$, are considered fixed and do not vary with the sample size.\n\nWe further present the running times of MMD-DUAL, MMDAgg, and MMD-FUSE across different datasets and sample sizes, as shown in the table below. In this experiment, all three methods are initialized with the same kernel set, following the experimental setup of [4]. Moreover, since MMD-DUAL involves a training process, we follow the procedure in [1] and perform a half-half train-test split. Compared to the MMDAgg and MMD-FUSE which directly testing on the whole samples, MMD-DUAL performs the testing prcoess only on half of the data.\n\n| Running Times (s) | BLOB | | | | | | MNIST | | | | | | ImageNet | | | | | | \n|--|--|--|---|----|----|---|---|-|---|---|---|---|-----|----|----|---|---|---|\n| **Training** | N=50 | N=100 | N=150 | N=200 | N=250 | N=300 | N=20 | N=30 | N=40 | N=50 | N=60 | N=70 | N=400 | N=500 | N=600 | N=700 | N=800 | N=900 |\n| MMD-DUAL | 0.04692 | 0.04774 | 0.04834 | 0.04989 | 0.05183 | 0.05418 | 1.29088 | 1.32365 | 1.34037 | 1.35724 | 1.36015 | 1.37741 | 9.68179 | 11.08146 | 13.54574 | 15.99571 | 18.72705 | 21.98263  |\n| MMDAgg | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |\n| MMD-FUSE | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |\n| **Testing** | | | | | | | | |  | | | | | | | | | |\n| MMD-DUAL | 0.00395 | 0.00403 | 0.00406 | 0.00417 | 0.00421 | 0.00429 | 0.01778 | 0.01782 | 0.01791 | 0.01818 | 0.01821 | 0.01837 | 0.70675 | 0.80746 | 1.02429 | 1.10982 | 1.27423 | 1.47108 |\n| MMDAgg | 0.00355 | 0.00372 | 0.00382 | 0.00402 | 0.00427 | 0.00431 | 0.02936 | 0.03051 | 0.03079 | 0.03103 | 0.03111 | 0.03163 | 0.76292 | 0.95655 | 1.16676 | 1.33279 | 1.44765 | 1.59419 | \n| MMD-FUSE | 0.00141 | 0.00174 | 0.00188 | 0.00246 | 0.00294 | 0.00389 | 0.02768 | 0.02792 | 0.02817 | 0.02823 | 0.02827 | 0.02831 | 0.67961 | 0.84282 | 0.98464 | 1.17171 | 1.40641 | 1.55239 |\n\nFrom the table, we observe that all methods have similar testing times. The key difference lies in the training phase: MMDAgg and MMD-FUSE do not involve any optimization and therefore incur no training time, while MMD-DUAL includes a kernel optimization phase, resulting in a training cost.We will add the above time complexity analysis and empirical running time comparisons in our updated version.\n\n\n**Question 2:**\n> Sample Size in Figure 3\n>- Q2.1: Any potential explanation why the sample size _N_ in Fig. 3(b) is so small (< 100), compared to Fig. 3(c) which is ∼ 1 k?\n>- Q2.2: What’s the feature dimensionality for MNIST and CIFAR‑10? Do you use raw images or some learned embedding?\n\n**Response 2:**\n\nThanks for your questions about the sample size and feature dimensionality in our experimental results.\n\nRegarding the sample size, the reason why $N$ in Fig. 3(b) and Fig. 3(c) differ is due to the dataset complexity. As discussed in [1], compared to the more challenging dataset ImageNet, two-sample testing methods generally perform better on the simpler dataset MNIST vs. MNIST.Fake [2]. This is empirically supported in [3], where achieving a converging test power of 1 on MNIST requires significantly fewer samples than on ImageNet.\n\nFor input feature dimensionality, we resize MNIST images to $32\\times32\\times1$ and ImageNet images to $32\\times32\\times3$, following the standard experimental settings used in previous two-sample testing works [1,3,4]. We use raw images as input. When the deep kernel from [1] is leveraged, a neural network is trained from scratch to learn feature embeddings, transforming raw inputs into learned representations. We will include these implementation details in the next version to enhance reproducibility.\n\n**Question 3:**\n\n> Trainable Parameters in Section 5.1\n>- Q3.1: What are the learnable/trainable parameters for kernel learning/selection in Eq.(5) of Section 5.1? \n>- Q3.2: (Open‑ended) Can we learn various encoders such that the learned features are orthogonal, which may help the multiple‑kernel aggregations?\n\n**Response 3:**\n\nThanks for the questions about the learnable parameters for kernel learning. As we use four different kinds of kernels, we will list the parameters for each kernel below:\n- Gaussian kernel: $κ(x, y) = \\exp(-||x-y||^2_2/σ^2)$, where bandwidth $σ$ is the trainable parameter. \n- Laplacian kernel: $κ(x, y) = \\exp(-||x-y||_1/σ)$, where bandwidth $σ$ is the trainable parameter.\n- Mahalanobis kernel: $κ(x, y) = \\exp(-(x-y)^{T}Σ^{-1}(x-y))$, where the covariance matrix $Σ$ is the trainable parameter. \n- Deep kernel: $κ(x, y) = [(1-ϵ)\\exp(-||ϕ_{ω}(x) - \\phi_{ω}(y)||^2_2/σ_{ϕ}^2) +ϵ]\\exp(-||x-y||^2_2/σ^2)$, where the parameter $ω$ in the deep neural network $ϕ_{ω}$, bandwidths $σ_{ϕ}$ and $σ$, and the weight $ϵ$ are all trainable.\n    \nWe appreciate the suggestion and will incorporate these details in the supplementary materials to improve clarity and reproducibility, in addition to the references already cited [1,5,6].\n\nFor the open questions, it is an interesting idea that orthogonal learned representations from multiple encoders contain the most diverse information from each other. This technique can indeed be compatible with and applied to our method to enhance the test power of multi-kernel aggregation, but we should be cautious about the trade-off between the training samples used for learning the encoders and the number of samples reserved for testing.\n\n\n\n**Question 4:**\n> Can DUAL Help Learn Better Generative Models?\nThis is another open research question. I am curious if DUAL, compared to the simple MMD loss, can help learn better generative models?\n\n**Response 4:**\n\nAs noted in prior work, MMD-GAN [7] is a generative model that relies on the standard MMD loss. In principle, we can replace this loss with our proposed multi-variate $U$-statistics in Eq.(4), i.e., $T_n^K(W)$. This statistic aggregates information from MMD statistics computed with multiple kernels, enabling it to capture richer information about the difference between the target and generative distributions. By minimizing $T_n^K(W)$, the generator potentially produces samples that better match the target distribution compared to using the standard MMD alone. This suggests a promising direction for improving generative models, and we will include this discussion as part of future work in the next version.\n\n---\n\n[1]. F. Liu, W. Xu, J. Lu, G. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML, 2020.\n\n[2]. A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICML, 2016.\n\n[3]. X. Tian, L. Peng, Z. Zhou, M. Gong, A. Gretton, F. Liu. A unified data representation learning for non-parametric two-sample testing. In UAI, 2025.\n\n[4]. A. Schrab, I. Kim, M. Albert, B. Laurent, B. Guedj, and A. Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research, 2023.\n\n[5]. A. Chatterjee and B. B. Bhattacharya. Boosting the power of kernel two-sample tests. Biometrika, 2025.\n\n[6]. Z. Zhou, J. Ni, J. Yao, and W. Gao. On the exploration of local significant differences for two-sample test. In NeurIPS, 2023.\n\n[7]. C. Li, W. Chang, Y. Cheng, Y. Yang, B. Póczos. MMD GAN: Towards Deeper Understanding of Moment Matching Network. In NeurIPS, 2017.\n\n**We hope the clarifications provided have addressed your concerns. Please feel free to reach out if you have any further questions or require additional details.**"}, {"review_id": "DOOsYkEy0Y", "reviewer": "Reviewer_67oQ", "summary": "The manuscript proposed a kernel aggregating methods in hypothesis testing aiming at improves kernel-based nonparametric tests by utilizing the diversity of kernels inspired by ensemble learning,", "questions": "Can the author provide a guarantee for the error rate of the test based on the limited sample size? If so, it will significantly enhance readers' understanding of the usability of their method.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The motivation behind the method is natural, and its effectiveness is guaranteed by certain theoretical analyses.\n\nWeaknesses: Its calculation has a relatively high level of complexity (O(B*c^3*n^2)). In addition, its theoretical guarantee is based on an infinite sample. In reality, people are more concerned about how the error rate of the test is controlled when using a limited n.", "was_revised": true, "final_justification": "I maintain my previous assessment.", "rebuttal": "Thank you very much for your thoughtful and constructive review. We truly appreciate your time and effort. Below, we answer your questions in detail.\n\n**Weakness 1:**\n> Its calculation has a relatively high level of complexity $(O(Bc^3n^2))$. \n\n**Response 1:**\n\nThank you for noticing the time complexity. Here, we first revise and correct the expression of the time complexity as follows:\n$$\nO((n^2(c^2+c)+c^3)B)=O((n^2c^2+c^3)B),\n$$\nwhere $n$ denotes the sample size, $c$ is the number of kernels and $B$ is the number of wild bootstrap iterations. In this expression: $n^2c^2$ corresponds to the complexity of computing the elements of the $c \\times c$ covariance matrix; $n^2c$ accounts for the complexity of evaluating MMD statistics using $c$ kernels; $c^3$ represents the complexity of matrix solves involving the estimated covariance matrix $\\hat{\\Sigma}_{H_0}$.\n\nSecond, the time complexity is cubic in terms of the number of selected kernels, i.e., $c$. However, in the context of two-sample and independence testing, increasing the number of kernels, i.e., $c$, beyond a moderate size does not lead to noticeable gains in test power. As shown in the empirical results from [1], specifically Figure 6 in Section 5.7, increasing the number of kernels from 10 to 100 and even to 1000 does not improve power. In fact, the referenced study even considers aggregating 12,000 kernels. This supports the choice of using a small number of kernels (e.g., $c = 10$), as there is nothing to gain empirically from using a finer discretization of kernel bandwidths. With such a small number of kernels, the time complexity is rarely a bottleneck in practice.\n\nWe will add a discussion in the updated version on this point to illustrate the scalability of our method.\n\n**Weakness 2 + Question 1:**\n> In addition, its theoretical guarantee is based on an infinite sample. In reality, people are more concerned about how the error rate of the test is controlled when using a limited $n$.\n\n> Can the author provide a guarantee for the error rate of the test based on the limited sample size? If so, it will significantly enhance readers' understanding of the usability of their method.\n\n**Response 2 + 1:**\n\nThanks for raising this important question. In Theorem 1, we establish exact type-I error control for any sample size, indicating that the type-I error is maintained at the level $\\alpha$ even with the sample size $n$ is limited.\n\nWe note that classical MMD/HSIC tests benefit from finite sample power guarantees taking the form of uniform separation rates with Sobolev regularity assumption [1,2]. However, to the best of our knowledge, it remains open to theoretically analyze the exact sample complexity of the type‑II error (i.e., 1 minus the test power) when accounting for the covariances between statistics computed with different kernels. Although we have endeavored to address this, the primary challenge lies in the technical characterization of those covariance dependencies. We view this as a promising direction for future work and will add a discussion on this point in the next version.\n\n---\n\n[1]. A. Schrab, I. Kim, M. Albert, B. Laurent, B. Guedj, and A. Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research, 2023.\n\n[2]. M. Albert, B. Laurent, A. Marrel, et al. Adaptive test of independence based on HSIC measures. The Annals of Statistics, 2022, 50(2): 858-879.\n\n**We hope the clarifications provided have addressed your concerns. Please feel free to reach out if you have any further questions or require additional details.**"}, {"review_id": "BHpKVhlrKw", "reviewer": "Reviewer_GJmC", "summary": "This paper addresses the challenge of optimizing kernel aggregation for nonparametric hypothesis testing.  A new method is introduced named DUAL (U-statistic aggregation with learned kernels). They propose using covariance-informed diversity measures to balance kernel diversity and kernel effectiveness. Theoretical results demonstrate the consistency of the test power and control of the Type I error. Experimental results show the superior performance of the proposed method.", "questions": "1.  The paper states that computational cost scales cubically with the number of kernels. This makes using a very large-scale kernel candidate set infeasible. Therefore, when employing a limited feasible set of kernels, how the initialization of this kernel set is performed might significantly impact the final results. Has this initialization sensitivity been explored?\n2.  Have the authors considered moving beyond a discrete set of kernel candidates? For instance, could a continuous optimization approach, directly learning the kernel parameters, be a viable alternative to mitigate the computational limitations?\n3.  Could the proposed methodology be extended to other kernel-based application domains？", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The research appears comprehensive, with sound methodology and potential practical utility. \n\nA potential limitation, however, is that the approach may not be scalable to scenarios involving very large-scale kernel candidate sets.", "was_revised": true, "final_justification": "The authors have solved my concerns on kernel initialization and computation complexity.  Thus I consider to raise my score.", "rebuttal": "Thank you very much for your thoughtful and constructive review. We truly appreciate your time and effort. Below, we answer your questions in detail.\n\n**Weakness 1:**\n> A potential limitation, however, is that the approach may not be scalable to scenarios involving very large-scale kernel candidate sets.\n\n**Response 1:**\n\nThank you for the question regarding the scalability of our method. It is true that using a very large candidate kernel pool may lead to computational inefficiency. However, in the context of two-sample and independence testing, increasing the number of kernels beyond a moderate size does not lead to noticeable gains in test power. As shown in the empirical results from [1], specifically Figure 6 in Section 5.7, increasing the number of kernels from 10 to 100 and even to 1000 does not improve power. **This supports the choice of using a small number of kernels (e.g., $c = 10$)**, as there is nothing to gain (empirically) from using a finer discretization for the bandwidths of kernels. In fact, the referenced study even considers aggregating 12,000 kernels. We will add a discussion in the next version on this point to illustrate the scalability of our method.\n\n**Question 1:**\n> The paper states that computational cost scales cubically with the number of kernels. This makes using a very large-scale kernel candidate set infeasible. Therefore, when employing a limited feasible set of kernels, how the initialization of this kernel set is performed might significantly impact the final results. Has this initialization sensitivity been explored?\n\n**Response 1:**\n\nThanks for the question on the cubical computational cost and the initialization of kernel sets. We would like to correct the time complexity reported in the original manuscript. Specifically, the complexity should be $O((n^2c^2 + c^3)B)$, rather than $O(Bc^3n^2)$, where $n$ is the sample size, $c$ is the number of kernels, and $B$ is the number of wild bootstrap iterations. **Notably, the $c^3$ term represents the complexity of matrix solves involving the estimated covariance matrix $\\hat{\\Sigma}_{H_0}$, which is independent of the sample size $n$**. As previously discussed, using a small number of kernels (e.g., $c = 10$) has been shown to be sufficient in practice [1].\n\n**For initializing the kernel set, we follow the methodology of [1,2,3] and adopt a heuristic-based approach.** Specifically, we compute the 0.05 and 0.95 quantiles of the pairwise distances between samples, and then select bandwidths uniformly between the minimum of the 0.05 quantile and the maximum of the 0.95 quantile. With a fixed kernel set size $c$, this procedure yields the same kernel set for the same data, ensuring both performance and reproducibility.\n\nIn the following table, we further analyze the sensitivity of the kernel set initialization. Specifically, we compare the test powers of the MMD-DUAL, MMDAgg, and MMD-FUSE methods when the kernel sets (with 10 kernels) are initialized using either the **heuristic approach or random initialization:**\n| Test power | BLOB | | | | | |\n|------------|------|-------|-------|-------|-------|-------|\n|**Heuristic**| N=50 | N=100 | N=150 | N=200 | N=250 | N=300 |\n| MMD-DUAL | 0.134±0.016 | **0.454±0.022** | **0.750±0.028** | **0.915±0.016** | **0.983±0.005** | **0.998±0.001** |\n| MMDAgg | 0.124±0.055 | 0.270±0.076 | 0.539±0.073 | 0.764±0.053 | 0.941±0.026 | 0.987±0.008 |\n| MMD-FUSE | **0.153±0.055** | 0.346±0.065 | 0.673±0.084 | 0.863±0.042 | 0.972±0.011  | **0.998±0.002** |\n|**Random**|  |  |  |  |  |  |\n| MMD-DUAL | 0.084±0.067 | **0.365±0.123** | **0.420±0.084** | **0.585±0.038** | **0.737±0.055** | **0.825±0.061** |\n| MMDAgg | 0.058±0.028 | 0.148±0.054 | 0.210±0.056 | 0.410±0.163 | 0.675±0.158 | 0.733±0.169 |\n| MMD-FUSE | **0.095±0.067** | 0.098±0.048 | 0.203±0.152 | 0.439±0.181 | 0.656±0.163 | 0.740±0.083 |\n\n\nFrom the table, we observe the following:\n- **Heuristic initialization:** Across all methods, heuristic initialization yields higher and more stable test power, as indicated by lower standard deviations, compared to random initialization.\n- **Random initialization:** Randomly initializing the kernel sets leads to a noticeable drop in performance for all methods, generally resulting in **lower test power and higher standard deviations**. However, MMD-DUAL differs from MMDAgg and MMD-FUSE in that it performs kernel optimization. **While MMDAgg and MMD-FUSE fully rely on the initial kernel sets, MMD-DUAL can adaptively optimize over a diverse kernel pool even when the initialization is not ideal**. This enables MMD-DUAL to maintain relatively high and stable test power under random initialization.\n\n**We will emphasize that we follow the methodology of [1,2,3] in adopting a heuristic-based initialization approach, and we will include these experimental details and dicussions in the updated version of the paper.**\n\n**Question 2:**\n> Have the authors considered moving beyond a discrete set of kernel candidates? For instance, could a continuous optimization approach, directly learning the kernel parameters, be a viable alternative to mitigate the computational limitations?\n\n**Response 2:**\n\nThanks for the valuable insights about continuous kernel parameter optimization. As we apply four different categories of kernels (Gaussian, Laplacian, Mahalanobis, and Deep), unifying these into a single parameterized kernel family and learning the parameters jointly is indeed a promising direction to improve computational efficiency. We will discuss this exciting future direction in our next version, focusing on how to simultaneously unify these different kernels while maintaining representational diversity.\n\n**Question 3:**\n> Could the proposed methodology be extended to other kernel-based application domains？\n\n**Response 3:**\n\nThanks for this open question about the application of our proposed method. Since MMD-DUAL and HSIC-DUAL are general two-sample and independence testing methods that do not rely on any strict assumptions, they can both be employed in real-world kernel-based applications.\n\nFor example, MMD-DUAL can be effectively applied to machine-generated text detection [7], where detecting subtle distributional differences between human-written and LLM-generated texts is critical. Similarly, HSIC-DUAL is applicable in machine unlearning evaluation [8], where independence testing can be used to verify whether a model has forgotten specific training data.\n\nMoreover, we stress that our newly proposed statistics with learned diverse kernels can also be used in more general settings beyond hypothesis testing. They can improve any method based on MMD or HSIC, for which the kernel selection problem is crucial. These discrepancies are often used as training objectives in Machine Learning.\n\nAs statistical testing-based approaches become increasingly popular in emerging domains due to their theoretical guarantees, they have strong potential for extension to broader applications. We will add this discussion in our next version to illustrate the broader utility of our method beyond conventional testing scenarios.\n\n---\n[1]. A. Schrab, I. Kim, M. Albert, B. Laurent, B. Guedj, and A. Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research, 2023.\n\n[2]. F. Biggs, A. Schrab, and A. Gretton. MMD-Fuse: Learning and combining kernels for two sample testing without data splitting. In NeurIPS, 2023.\n\n[3]. A. Schrab, I. Kim, B. Guedj, and A. Gretton. Efficient aggregated kernel tests using incomplete U-statistics. In NeurIPS 2022.\n\n[4]. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. J. Smola. A kernel method for the two-sample-problem. In NeurIPS, 2006.\n\n[5]. A. Gretton, K. Fukumizu, C. H. Teo, L. Song, B. Schölkopf, and A. J. Smola. A kernel statistical test of independence. In NeurIPS, 2007.\n\n[6]. F. Liu, W. Xu, J. Lu, G. Zhang, A.Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML, 2020.\n\n[7]. S. Zhang, Y. Song, J. Yang, Y. Li, B. Han, and M. Tan. Detecting machine-generated texts by multi-population aware optimization for maximum mean discrepancy. In ICLR, 2024.\n\n[8]. R. Mehta, S. Pal, V. Singh, S. N. Ravi. Deep Unlearning via Randomized Conditionally Independent Hessians. In CVPR, 2022.\n\n**We hope our response has clarified the confusion and addressed your concerns. We would greatly appreciate it if you could kindly reconsider your score. Thank you.**"}, {"review_id": "Bum8hs0ITS", "reviewer": "Reviewer_BHAJ", "summary": "This paper addresses the challenge of improving kernel-based nonparametric hypothesis testing, specifically for two-sample and independence tests, by introducing a method called Diverse U-statistic Aggregation with Learned Kernels (DUAL). TThe proposed method leverages a data-splitting approach to learn diverse kernels during training and employs selection inference during testing to focus on the most informative kernels. Theoretical guarantees for Type-I error control and asymptotic power are provided, and extensive experiments demonstrate superior performance over state-of-the-art methods across multiple benchmarks.", "questions": "1. How does the method scale with larger kernel pools (e.g., c > 20 ) or high-dimensional data? Could the authors provide guidelines for choosing c in practice?  \n\n2. The paper mentions gradient-based kernel learning but does not discuss sensitivity to initialization or optimization challenges. \n\n3. The experiments include MMD-D (deep kernels) as a baseline. How does DUAL compare when combined with deep kernels? Is there a synergy?", "limitations": "The authors acknowledge limitations (e.g., redundancy in kernels) but could better discuss computational trade-offs and assumption violations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1.The paper is technically sound, with rigorous theoretical proofs and comprehensive experiments. \n\n2.The paper is well-structured, with clear motivations, methodology, and results. \n\n3.The work addresses a practical limitation in kernel aggregation methods, offering a solution that enhances the power of widely used tests like MMD and HSIC. \n\nWeaknesses：\n\n1.While the method is computationally feasible, the impact of kernel pool size c on scalability for very large datasets or high-dimensional kernels could be further explored.\n\n2.The paper does not deeply discuss how sensitive the method is to the choice of kernel initialization or the number of kernels c. This could affect reproducibility in practice.", "was_revised": true, "final_justification": "The author has addressed most of my concerns, and I am inclined to maintain my positive score.", "rebuttal": "Thank you very much for your thoughtful and constructive review. We truly appreciate your time and effort. Below, we answer your questions in detail.\n\n**Weakness 1 + Question 1:**\n> While the method is computationally feasible, the impact of kernel pool size c on scalability for very large datasets or high-dimensional kernels could be further explored.\n\n> How does the method scale with larger kernel pools (e.g., $c$ > 20 ) or high-dimensional data? Could the authors provide guidelines for choosing $c$ in practice?\n\n**Response 1 + 1:**\n\nThank you for the question regarding the scalability of our method. It is true that using a very large candidate kernel pool may lead to computational inefficiency. However, in the context of two-sample and independence testing, the kernel sets are selected using heuristics, following the methodology of [1,2,3], and increasing the number of kernels beyond a moderate size does not lead to noticeable gains in test power. As shown in the empirical results from [1], specifically Figure 6 in Section 5.7, increasing the number of kernels from 10 to 100 and even to 1000 does not improve power. **This supports the choice of using a small number of kernels (e.g., $c = 10$)**, as there is nothing to gain empirically from using a finer discretization for the bandwidths of kernels. In fact, the referenced study even considers aggregating 12,000 kernels.\n\nFor high-dimensional data, kernel methods scale linearly with the input dimension, but this cost only arises during the computation of the kernel matrix (often via pairwise distances), which can be implemented very efficiently and is rarely the bottleneck in practice.\n\nWe will add a discussion in the next version on these points to illustrate the scalability of our method.\n\n**Weakness 2 + Question 2:**\n> The paper does not deeply discuss how sensitive the method is to the choice of kernel initialization or the number of kernels $c$. This could affect reproducibility in practice.\n\n> The paper mentions gradient-based kernel learning but does not discuss sensitivity to initialization or optimization challenges.\n\n**Response 2 + 2:**\n\nThanks for the question on the initialization and number of kernel sets. As discussed above, in practice, using a small set of kernels (e.g., 10 kernels) is enough [1]. **For initializing the kernel set, we follow the methodology of [1,2,3] and use a heuristic-based approach.** Specifically, we compute the 0.05 and 0.95 quantiles of the pairwise distances between samples, and then select bandwidths uniformly between the minimum of the 0.05 quantile and the maximum of the 0.95 quantile. With a fixed kernel set size $c$, this procedure produces the same kernel set for the same data, ensuring both performance and reproducibility.\n\nIn the following table, we further analyze the sensitivity of the kernel set initialization. Specifically, we compare the test powers of the DUAL, Agg, and FUSE methods when the kernel sets (with 10 kernels) are initialized using either the **heuristic approach or random initialization:**\n| Test power | BLOB | | | | | |\n|------------|------|-------|-------|-------|-------|-------|\n|**Heuristic**| N=50 | N=100 | N=150 | N=200 | N=250 | N=300 |\n| MMD-DUAL | 0.134±0.016 | **0.454±0.022** | **0.750±0.028** | **0.915±0.016** | **0.983±0.005** | **0.998±0.001** |\n| MMDAgg | 0.124±0.055 | 0.270±0.076 | 0.539±0.073 | 0.764±0.053 | 0.941±0.026 | 0.987±0.008 |\n| MMD-FUSE | **0.153±0.055** | 0.346±0.065 | 0.673±0.084 | 0.863±0.042 | 0.972±0.011  | **0.998±0.002** |\n|**Random**|  |  |  |  |  |  |\n| MMD-DUAL | 0.084±0.067 | **0.365±0.123** | **0.420±0.084** | **0.585±0.038** | **0.737±0.055** | **0.825±0.061** |\n| MMDAgg | 0.058±0.028 | 0.148±0.054 | 0.210±0.056 | 0.410±0.163 | 0.675±0.158 | 0.733±0.169 |\n| MMD-FUSE | **0.095±0.067** | 0.098±0.048 | 0.203±0.152 | 0.439±0.181 | 0.656±0.163 | 0.740±0.083 |\n\nFrom the table, we observe the following:\n- **Heuristic initialization:** Across all methods, heuristic initialization yields higher and more stable test power, as indicated by lower standard deviations, compared to random initialization.\n- **Random initialization:** Randomly initializing the kernel sets leads to a noticeable drop in performance for all methods, generally resulting in **lower test power and higher standard deviations**. However, MMD-DUAL differs from MMDAgg and MMD-FUSE in that it performs kernel optimization. **While MMDAgg and MMD-FUSE fully rely on the initial kernel sets, MMD-DUAL can adaptively optimize over a diverse kernel pool even when the initialization is not ideal**. This enables MMD-DUAL to maintain relatively high and stable test power under random initialization.\n\n**We will emphasize that we follow the methodology of [1,2,3] in adopting a heuristic-based initialization approach, and we will include these experimental details and dicussions in the updated version of the paper.**\n\n**Question 3:**\n> The experiments include MMD-D (deep kernels) as a baseline. How does DUAL compare when combined with deep kernels? Is there a synergy?\n\n**Response 3:**\n\nThank you for the question regarding the combination of deep kernels. We do aggregate over deep kernels in our method. In our experiments, we can use four categories of kernels: Gaussian, Laplace, Mahalanobis, and deep kernels. We will include these important implementation details in the next revision.\n\n**Question 4:**\n>The authors acknowledge limitations (e.g., redundancy in kernels) but could better discuss computational trade-offs and assumption violations.\n\n**Response 4:**\n\nThank you for the suggestion. We will include a discussion on the computational trade-offs as noted in Response 1+1, and also discuss the assumptions made in our paper in more detail. The boundedness assumption on the kernel is standard [1–6]. The kernels used in our paper, including Gaussian, Laplace, Mahalanobis, and deep kernels, are all bounded. For unbounded kernels, such as $\\kappa(x,y)=\\\\|x - y\\\\|^2_2$, the data can be normalized to a fixed range to ensure this assumption is satisfied. Assumption 1 is used to analyze the consistency of our test as $n \\to \\infty$. As shown in Figures 4(c) and 4(g), this assumption is easily satisfied in practice, since the probability of selecting at least one kernel quickly approaches 1 even with a limited sample size.\n\n---\n\n[1]. A. Schrab, I. Kim, M. Albert, B. Laurent, B. Guedj, and A. Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research, 2023.\n\n[2]. F. Biggs, A. Schrab, and A. Gretton. MMD-Fuse: Learning and combining kernels for two sample testing without data splitting. In NeurIPS, 2023.\n\n[3]. A. Schrab, I. Kim, B. Guedj, and A. Gretton. Efficient aggregated kernel tests using incomplete U-statistics. In NeurIPS 2022.\n\n[4]. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. J. Smola. A kernel method for the two-sample-problem. In NeurIPS, 2006.\n\n[5]. A. Gretton, K. Fukumizu, C. H. Teo, L. Song, B. Schölkopf, and A. J. Smola. A kernel statistical test of independence. In NeurIPS, 2007.\n\n[6]. F. Liu, W. Xu, J. Lu, G. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML, 2020.\n\n**We hope the clarifications provided have addressed your concerns. Please feel free to reach out if you have any further questions or require additional details.**"}]

Author rebuttal:

Meta-review: This paper presents a novel framework, DUAL (Diverse U-statistic Aggregation with Learned Kernels), to enhance kernel-based nonparametric hypothesis testing, specifically targeting two-sample and independence tests. The core innovation lies in promoting kernel diversity alongside individual kernel effectiveness. The authors provide solid theoretical support, including Type-I error control and power consistency. 

Strengths：All reviewers agree that the paper is technically sound, well-structured, and supported by rigorous theoretical analysis and comprehensive experimental validation. The writing is clear and pleasant to read, making the methodology and contributions easy to follow. Reviewer BHAJ particularly appreciates the combination of theoretical rigor and clear empirical improvements over existing kernel aggregation methods. 

Weaknesses：The reviewers primarily raised concerns regarding the computational overhead and scalability of the proposed method, especially when the number of kernels c becomes large. In addition, Reviewer BHAJ expressed concerns about the sensitivity of the method to initialization or the number of kernels. 

The most important reasons for the decision：This paper addresses a practical limitation in kernel aggregation methods. The proposed methods are both theoretically solid and experimentally well grounded.

Summarization of the discussion and changes during the rebuttal period:  Some reviewers raised concerns about the scalability of the method, which the authors addressed with additional experiments and supportive literature in the rebuttal. 

Overall comments: 

After the rebuttal phase, all reviewers raised their score to accept, resulting in unanimous acceptance. The paper investigates aggregated kernel-based nonparametric hypothesis testing. All reviewers agree that the proposed methods are both theoretically solid and experimentally well grounded. On the other hand, concerns remain regarding the computational overhead and scalability of the proposed method, especially when the number of kernels c becomes large. Overall, the quality of the work is high and solid, and it is likely to be of interest to the community. Therefore, I recommend acceptance as a poster.
Decision: Accept (poster)