Title: MMD-Fuse: Learning and Combining Kernels for Two-Sample Testing Without Data Splitting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Testing

MMD

Kernel Methods

Two-sample testing
Abstract: We propose novel statistics which maximise the power  of a two-sample test based on the Maximum Mean Discrepancy (MMD), by
adapting over the set of kernels used in defining it.
For finite sets, this reduces to combining (normalised) MMD values under each of these kernels via a weighted soft maximum.
Exponential concentration bounds are proved for our proposed statistics under the null and alternative.
We further show how these kernels can be chosen in a data-dependent but permutation-independent way, in a well-calibrated test, avoiding data splitting.
This technique applies more broadly to general permutation-based MMD testing, and includes the use of deep kernels with features learnt using unsupervised models such as auto-encoders.
We highlight the applicability of our MMD-Fuse tests on both synthetic low-dimensional and real-world high-dimensional data, and compare its performance in terms of power against current state-of-the-art kernel tests.

MMD-FUSE: Learning and Combining Kernels for Two-Sample Testing Without Data Splitting

Abstract

We propose novel statistics which maximise the power of a two-sample test based on the Maximum Mean Discrepancy (MMD), by adapting over the set of kernels used in defining it. For finite sets, this reduces to combining (normalised) MMD values under each of these kernels via a weighted soft maximum. Exponential concentration bounds are proved for our proposed statistics under the null and alternative. We further show how these kernels can be chosen in a data-dependent but permutation-independent way, in a well-calibrated test, avoiding data splitting. This technique applies more broadly to general permutation-based MMD testing, and includes the use of deep kernels with features learnt using unsupervised models such as auto-encoders. We highlight the applicability of our MMD-FUSE test on both synthetic low-dimensional and real-world high-dimensional data, and compare its performance in terms of power against current state-of-the-art kernel tests.

1 Introduction

The fundamental problem of non-parametric two-sample testing consists in detecting the difference between any two distributions having access only to samples from these. Kernel-based tests relying on the Maximum Mean Discrepancy (MMD; Gretton et al., 2012a) as a measure of distance on distributions are well-suited for this framework as they can identify complex non-linear features in the data, and benefit from both strong theoretical guarantees and ease of implementation. This explains their popularity among practitioners and justifies their wide use for real-world applications.

However, the performance of these tests is crucially impacted by the choice of kernel. This is commonly tackled by either: choosing the kernel by some weakly data-dependent heuristic (Gretton et al., 2012a); or splitting off a hold-out set of data for kernel selection, with the other half used for the actual test (Gretton et al., 2012b; Sutherland et al., 2017). This latter method includes training feature extractors such as deep kernels on the first selection half. Both of these methods can incur a significant loss in test power, since heuristics may lead to poor kernel choices, and data splitting reduces the number of data points for the actual test.

Our contribution is to present MMD-based tests which can strongly adapt to the data without data splitting. This comes in two parallel parts: firstly we show how the kernel can be chosen in an

unsupervised fashion using the entire dataset, and secondly we show how multiple such kernels can be adaptively weighted in a single test statistic, optimising test power.

Data Splitting. The data splitting approach selects test parameters on a held-out half of the dataset, and applies the test to the other half. Commonly, this involves optimising a kernel on held-out data in a supervised fashion to distinguish which sample originated from which distribution, either by learning a deep kernel directly (Sutherland et al., 2017; Liu et al., 2020, 2021), or indirectly through the associated witness function (Kübler et al., 2022a,b). Jitkrittum et al. (2016) propose tests which select witness function features (in either spatial or frequency space) on the held-out data, running the analytic representation test of Chwialkowski et al. (2015) on the remaining data.

Our first contribution is to show that it is possible to learn a feature extractor for our test ( e.g. , a deep kernel) on the entirety of the data in an unsupervised manner while retaining the desired non-asymptotic test level. Specifically, any method can be used that is ignorant of which distribution generated which samples. We can thus leverage many feature extraction methods, from auto-encoders (Hinton and Salakhutdinov, 2006) to recent powerful developments in self-supervised learning (He et al., 2020; Chen et al., 2020a,b,c; Chen and He, 2021; Grill et al., 2020; Caron et al., 2020; Zbontar et al., 2021; Li et al., 2021). Remarkably, our method applies to any permutation-based two-sample test, even non-kernel tests, provided the parameters are chosen in this unsupervised fashion. This includes a very wide array of MMD-based tests, and finally provides a formal justification for the commonly-used median heuristic (Gretton et al., 2012b; Ramdas et al., 2015; Reddi et al., 2015).

Adaptive Kernel Selection. A newer approach originating in Schrab et al. (2023) performs adaptive kernel selection through multiple testing. Aggregating several MMD-based tests with different kernels, each on the whole dataset, results in an overall adaptive test with optimal power guarantees in terms of minimax separation rate over Sobolev balls. Variants of this kernel adaptivity through aggregation have been proposed with linear-time estimators (Schrab et al., 2022b), spectral regularisation (Hagrass et al., 2022), kernel thinning compression (Domingo-Enrich et al., 2023), and in the asymptotic regime (Chatterjee and Bhattacharya, 2023). Another adaptive approach using the entire dataset for both kernel selection and testing is given by Kübler et al. (2020); this leverages the Post Selection Inference framework, but the resulting test suffers from low power in practice.

While our unsupervised feature extraction method is an extremely general technique and potentially powerful, particularly for high-dimensional structured data like images, it is not always sufficient for data where such feature extraction is difficult. This motivates our second contribution, a method for combining whole-dataset MMD estimates under multiple different kernels into single test statistics, for which we prove exponential concentration. Kernel parameters such as bandwidth can then be chosen in a non-heuristic manner to optimise power, even on data with less structure and varying length scales. Using a single statistic also ensures that a single test can be used, instead of the multiple testing approach outlined above, reducing computational expense.

MMD-FUSE. By combining these contributions we construct two closely related MMD-FUSE tests. Each chooses a set of kernels based on the whole dataset in an unsupervised fashion, and then adaptively weights and fuses this (potentially infinite) set of kernels through our new statistics; both parts of this procedure are done using the entire dataset without splitting. On the finite sets of kernels we use in practice, the weighting procedure is done in closed form via a weighted soft maximum. We show these new tests to be well-calibrated and give sufficient power conditions which achieve the minimax optimal rate in terms of MMD. In empirical comparisons, our test compares favourably to the state-of-the-art aggregated tests in terms of power and computational cost.

Outline and Summary of Contributions. In Section 2 we outline our setting, crucial results underlying our work, and alternative existing approaches. Section 3 covers the construction of permutation tests and discusses how we can choose the parameters for any such test in any unsupervised fashion including with deep kernels and by those methods mentioned above. Section 4 introduces and motivates our two proposed tests. Section 5 discusses sufficient conditions for test power (at a minimax optimal rate in MMD), and the exponential concentration of our statistics. Finally, we show that our test compares favourably with a wide variety of competitors in Section 6 and discuss in Section 7.

2 Background

Two-Sample Testing. The two-sample testing problem is to determine whether two distributions p and q are equal or not. In order to test this hypothesis, we are given access to two samples, X := ( X 1 , . . . , X n ) iid ∼ p and Y := ( Y 1 , . . . , Y m ) iid ∼ q as tuples of data points with sizes n and m . We write the combined (ordered) sample as Z := ( Z 1 , . . . , Z n + m ) = ( X , Y ) = ( X 1 , . . . , X n , Y 1 , . . . , Y m ) .

̸

We define the null hypothesis H 0 as p = q and the alternative hypothesis H 1 as p = q , usually with a requirement D ( p, q ) > ϵ for a distance D (such as the MMD) and some ϵ > 0 . A hypothesis test ∆ is a { 0 , 1 } -valued function of Z , which rejects the null hypothesis if ∆( Z ) = 1 and fails to reject it otherwise. It will usually be formulated to control the probability of a type I error at some level α ∈ (0 , 1) , so that P p × p (∆( Z ) = 1) ≤ α, while simultaneously minimising the probability of a type II error, P p × q (∆( Z ) = 0) . In the above we have used the notation P p × p and P p × q to indicate that the sample Z is either drawn from the null, p = q , or the alternative p = q . Similar notation will be used for expectations and variances. When a bound β ∈ (0 , 1) on the probability of a type II error is given (which may depend on the precise formulation of the alternative), we say the test has power 1 -β .

̸

Maximum Mean Discrepancy. The Maximum Mean Discrepancy (MMD) is a kernel-based measure of distance between two distributions p and q , which is often used for two-sample testing. The MMD quantifies the dissimilarity between these distributions by comparing their mean embeddings in a reproducing kernel Hilbert space (RKHS; Aronszajn, 1950) with kernel k ( · , · ) . Formally, if H k is the RKHS associated with kernel function k , the MMD between distributions p and q is the integral probability metric defined by:

The minimum variance unbiased estimate of MMD 2 k , is given by the sum of two U-statistics and a sample average as: 2

̸

where we introduced the notation [ n ] 2 = { ( i, i ′ ) ∈ [ n ] 2 : i = i ′ } for the set of all pairs of distinct indices in [ n ] = { 1 , . . . , n } . Tests based on the MMD usually reject the null when ̂ MMD 2 k exceeds some critical threshold, with the resulting power being greatly affected by the kernel choice. For characteristic kernel functions (Sriperumbudur et al., 2011), it can be shown that MMD k ( p, q ) = 0 if and only if p = q , leading to consistency results. However, on finite sample sizes, convergence rates of MMD estimates typically have strong dependence on the data dimension, so there are settings in which kernels ignoring redundant or unimportant features ( i.e. non-characteristic kernels) will give higher test power in practice than characteristic kernels (which can over-weight redundant features).

Distributions Over Kernels. In our test statistic, we will consider the case where the kernel k ∈ K is drawn from a distribution ρ ∈ M 1 + ( K ) (with the latter notation denoting a probability measure; c.f. Benton et al., 2019 in the Gaussian Process literature). This distribution will be adaptively chosen based on the data subject to a regularisation term based on a 'prior' π ∈ M 1 + ( K ) and defined through the Kullback-Liebler divergence: KL( ρ ∥ π ) := E ρ [log(d ρ/ d π )] for ρ ≪ π and KL( ρ ∥ π ) := ∞ otherwise. When constructing these statistics the Donsker-Varadhan equality (Donsker and Varadhan, 1975), holding for any measurable g : K → R , will be useful:

This can be further related to the notion of soft maxima. If π is a uniform distribution on finite K with |K| = r , then KL( ρ ∥ π ) ≤ log( r ) . Setting g = tf for some t > 0 , Equation (1) relaxes to

which approximates the maximum with error controlled by t . Our approach in considering these soft maxima is reminiscent of the PAC-Bayesian (McAllester, 1998; Seeger et al., 2001; Maurer, 2004; Catoni, 2007; see Guedj, 2019 or Alquier, 2021 for a survey) approach to capacity control and generalisation. This framework has raised particular attention recently as it has been used to provide the only non-vacuous generalisation bounds for deep neural networks (Dziugaite and Roy, 2017, 2018; Dziugaite et al., 2021; Zhou et al., 2019; Perez-Ortiz et al., 2021; Biggs and Guedj, 2021, 2022a); it has also been fruitfully applied to other varied problems from ensemble methods (Lacasse et al., 2006, 2010; Masegosa et al., 2020; Wu et al., 2021; Zantedeschi et al., 2021; Biggs and Guedj, 2022b; Biggs et al., 2022) to online learning (Haddouche and Guedj, 2022). Our proofs draw on techniques in that literature for U-Statistics (Lever et al., 2013) and martingales (Seldin et al., 2012; Biggs and Guedj, 2023; Haddouche and Guedj, 2023; Chugg et al., 2023). By considering these soft maxima, we gain a major advantage over the standard approaches, as we can obtain concentration and power results for our statistics without incurring Bonferroni-type multiple testing penalties.

Permutation Tests. The tests we discuss in this paper use permutations of the data Z to approximate the null distribution. We begin our discussion in a fairly general form to include other close settings such as independence testing (Albert et al., 2022; Rindt et al., 2021) or wild bootstrap-based twosample tests (Fromont et al., 2012, 2013; Chwialkowski et al., 2014; Schrab et al., 2023). Let G be a group of transformations on Z ; in our setting G = S n + m , denoting the permutation (or symmetric) group of [ N ] by S N and its elements by σ ∈ S N , σ : [ N ] → [ N ] . We write g Z for the action of g ∈ G on Z ; e.g. defining σ Z = ( Z σ (1) , . . . , Z σ ( n + m ) ) for the group elements σ ∈ S n + m . We will suppose Z is invariant under the action of G when the sample is drawn from the null distribution, i.e. if the null is true than g Z = d Z (by which we notate equality of distribution) for all g ∈ G . This is clearly the case for G = S n + m in two-sample testing, since under the null Z = ( Z 1 , . . . , Z m + n ) iid ∼ p with p = q , while under the alternative, the permuted sample σ Z for randomised σ ∼ Uniform( S n + m ) simulates the null distribution.

We can use permutations to construct an approximate cumulative distribution function (CDF) of our test statistic under the null, and choose an appropriate quantile of this CDF as our test threshold, which must be exceeded under the null with probability less that level α . For this we introduce a quantile operator (analogous to the max and min operators) for a finite set { f ( a ) ∈ R : a ∈ A} :

Various different results can be used to choose the threshold giving a correct level; we will here highlight a very general and easy-to-use theorem for constructing permutation tests, and in Section 3 will discuss previously unconsidered implications for using unsupervised feature extraction methods as a part of our tests. Although this result is not new, we believe that its usefulness has been underappreciated in the kernel testing community, and can be more conveniently applied than Romano and Wolf (2005, Lemma 1), which requires exchangeablity and is commonly used, e.g. in Albert et al. (2022, Proposition 1) and Schrab et al. (2023, Proposition 1).

Theorem 1 (Hemerik and Goeman, 2018, Theorem 2.) . Let G be a vector of elements from G , G = ( g 1 , g 2 , . . . , g B +1 ) , with g B +1 = id (the identity permutation) for any B ≥ 1 . The elements g 1 , . . . , g B are drawn uniformly from G either i.i.d. or without replacement (which includes the possibility of G = G ). If τ ( Z ) is a statistic of Z and Z = d g Z for all g ∈ G under the null then

In other words, if we compare τ ( Z ) with the empirical quantile of τ ( g Z ) as a test threshold, the type I error rate is no more than α . 3 The potentially complex task of constructing a permutation test reduces to the trivial task of choosing the statistic τ ( Z ) . This result is also true for randomised permutations of any number B ≥ 1 without approximation, so an exact and computationally efficient test can be straightforwardly constructed this way.

We finally mention the related approach of the MMD Aggregated test (MMDAgg; Schrab et al., 2023), which combines multiple MMD-based permutation tests with different kernels, and rejects if

̸

̸

any of these reject, using distinct quantiles for each kernel. To ensure the overall aggregated test is well-calibrated, these quantiles must be adjusted using a second level of permutations. This incurs additional computational cost, a pitfall avoided by our fused single statistic.

Faster Sub-Optimal Tests. While the main focus of this paper revolves around the kernel selection problem for optimal, quadratic MMD testing, we also highlight the existence of a rich literature on efficient kernel tests which run in linear (or near-linear) time. These speed improvements are achieved using various tools such as: incomplete U-statistics (Gretton et al., 2012b; Yamada et al., 2019; Lim et al., 2020; Kübler et al., 2020; Schrab et al., 2022b), block U-statistics (Zaremba et al., 2013; Deka and Sutherland, 2022), eigenspectrum approximations (Gretton et al., 2009), Nyström approximations (Cherfaoui et al., 2022), random Fourier features (Zhao and Meng, 2015), analytic representations (Chwialkowski et al., 2015; Jitkrittum et al., 2016), deep linear kernels (Kirchler et al., 2020), kernel thinning (Dwivedi and Mackey, 2021; Domingo-Enrich et al., 2023), etc.

The efficiency of these tests usually entail weaker theoretical power guarantees compared to their quadratic-time counterparts, which are minimax optimal 4 (Kim et al., 2022; Li and Yuan, 2019; Fromont et al., 2013; Schrab et al., 2023; Chatterjee and Bhattacharya, 2023). These optimal quadratic tests are either permutation-based non-asymptotic tests (Kim et al., 2022; Schrab et al., 2023) or studentised asymptotic tests (Kim and Ramdas, 2023; Shekhar et al., 2022; Li and Yuan, 2019; Gao and Shao, 2022; Florian et al., 2023). We emphasise that the parameters of any of these permutationbased two-sample tests can be chosen in the unsupervised way we outline in Section 3. We choose to focus in this work on optimal quadratic-time results, but note that our general approach could be extended to sub-optimal faster tests as well.

3 Learning Statistics for Permutation Tests

Here we discuss how we can improve permutation-based tests by learning parameters for them in an unsupervised manner. The reason that we highlighted Theorem 1 in the previous section is that it holds for any τ . For example, we could use the MMD with a kernel chosen based on the data, τ ( Z ) = ̂ MMD 2 k = k ( Z ) ( Z ) ; however, for each permutation σ we would need to re-compute τ ( σ Z ) = ̂ MMD 2 k = k ( σ Z ) ( σ Z ) , so the kernel being used would be different for each permutation. This has two major disadvantages: firstly, it might be computationally expensive to re-compute k for each permutation, especially for a deep kernel 5 . Secondly, the scale of the resulting MMD could be dramatically different for each permutation, so the empirical quantile might not lead to a powerful test. This second problem is related to the problem of combining multiple different MMD values into a single test which our MMD-FUSE statistics are designed to combat (Section 4; c.f. also MMDAgg, Schrab et al., 2023).

Our Proposal. In two-sample testing we propose to use a statistic τ θ parameterised by some θ , where θ is fixed for all permutations, but depends on the data in an unsupervised or permutation-invariant way. Specifically, we allow such a parameter to depend on ⟨ Z ⟩ := { Z 1 , . . . , Z n + m } , the unordered combined sample. Since our tests will not depend on the internal ordering of X and Y (which are assumed i.i.d. under both hypotheses), the only additional information contained in Z over ⟨ Z ⟩ is the label assigning Z i to its initial sample. This is justified since ⟨ Z ⟩ = ⟨ σ Z ⟩ for all σ ∈ S n + m , so setting τ ( Z ) = τ θ ( ⟨ Z ⟩ ) ( Z ) gives a fixed θ and statistic for all permutations to use in Theorem 1. The information in ⟨ Z ⟩ can be used to fine-tune test parameters for any test fitting this setup. This solves both the computation and scaling issues mentioned above.

The above provides a first formal justification for the use of the median heuristic in Gretton et al. (2012a), since it is a permutation-invariant function of the data. However, a far richer set of possibilities are available even when restricting ourselves to these permutation-free functions of Z . For example, we can use any unsupervised or self-supervised learning method to learn representations to use as the input to an MMD-based test statistic, while paying no cost in terms of calibration and needing to train such methods only once. Given the wide variety of methods dedicated to feature extraction and dimensionality reduction, this opens up a huge range of possibilities for the design of new and principled two-sample tests. The simplicity and generality of our proposal might lead one to

expect that this idea has been used before, but it has not to our best knowledge, underlined by the fact that the median heuristic has been widely used without such justification when one follows from this method. This potentially powerful and widely-applicable possibility represents one of our most practical contributions.

4 MMD-FUSE: Fusing U-Statistics by Exponentiation

Say we have computed several MMD values ̂ MMD 2 k under different kernels k ∈ K . How might we combine these? One possibility is to perform multiple testing as in the case of MMDAgg. An even simpler approach would simply take the maximum of those values max k ̂ MMD 2 k since Theorem 1 shows that this will not prevent us from controlling the level of our test by α ; note though that for each permutation we would take this maximum separately. Indeed, Cárcamo et al. (2022) show that for certain kernel choices the supremum of the MMD values with respect to the bandwidth is a valid integral probability metric (Müller, 1997). There are two main problems with this approach: a capacity control issue and a normalisation issue.

Firstly, if the class over which we are maximising is sufficiently rich (for example a complex neural network), then the maximum may be able to effectively memorise the entire sample for each possible permutation, saturating the statistic for every permutation and limiting test power. Any convergence results would need to hold simultaneously for every k in K , and so power results suffer: for finite |K| we would incur a sub-optimal Bonferroni correction (see Section 5); while for infinite classes, results would need to involve capacity control quantities like Rademacher complexity. Further, only information from a single maximising 'base' kernel can be considered at a time.

Therefore, in both our statistics, we prefer a 'soft' maximum, which considers information from every kernel simultaneously and when more than one of the kernels is well-suited (Section 5) it therefore avoids the Bonferroni correction arising from uniform bounds. From Equation (1), our approach is equivalent to using a KL complexity penalty, and is strongly reminiscent of PAC-Bayesian (Section 2) capacity control. We note that other soft maxima could be considered, but our choice makes obtaining exponential concentration inequalities relatively easy (Appendix C), and the dual formulation (Equation (2)) allows us to derive power results in terms of MMD directly.

The second issue is that the MMD estimates might have different scales or variances per kernel which need to be accounted for. In order to be able to meaningfully compare MMD values between each other, these need to be normalised somehow before taking a maximum. We use the common approach of dividing through by a variance-like term (as in 'studentised' tests, see Section 2). The specific normaliser is permutation invariant and gives our statistic tight sub-Gaussian null concentration (for well-chosen regularisation parameter λ ; see Appendix C).

Based on the above we introduce the FUSE-1 statistic which uses a log-sum-exp soft maximum, and the FUSE-N which combines this with normalisation. Both statistics use a 'prior' distribution on K , denoted π ( ⟨ Z ⟩ ) , which is either fixed independently of the data, or is a function of the data which is invariant under permutation (as discussed in Section 3).

Definition 1. We define the un-normalised (subscript 1 for normaliser of 1 ) and normalised (subscript N) test statistics with parameter λ > 0 , respectively, as

where ̂ N k ( Z ) := 1 n ( n -1) ∑ ( i,j ) ∈ [ n + m ] 2 k ( Z i , Z j ) 2 is permutation invariant.

Although these statistics appear complex, we note that in the case where π has finite support, their calculation reduces to a log-sum-exp of MMD estimates normalised by ̂ N k . This is even clearer when π is also uniform on its support, as we consider experimentally; then Equation (2) shows that our statistics reduce to soft maxima, with λ controlling the smoothness or 'temperature'.

From this, we define the FUSE-1 test (with the FUSE-N test ∆ FUSE N defined analogously) as

for sampled set S of permutations as described above. It compares the test statistic ̂ FUSE 1 with its quantiles under permutation, and rejects if the overall value exceeds a quantile controlled by α . Note that since ̂ N k ( Z ) is permutation invariant, it only needs to be calculated once per kernel in ∆ FUSE N (and not separately for each permutation) as per Section 3. See Appendix A.5 for its time complexity.

Comparison with MMDAgg. MMDAggis a different way to think about combining multiple kernels and MMD values, but it relies on a framework based on multiple testing. This can be problematic in the case where the number of kernels considered is large, since as this number increases in the multiple testing approach MMDAgg, the level needs to be corrected differently, so its theretical power behaviour becomes unclear (though empirically MMDAgg retains its power in this setting). By contrast, our proposed FUSE methods avoid multiple testing as a single statistic and quantile are used, avoiding such issues. The FUSE statistics are even defined in the limit of an infinite number of kernels (continuous/uncountable collection of kernels) by considering distributions on the space of kernels, which is not the case for MMDAgg. Moreover, while the MMDAgg approach is only useful for hypothesis testing, having a quantity like FUSE combining multiple kernel-based measures of distance with exponential concentration bounds could be of interest in a wider range of applications.

FUSE-1 and the Mean Kernel. Although the un-normalised test has worse performance in practice than our normalised test, the FUSE-1 statistic is interesting in its own right because of various theoretical properties, as we discuss below. Firstly, we introduce the mean kernel K ρ ( x, y ) = E k ∼ ρ k ( x, y ) under a 'posterior' ρ ∈ M 1 + ( K ) , which is indeed a reproducing kernel in its own right. In the finite case, this is simply a weighted sum of 'base' kernels.

Note that the linearity of ̂ MMD 2 k and MMD 2 k in the kernel k implies that E k ∼ ρ MMD 2 k ( p, q ) = MMD 2 K ρ ( p, q ) , and similarly for ̂ MMD 2 k , with these terms appearing in our power results (Section 5). ̂

Combining this linearity with the dual formulation of FUSE 1 via Equation (1) gives

This re-states ̂ FUSE 1 in terms of 'posterior' ρ , and makes the interpretation of our statistic as a KL-regularised kernel-learning method clear. In the finite case, our test simply optimises the weightings of the different kernels in a constrained way. We note that for certain infinite kernel sets and choices of prior it is be possible to express the mean kernel in closed form. This happens because, e.g. the expectation of a Gaussian kernel with respect to a Gamma prior over the bandwidth is simply a (closed form) rational quadratic kernel. We discuss this point further in Appendix B.

5 Theoretical Power of MMD-FUSE

In this section we outline possible sufficient conditions for our tests to obtain power at least 1 -β at given level α . The conditions will depend on a fixed data-independent 'prior' π ∈ M 1 + ( K ) and thus hold even without unsupervised parameter optimisation. They are stated as requirements for the existence of a 'posterior' ρ ∈ M 1 + ( K ) with corresponding mean kernel K ρ as defined in Section 4. In the finite case, K ρ is simply a weighted sum of kernels, so these requirements are also satisfied under the same conditions for any single kernel, corresponding to the case where the posterior puts all its weight on a single kernel.

Technically, these results require that there is some constant κ upper bounding all of the kernels, and that n and m are within a constant multiple ( i.e. n ≤ m ≤ cn for some c ≥ 1 , notated n ≍ m ). They hold when using randomised permutations provided B > c ′ α -2 log( β -1 ) for small constant c ′ > 0 .

Theorem 2 (FUSE-1 Power) . Fix prior π independently of the data. For the un-normalised test FUSE-1 with λ ≍ n/κ and n ≍ m , there exists a universal constant C > 0 such that

is sufficient for FUSE-1 to achieve power at least 1 -β at level α .

A similar result is obtained for FUSE-N under an assumption that the normalising term is well behaved (see assumption in Theorem 3). This requirement will be satisfied for kernels (including most common ones) that tend to zero only in the limit of the data being infinitely far apart.

Theorem 3 (FUSE-N Power) . Fix prior π independently of the data such that for all k ∈ supp( π ) the expectation E Z ∼ p × q [ ̂ N k ( Z ) -1 ] < c/κ is bounded for some c < ∞ . For the normalised test FUSE-N with λ ≍ n ≍ m , there exists a universal constant C > 0 such that

is sufficient for FUSE-N to achieve power at least 1 -β at level α .

Discussion. The conditions in Theorems 2 and 3 give the optimal MMD 2 separation rate in n (Domingo-Enrich et al., 2023, Proposition 2). These results also imply consistency if the prior π assigns non-zero weight to characteristic kernels. 6 Applying these results to uniform priors supported on r points, the KL penalty can be upper bounded as KL( ρ, π ) ≤ log( r ) . Thus in the worst case, where only a single kernel achieves large MMD 2 k ( p, q ) , the price paid for adaptivity is only log( r ) . In many cases, most of the kernels will give large MMD 2 k ( p, q ) . The posterior will then mirror the prior, and this KL penalty will be even smaller. Thus very large numbers of kernels could be considered, and if all give large MMD values the power would not be greatly affected.

Additional Technical Results. Aside from the presentation of our new statistics and tests, we make a number of technical contributions on the way to proving Theorems 2 and 3, as well as proving some additional results. In particular, we give exponential concentration bounds for our statistics under permutations and the null, which do not require bounded kernels. This refined analysis requires the construction of a coupled Rademacher chaos and concentration thereof. We obtain intermediate results using variances from the proofs of Theorems 2 and 3 that could be used in future work to obtain power guarantees under alternative assumptions such as Sobolev spaces (Schrab et al., 2023). Finally, we prove exponential concentration for ̂ FUSE 1 under the alternative and bounded kernels, requiring the proof of a 'PAC-Bayesian' bounded differences-type concentration inequality. See the appendix for a more detailed overview.

6 Experiments

We compare the test power of MMD-FUSE-N ( λ = √ n ( n -1) ) against various MMD-based kernel selective tests (see Section 1 for details) using: the median heuristic (MMD-Median; Gretton et al., 2012a), data splitting (MMD-Split; Sutherland et al., 2017), analytic Mean Embeddings and Smooth Characteristic Functions (ME & SCF; Jitkrittum et al., 2016), the MMD Deep kernel (MMD-D; Liu et al., 2020), Automated Machine Learning (AutoML; Kübler et al., 2022b), kernel thinning to (Aggregate) Compress Then Test (CTT & ACTT; Domingo-Enrich et al., 2023), and MMD Aggregated (Incomplete) tests (MMDAgg & MMDAggInc; Schrab et al., 2023, 2022b). Additional details and code link for experimental reproducibility are provided in Appendix A.

Distribution on Kernels. We choose our kernel prior distribution π as uniform over a collection of Gaussian, k g γ ( x, y ) = exp ( -∥ x -y ∥ 2 2 / 2 γ 2 ) , and Laplace, k ℓ γ ( x, y ) = exp ( - √ 2 ∥ x -y ∥ 1 / γ ) , kernels with various bandwidths γ > 0 . These bandwidths are chosen as the uniform discretisation of the interval between half the 5% and twice the 95% quantiles (for robustness) of {∥ z -z ′ ∥ r : z, z ′ ∈ Z } , with r ∈ { 1 , 2 } , respectively. This choice is similar to that of Schrab et al. (2023, Section 5.2), who empirically show that ten points for the discretisation is sufficient (Schrab et al., 2023, Figure 6), which we verify in Appendix A.4. This set of distances is permutation-invariant, so Theorem 1 guarantees a well-calibrated test even though the kernels are data-dependent.

Mixture of Gaussians. Our first experiments (Figure 1) consider multimodal distributions p and q , each a 2 -dimensional mixture of four Gaussians with means ( ± µ, ± µ ) with µ = 20 and diagonal covariances. For p , the four components all have unit variances, while for q we vary the standard deviation σ of one of the Gaussians, σ = 1 corresponds to the null hypothesis p = q . Intuitively, an appropriate kernel bandwidth to distinguish p from q would correspond to that separating Gaussians

̸

0.5

1

2

Standard deviation

σ

0

.

0

0

.

5

1

.

0

Power

Mixture of Gaussians

d

= 2

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Perturbation amplitude

a

Perturbed Uniform

d

= 1

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Perturbation amplitude

a

Perturbed Uniform

d

= 2

0.1

0.2

0.3

0.4

Corruption probability

c

Galaxy MNIST

d

=3

×

64

×

64

1000

2000

3000

Sample size

0

.

0

0

.

5

1

.

0

Power

1000

2000

3000

Sample size

MMD-FUSE

MMDAgg

MMDAggInc

ACTT

CTT

ME

SCF

MMD-D

AutoML

MMD-Median

MMD-Split

1000

2000

3000

Sample size

500

1000

1500

2000

2500

Sample size

with standard deviations 1 and σ . This is significantly smaller than the median bandwidth which scales with the distance µ between modes.

Perturbed Uniform. In Figure 1, we report test power for detecting perturbations on uniform distributions in one and two dimensions. We vary the amplitude a of two perturbations from a = 0 (null) to a = 1 (maximum value for the density to remain non-negative). A similar benchmark was first proposed by Schrab et al. (2023, Section 5.5) and considered in several other works (Schrab et al., 2022b; Hagrass et al., 2022; Chatterjee and Bhattacharya, 2023). Different bandwidths are required to detect different amplitudes of the perturbations.

Galaxy MNIST. We examine performance on real-world data in Figure 1, through galaxy images (Walmsley et al., 2022) in dimension d = 3 × 64 × 64 = 12288 captured by a groundbased telescope. These consist of four classes: 'smooth and cigar-shaped', 'edge-on-disk', 'unbarred spiral', and 'smooth and round'. One distribution uniformly samples images from the first three categories, while the other does the same with probability 1 -c and uniformly samples a 'smooth and round' galaxy image with probability of corruption c ∈ [0 , 1] . The null hypothesis corresponds to the case c = 0 .

CIFAR 10 vs 10.1. The aim of this experiment is to detect the difference between images from the CIFAR-10 (Krizhevsky, 2009) and CIFAR-10.1 (Recht et al., 2019) test sets. This is a challenging problem as CIFAR-10.1 was specifically created to consist of new samples from the CIFAR-10 distribution so that it can be used as an alternative test set for models trained on CIFAR-10. Samples from the two distributions are presented in Figure 6 in Appendix A.3 (Liu et al., 2020, Figure 5). This

benchmark was proposed by Liu et al. (2020, Table 3) who introduced the deep MMD test MMD-D and the MMD-Split test (here referred to as MMD-O to point out that their implementation has been used rather than ours). They also compare to ME and SCF, as well as to C2ST-L and C2ST-S (Lopez-Paz and Oquab, 2017) which correspond to Classifier Two-Sample Tests based on Sign or

Linear kernels. For the tests slitting the data, 1000 images from both datasets are used for parameter selection and/or model training, and 1021 other images from each distributions are used for testing. Consequently, tests avoiding data splitting are given the full 2021 images from CIFAR-10.1 and 2021 images sampled from CIFAR-10.

Experimental Results of Figure 1. We observe similar trends in all eight experiments in Figure 1: MMD-FUSE matches the power of state-of-the-art MMDAgg while being computationally faster (both theoretically and practically). These two tests consistently obtain the highest power in every experiment, except when increasing the number of Galaxy MNIST images where MMD-D obtains higher power. However, we observe in Table 1 that MMD-FUSE outperforms MMD-D on another image data problem, also with large sample size. On synthetic data, the deep kernel test MMD-D surprisingly only obtains power similar to MMD-Split in most experiments (even lower in the 2dimensional perturbed uniform experiment). The two near-linear aggregated variants ACTT and MMDAggInc trade-off a small portion of test power for computational efficiency, with the former outperforming the latter for large sample sizes. The importance of kernel selection is emphasised by the fact that the two tests using the median bandwidth (MMD-Median and CTT) achieve very low power. While the linear-time SCF test, based in the frequency domain, attains high power in the Mixture of Gaussians experiments, it has low power in the three other experiments. Its spatial domain variant ME performs better on Perturbed Uniform d ∈ { 1 , 2 } experiments but in general still has reduced power compared to both linear and quadratic time alternatives. Finally, the AutoML test performs well for fixed sample size m = n = 500 (first row of Figure 1), but its power compared to other tests considerably deteriorates as the sample size increases (second row of Figure 1). Overall, MMD-FUSE achieves state-of-the-art performance across all experiments on both low-dimensional synthetic data and high-dimensional real-world data.

Experimental Results of Table 1. We report in Table 1 the power achieved by each test on the CIFAR 10 vs 10.1 experiment, which is averaged over 1000 repetitions. We observe that MMD-FUSE performs the best and obtains power 0 . 937 , which means that out of 1000 repetitions, it was 937 times able to distinguish between samples from CIFAR-10 and from CIFAR-10.1. This demonstrates that the images in CIFAR-10.1 do not come from the same distribution as those in CIFAR-10.

Experimental Results of Figure 7. We observe in Figure 7 of Appendix A.4 that MMD-FUSE can achieve higher power than MMDAgg in some additional perturbed uniform experiment. We also note that using a relatively small number of kernels ( e.g. , 10 ) is enough to capture all the required information, and that the test power is retained when further increasing the number of kernels.

7 Conclusions

In this work, we propose MMD-FUSE, an MMD-based test which fuses kernels through a soft maximum and a method for learning general two-sample testing parameters in an unsupervised fashion. We demonstrate the empirical performance of MMD-FUSE and show that it achieves the optimal MMD separation rate guaranteeing high test power. This optimality holds with respect to the sample size and likely also for the logarithmic dependence in α , but we believe the dependence on β could be improved in future work; a general question is whether lower bounds in terms of α and β can be proved. Obtaining separation rates in terms of the L 2 -norm between the densities (Schrab et al., 2023) may also be possible but challenging since this distance is independent of the kernel.

An open question is in explaining the significant empirical power advantage of the normalised test over its un-normalised variant, which is currently not reflected in the derived rates. The importance of this normalisation is clear when considering kernels with different bandwidths, leading to vastly different scaling in the un-normalised permutation distributions. Work here could begin with finitesample concentration guarantees for our normalised statistic or other 'studentised' variants, some of which might obtain better performance.

Future work could also examine computationally efficient variants of MMD-FUSE, by either relying on incomplete U -statistics (Schrab et al., 2022b) and leading to suboptimal rates, or by relying on recent ideas of kernel thinning (Domingo-Enrich et al., 2023) which can lead to the same optimal rate under stronger assumptions on the data distributions, or by considering the permutation-free approach of Shekhar et al. (2022). Finally, our two-sample MMD fusing approach could be extended to the HSIC independence framework (Gretton et al., 2005, 2008; Albert et al., 2022) and to the KSD goodness-of-fit setting (Chwialkowski et al., 2016; Liu et al., 2016; Schrab et al., 2022a).

References

Mélisande Albert, Béatrice Laurent, Amandine Marrel, and Anouar Meynaoui. Adaptive test of independence based on HSIC measures. The Annals of Statistics , 50(2):858-879, 2022. (Cited on pages 4 and 10.)

Pierre Alquier. User-friendly introduction to PAC-Bayes bounds. CoRR , abs/2110.11216, 2021. URL https://arxiv.org/abs/2110.11216 . (Cited on page 4.)

Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society , 68(3):337-404, 1950. (Cited on page 3.)

Gregory Benton, Wesley J Maddox, Jayson Salkey, Julio Albinati, and Andrew Gordon Wilson. Function-space distributions over kernels. In H. Wallach, H. Larochelle, A. Beygelzimer, F. AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_ files/paper/2019/file/39d929972619274cc9066307f707d002-Paper.pdf . (Cited on page 3.)

Felix Biggs and Benjamin Guedj. Differentiable PAC-Bayes objectives with partially aggregated neural networks. Entropy , 23(10):1280, 2021. doi: 10.3390/e23101280. URL https://doi. org/10.3390/e23101280 . (Cited on page 4.)

Felix Biggs and Benjamin Guedj. Non-vacuous generalisation bounds for shallow neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 1963-1981. PMLR, 2022a. URL https://proceedings.mlr.press/v162/biggs22a.html . (Cited on page 4.)

Felix Biggs and Benjamin Guedj. On margins and derandomisation in PAC-Bayes. In Gustau CampsValls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 3709-3731. PMLR, 28-30 Mar 2022b. URL https://proceedings. mlr.press/v151/biggs22a.html . (Cited on page 4.)

Felix Biggs and Benjamin Guedj. Tighter pac-bayes generalisation bounds by leveraging example difficulty. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 8165-8182. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/biggs23a.html . (Cited on page 4.)

Felix Biggs, Valentina Zantedeschi, and Benjamin Guedj. On margins and generalisation for voting classifiers. In NeurIPS , 2022. doi: 10.48550/arXiv.2206.04607. URL https://doi.org/10. 48550/arXiv.2206.04607 . (Cited on page 4.)

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence . Oxford university press, 2013. (Cited on page 31.)

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax . (Cited on page 18.)

Javier Cárcamo, Antonio Cuevas, and Luis-Alberto Rodríguez. A uniform kernel trick for highdimensional two-sample problems. arXiv preprint arXiv:2210.02171 , 2022. (Cited on page 6.)

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle,

Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. (Cited on page 2.)

Olivier Catoni. PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning . Institute of Mathematical Statistics lecture notes-monograph series. Institute of Mathematical Statistics, 2007. ISBN 9780940600720. URL https://books.google.fr/books?id= acnaAAAAMAAJ . (Cited on page 4.)

Anirban Chatterjee and Bhaswar B. Bhattacharya. Boosting the power of kernel two-sample tests. arXiv preprint arXiv:2302.10687 , 2023. (Cited on pages 2, 5, and 9.)

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 1597-1607. PMLR, 2020a. (Cited on page 2.)

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020b. (Cited on page 2.)

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 15750-15758. Computer Vision Foundation / IEEE, 2021. (Cited on page 2.)

Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR , abs/2003.04297, 2020c. URL https://arxiv.org/abs/2003. 04297 . (Cited on page 2.)

Farah Cherfaoui, Hachem Kadri, Sandrine Anthoine, and Liva Ralaivola. A discrete RKHS standpoint for Nyström MMD. HAL preprint hal-03651849 , 2022. (Cited on page 5.)

Ben Chugg, Hongjian Wang, and Aaditya Ramdas. A unified recipe for deriving (time-uniform) pac-bayes bounds. CoRR , abs/2302.03421, 2023. doi: 10.48550/arXiv.2302.03421. URL https: //doi.org/10.48550/arXiv.2302.03421 . (Cited on page 4.)

Kacper Chwialkowski, Dino Sejdinovic, and Arthur Gretton. A wild bootstrap for degenerate kernel tests. In Advances in neural information processing systems , pages 3608-3616, 2014. (Cited on page 4.)

Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In International Conference on Machine Learning , pages 2606-2615. PMLR, 2016. (Cited on page 10.)

Kacper P Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. In Advances in Neural Information Processing Systems , volume 28, pages 1981-1989, 2015. (Cited on pages 2 and 5.)

Namrata Deka and Danica J. Sutherland. Mmd-b-fair: Learning fair representations with statistical testing. CoRR , abs/2211.07907, 2022. doi: 10.48550/arXiv.2211.07907. URL https://doi. org/10.48550/arXiv.2211.07907 . (Cited on page 5.)

Carles Domingo-Enrich, Raaz Dwivedi, and Lester Mackey. Compress then test: Powerful kernel testing in near-linear time. The 26th International Conference on Artificial Intelligence and Statistics, AISTATS 2023 , 2023. (Cited on pages 2, 5, 8, 10, and 18.)

D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics , 28(1):1-47, 1975. doi: https: //doi.org/10.1002/cpa.3160280102. URL https://onlinelibrary.wiley.com/doi/abs/10. 1002/cpa.3160280102 . (Cited on page 3.)

Dvoretzky, J. Kiefer, and J. Wolfowitz. Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator. The Annals of Mathematical Statistics , 27 (3):642 - 669, 1956. doi: 10.1214/aoms/1177728174. URL https://doi.org/10.1214/aoms/ 1177728174 . (Cited on page 34.)

Raaz Dwivedi and Lester Mackey. Kernel thinning. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado,

USA , volume 134 of Proceedings of Machine Learning Research , page 1753. PMLR, 2021. URL http://proceedings.mlr.press/v134/dwivedi21a.html . (Cited on page 5.)

Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Conference on Uncertainty in Artificial Intelligence 33. , 2017. (Cited on page 4.)

Gintare Karolina Dziugaite and Daniel M. Roy. Data-dependent PAC-Bayes priors via differential privacy. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò CesaBianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 8440-8450, 2018. URL https://proceedings.neurips. cc/paper/2018/hash/9a0ee0a9e7a42d2d69b8f86b3a0756b1-Abstract.html . (Cited on page 4.)

Gintare Karolina Dziugaite, Kyle Hsu, Waseem Gharbieh, Gabriel Arpino, and Daniel Roy. On the role of data in PAC-Bayes. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 604612. PMLR, 2021. URL http://proceedings.mlr.press/v130/karolina-dziugaite21a. html . (Cited on page 4.)

Brück Florian, Jean-David Fermanian, and Aleksey Min. Distribution free mmd tests for model selection with estimated parameters. arXiv preprint arXiv:2305.07549 , 2023. (Cited on page 5.)

Magalie Fromont, Béatrice Laurent, Matthieu Lerasle, and Patricia Reynaud-Bouret. Kernels based tests with non-asymptotic bootstrap approaches for two-sample problems. In Conference on Learning Theory , volume 23 of Journal of Machine Learning Research Proceedings , 2012. (Cited on page 4.)

Magalie Fromont, Béatrice Laurent, and Patricia Reynaud-Bouret. The two-sample problem for Poisson processes: Adaptive tests with a nonasymptotic wild bootstrap approach. The Annals of Statistics , 41(3):1431-1461, 2013. (Cited on pages 4 and 5.)

Hanjia Gao and Xiaofeng Shao. Two sample testing in high dimension via maximum mean discrepancy. arXiv preprint arXiv:2109.14913 , 2022. (Cited on page 5.)

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In International Conference on Algorithmic Learning Theory . Springer, 2005. (Cited on page 10.)

Arthur Gretton, Kenji Fukumizu, Choon H Teo, Le Song, Bernhard Schölkopf, and Alex J Smola. A kernel statistical test of independence. In Advances in Neural Information Processing Systems , volume 1, pages 585-592, 2008. (Cited on page 10.)

Arthur Gretton, Kenji Fukumizu, Zaid Harchaoui, and Bharath K Sriperumbudur. A fast, consistent kernel two-sample test. Advances in Neural Information Processing Systems , 22, 2009. (Cited on page 5.)

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel two-sample test. J. Mach. Learn. Res. , 13:723-773, 2012a. doi: 10.5555/2503308. 2188410. URL https://dl.acm.org/doi/10.5555/2503308.2188410 . (Cited on pages 1, 5, 8, and 18.)

Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems , volume 1, pages 1205-1213, 2012b. (Cited on pages 1, 2, and 5.)

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. (Cited on page 2.)

Benjamin Guedj. A primer on PAC-Bayesian learning. In Proceedings of the second congress of the French Mathematical Society , volume 33, 2019. URL https://arxiv.org/abs/1901.05353 . (Cited on page 4.)

Maxime Haddouche and Benjamin Guedj. Online pac-bayes learning. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ a4d991d581accd2955a1e1928f4e6965-Abstract-Conference.html . (Cited on page 4.)

Maxime Haddouche and Benjamin Guedj. Pac-bayes generalisation bounds for heavy-tailed losses through supermartingales. Trans. Mach. Learn. Res. , 2023, 2023. URL https://openreview. net/forum?id=qxrwt6F3sf . (Cited on page 4.)

Omar Hagrass, Bharath K. Sriperumbudur, and Bing Li. Spectral regularized kernel two-sample tests. arXiv preprint arXiv:2212.09201 , 2022. (Cited on pages 2 and 9.)

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 9726-9735. Computer Vision Foundation / IEEE, 2020. (Cited on page 2.)

Jesse Hemerik and Jelle Goeman. Exact testing with random permutations. TEST , 27(4):811-825, Dec 2018. ISSN 1863-8260. doi: 10.1007/s11749-017-0571-1. URL https://doi.org/10. 1007/s11749-017-0571-1 . (Cited on page 4.)

Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science , 313(5786):504-507, 2006. (Cited on page 2.)

Wittawat Jitkrittum, Zoltán Szabó, Kacper P Chwialkowski, and Arthur Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems , volume 29, pages 181-189, 2016. (Cited on pages 2, 5, 8, and 18.)

Ilmun Kim and Aaditya Ramdas. Dimension-agnostic inference using cross u-statistics. 2023. (Cited on page 5.)

Ilmun Kim, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimality of permutation tests. Annals of Statistics , 50(1):225-251, February 2022. ISSN 0090-5364. doi: 10.1214/21-AOS2103. Funding Information: Funding. This work was partially supported by the NSF Grant DMS17130003 and EP-SRC Grant EP/N031938/1. Publisher Copyright: © Institute of Mathematical Statistics, 2022. (Cited on pages 3, 5, 27, 34, and 35.)

Matthias Kirchler, Shahryar Khorasani, Marius Kloft, and Christoph Lippert. Two-sample testing using deep learning. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pages 1387-1398. PMLR, 2020. URL http://proceedings.mlr.press/v108/kirchler20a.html . (Cited on page 5.)

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. (Cited on pages 9 and 21.)

M. Kübler, W. Jitkrittum, B. Schölkopf, and K. Muandet. Learning kernel tests without data splitting. In Advances in Neural Information Processing Systems 33 , pages 6245-6255. Curran Associates, Inc., 2020. (Cited on pages 2 and 5.)

Jonas M Kübler, Wittawat Jitkrittum, Bernhard Schölkopf, and Krikamol Muandet. A witness twosample test. In International Conference on Artificial Intelligence and Statistics , pages 1403-1419. PMLR, 2022a. (Cited on page 2.)

Jonas M Kübler, Vincent Stimper, Simon Buchholz, Krikamol Muandet, and Bernhard Schölkopf. AutoML two-sample test. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 , 2022b. (Cited on pages 2, 8, and 18.)

Alexandre Lacasse, François Laviolette, Mario Marchand, Pascal Germain, and Nicolas Usunier. PAC-Bayes bounds for the risk of the majority vote and the variance of the Gibbs classifier. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann, editors, Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006 , pages 769-776. MIT Press, 2006. URL https://proceedings.neurips.cc/paper/2006/hash/ 779efbd24d5a7e37ce8dc93e7c04d572-Abstract.html . (Cited on page 4.)

Alexandre Lacasse, François Laviolette, Mario Marchand, and Francis Turgeon-Boutin. Learning with randomized majority votes. In José L. Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag, editors, Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part II , volume 6322 of Lecture Notes in Computer Science , pages 162-177. Springer, 2010. doi: 10.1007/ 978-3-642-15883-4\_11. URL https://doi.org/10.1007/978-3-642-15883-4_11 . (Cited on page 4.)

Justin Lee. U -statistics: Theory and Practice . Citeseer, 1990. (Cited on page 35.)

Guy Lever, François Laviolette, and John Shawe-Taylor. Tighter PAC-Bayes bounds through distribution-dependent priors. Theoretical Computer Science , 473:4-28, February 2013. ISSN 0304-3975. doi: 10.1016/j.tcs.2012.10.013. URL https://linkinghub.elsevier.com/ retrieve/pii/S0304397512009346 . (Cited on page 4.)

Tong Li and Ming Yuan. On the optimality of gaussian kernel based nonparametric tests against smooth alternatives. arXiv preprint arXiv:1909.03302 , 2019. (Cited on page 5.)

Yazhe Li, Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 15543-15556, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 83004190b1793d7aa15f8d0d49a13eba-Abstract.html . (Cited on page 2.)

Jen Ning Lim, Makoto Yamada, Wittawat Jitkrittum, Yoshikazu Terada, Shigeyuki Matsui, and Hidetoshi Shimodaira. More powerful selective kernel tests for feature selection. In International Conference on Artificial Intelligence and Statistics , pages 820-830. PMLR, 2020. (Cited on page 5.)

Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Dougal J Sutherland. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning , 2020. (Cited on pages 2, 5, 8, 9, 18, and 21.)

Feng Liu, Wenkai Xu, Jie Lu, and Danica J. Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 5848-5860, 2021. URL https://proceedings.neurips. cc/paper/2021/hash/2e6d9c6052e99fcdfa61d9b9da273ca2-Abstract.html . (Cited on page 2.)

Qiang Liu, Jason Lee, and Michael Jordan. A kernelized Stein discrepancy for goodness-of-fit tests. In International Conference on Machine Learning , pages 276-284. PMLR, 2016. (Cited on page 10.)

David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id= SJkXfE5xx . (Cited on page 9.)

Andrés R. Masegosa, Stephan Sloth Lorenzen, Christian Igel, and Yevgeny Seldin. Second order PAC-Bayesian bounds for the weighted majority vote. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/386854131f58a556343e056f03626e00-Abstract.html . (Cited on page 4.)

Massart. The Tight Constant in the Dvoretzky-Kiefer-Wolfowitz Inequality. The Annals of Probability , 18(3):1269 - 1283, 1990. doi: 10.1214/aop/1176990746. URL https://doi.org/ 10.1214/aop/1176990746 . (Cited on page 34.)

Andreas Maurer. A note on the PAC-Bayesian theorem. CoRR , cs.LG/0411099, 2004. URL http://arxiv.org/abs/cs.LG/0411099 . (Cited on page 4.)

David A McAllester. Some PAC-Bayesian theorems. In Proceedings of the eleventh annual conference on Computational Learning Theory , pages 230-234. ACM, 1998. (Cited on page 4.)

Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability , 1:429-443, 1997. (Cited on page 6.)

Maria Perez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvari. Tighter risk certificates for neural networks. Journal of Machine Learning Research , 22(227):1-40, 2021. URL http://jmlr.org/papers/v22/20-879.html . (Cited on page 4.)

Aaditya Ramdas, Sashank Jakkam Reddi, Barnabás Póczos, Aarti Singh, and Larry Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015. (Cited on page 2.)

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5389-5400. PMLR, 09-15 Jun 2019. URL https://proceedings. mlr.press/v97/recht19a.html . (Cited on pages 9 and 21.)

Sashank Reddi, Aaditya Ramdas, Barnabás Póczos, Aarti Singh, and Larry Wasserman. On the high dimensional power of a linear-time two sample test under mean-shift alternatives. In Artificial Intelligence and Statistics , pages 772-780. PMLR, 2015. (Cited on page 2.)

David Rindt, Dino Sejdinovic, and David Steinsaltz. Consistency of permutation tests of independence using distance covariance, hsic and dhsic. Stat , 10(1):e364, 2021. doi: https://doi.org/10.1002/ sta4.364. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/sta4.364 . e364 sta4.364. (Cited on pages 4 and 19.)

Joseph P Romano and Michael Wolf. Exact and approximate stepdown methods for multiple hypothesis testing. Journal of the American Statistical Association , 100(469):94-108, 2005. (Cited on page 4.)

Mark Rudelson and Roman Vershynin. Hanson-Wright inequality and sub-gaussian concentration. Electronic Communications in Probability , 18(none):1 - 9, 2013. doi: 10.1214/ECP.v18-2865. URL https://doi.org/10.1214/ECP.v18-2865 . (Cited on page 25.)

Antonin Schrab, Benjamin Guedj, and Arthur Gretton. KSD aggregated goodness-of-fit test. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 , 2022a. (Cited on page 10.)

Antonin Schrab, Ilmun Kim, Benjamin Guedj, and Arthur Gretton. Efficient aggregated kernel tests using incomplete U -statistics. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 , 2022b. (Cited on pages 2, 5, 8, 9, 10, and 18.)

Antonin Schrab, Ilmun Kim, Mélisande Albert, Béatrice Laurent, Benjamin Guedj, and Arthur Gretton. MMD aggregated two-sample test. Journal of Machine Learning Research , 24(194):1-81, 2023. URL http://jmlr.org/papers/v24/21-1289.html . (Cited on pages 2, 4, 5, 8, 9, 10, 18, 22, and 35.)

Matthias Seeger, John Langford, and Nimrod Megiddo. An improved predictive accuracy bound for averaging classifiers. In Proceedings of the 18th International Conference on Machine Learning , number CONF, pages 290-297, 2001. (Cited on page 4.)

Yevgeny Seldin, François Laviolette, Nicolò Cesa-Bianchi, John Shawe-Taylor, and Peter Auer. PAC-Bayesian inequalities for martingales. In Nando de Freitas and Kevin P. Murphy, editors, Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, August 14-18, 2012 , page 12. AUAI Press, 2012. URL https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2& article_id=2341&proceeding_id=28 . (Cited on page 4.)

Shubhanshu Shekhar, Ilmun Kim, and Aaditya Ramdas. A permutation-free kernel two-sample test. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 , 2022. (Cited on pages 5 and 10.)

Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research , 12(7), 2011. (Cited on page 3.)

Danica J Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations , 2017. (Cited on pages 1, 2, 8, and 18.)

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2019. doi: 10.1017/9781108627771. (Cited on page 32.)

Mike Walmsley, Chris J. Lintott, Tobias Géron, Sandor Kruk, Coleman Krawczyk, Kyle W. Willett, Steven Bamford, William Keel, Lee S. Kelvin, Lucy Fortson, Karen L. Masters, Vihang Mehta, Brooke D. Simmons, Rebecca J. Smethurst, Elisabeth M. Baeten, and Christine Macmillan. Galaxy zoo decals: Detailed visual morphology measurements from volunteers and deep learning for 314, 000 galaxies. Monthly Notices of the Royal Astronomical Society , 509:3966-3988, 2022. (Cited on pages 9 and 20.)

Yi-Shan Wu, Andrés R. Masegosa, Stephan Sloth Lorenzen, Christian Igel, and Yevgeny Seldin. Chebyshev-Cantelli PAC-Bayes-Bennett inequality for the weighted majority vote. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 12625-12636, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 69386f6bb1dfed68692a24c8686939b9-Abstract.html . (Cited on page 4.)

Makoto Yamada, Denny Wu, Yao-Hung Hubert Tsai, Hirofumi Ohta, Ruslan Salakhutdinov, Ichiro Takeuchi, and Kenji Fukumizu. Post selection inference with incomplete maximum mean discrepancy estimator. In International Conference on Learning Representations , 2019. (Cited on page 5.)

Valentina Zantedeschi, Paul Viallard, Emilie Morvant, Rémi Emonet, Amaury Habrard, Pascal Germain, and Benjamin Guedj. Learning stochastic majority votes by minimizing a PAC-Bayes generalization bound. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 455-467, 2021. URL https://proceedings.neurips.cc/paper/ 2021/hash/0415740eaa4d9decbc8da001d3fd805f-Abstract.html . (Cited on page 4.)

Wojciech Zaremba, Arthur Gretton, and Matthew Blaschko. B-test: A non-parametric, low variance kernel two-sample test. Advances in neural information processing systems , 26, 2013. (Cited on page 5.)

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 12310-12320. PMLR, 2021. URL http://proceedings.mlr.press/v139/zbontar21a.html . (Cited on page 2.)

Ji Zhao and Deyu Meng. FastMMD: Ensemble of circular discrepancy for efficient two-sample test. Neural computation , 27(6):1345-1372, 2015. (Cited on page 5.)

Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id= BJgqqsAct7 . (Cited on page 4.)

Overview of Appendices

Appendix A We give further experimental details, discuss time complexity of our results while graphing run-times.

Appendix B We discuss how ̂ FUSE 1 can be expressed in a simple form for certain uncountable priors.

Appendix C We prove exponential concentration bounds for both our statistics under the null and permutations.

Appendix D We prove exponential concentration bounds for the ̂ FUSE 1 statistic under the alternative hypothesis.

Appendix E We prove the power results stated in Section 5 and other interesting intermediate results.

A Additional Experimental Details

A.1 Code and licenses

Our implementation of MMD-FUSE in Jax (Bradbury et al., 2018), as well as the code for the reproducibility of the experiments, are made publicly available at:

Our code is under the MIT License, and we also implement ourselves the MMD-Median (Gretton et al., 2012a) and MMD-Split (Sutherland et al., 2017) tests. For ME & SCF (Jitkrittum et al., 2016), MMD-D (Liu et al., 2020), AutoML (Kübler et al., 2022b), CTT & ACTT (Domingo-Enrich et al., 2023), MMDAgg & MMDAggInc (Schrab et al., 2023, 2022b), we use the implementations of the respective authors, which are all under the MIT license.

The experiments were run on an AMD Ryzen Threadripper 3960X 24 Cores 128Gb RAM CPU at 3.8GHz and on an NVIDIA RTX A5000 24Gb Graphics Card, with a compute time of a couple of hours.

A.2 Test parameters

In general, we use the default test parameters recommended by the authors of the tests (listed above). For the ME and SCF tests, ten test locations are chosen on half of the data. AutoML is run with the recommended training time limit of one minute.

Kernels. As explained in Section 6 (§Distribution on Kernels), for MMD-Fuse, we use Gaussian and Laplace kernels with bandwidths in { q r 5% + i ( q r 95% -q r 5% ) / 9 : i = 0 , . . . , 9 } where q r 5% is half the 5% quantile of all the inter-sample distances {∥ z -z ′ ∥ r : z, z ′ ∈ Z } with r = 1 and r = 2 for Laplace and Gaussian kernels, respectively. Similarly, q r 95% is twice the 95% quantile.

MMD-Split selects a Gaussian kernel on half of the data with bandwidth in { q r 5% + i ( q r 95% -q r 5% ) / 99 : i = 0 , . . . , 99 } by maximizing a proxy for asymptotic power which is the ratio of the estimated MMDwith its estimated standard deviation under the alternative (Liu et al., 2020, Equation 3). The MMDtest is then run on the other half with the selected kernel.

CTT and MMD-Median both use a Gaussian kernel with bandwidth the median of {∥ z -z ′ ∥ 2 : z, z ′ ∈ Z } , while ACTT is run with Gaussian kernels with bandwidths in { q 2 5% + i ( q 2 95% -q 2 5% ) / 9 : i = 0 , . . . , 9 } .

The MMDAgg and MMDAggInc tests are run with their default implementations, which similarly use collections of 20 kernels split equally between Gaussian and Laplace kernels with ten bandwidths each, but they use a different (non-uniform) discretisation of the intervals [ q r 95% , q r 5% ] .

Permutations. For MMD-Median, MMD-Split, and MMD-FUSE, we use 2000 permutations to estimate the quantiles. MMDAgg and MMDAggInc use 2000 + 2000 and 500 + 500 permutations, respectively, to approximate the quantiles and the multiple testing correction. The CTT and ACTT tests are run with 39 and 299 + 200 permutations, respectively. AutoML uses 10000 permutations

and MMD-D 100 of them. The ME and SCF tests use asymptotic quantiles. We recall that using a higher number of permutations for the quantile does not necessarily lead to a more powerful test (Rindt et al., 2021, Section 7).

A.3 Details on the experiments of Section 6

In this section, we present figures illustrating the four experimental settings described in Section 6 with results in Figure 1: Mixture of Gaussians (Figure 2), Perturbed Uniform d = 1 (Figure 3), Perturbed Uniform d = 2 (Figure 4), Galaxy MNIST (Figure 5), and CIFAR 10 vs 10.1 (Figure 6).

-

20

0

20

x

-

20

0

20

x

2

1

0

.

0

0

.

5

1

.

0

x

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

y

a

= 0

a

= 0

.

1

a

= 0

.

2

a

= 0

.

3

a

= 0

.

4

a

= 0

.

5

0

0.5

1

x

1

0

0.5

1

x

2

a

= 1

0

0.5

1

x

1

a

= 0

.

6

0

0.5

1

x

1

a

= 0

.

2

0

1

2

T.

(b) CIFAR-10.1 images

A.4 Experiment varying the number of kernels

In Figure 7, we provide an additional perturbed uniform experiment where we vary the number of kernels for MMD-FUSE and for MMDAgg. We consider the task of detecting six one-dimensional perturbations with amplitude a = 0 . 5 for sample sizes m = n = 500 while varying the number of Gaussian and Laplace kernels from 10 to 1000 .

Firstly, we observe that MMD-FUSE achieves higher power than MMDAgg in this setting. Secondly, both MMD-FUSE and MMDAgg retain their power when increasing the number of bandwidths. This matches the empirical observation of Schrab et al. (2023, Section 5.7) that a discretisation of 10 bandwidths is enough to capture all the information in certain two-sample problems and that using a finer discretisation (which is computationally more expensive) does not improve the test power.

10

200

400

600

800

1000

Number of kernels

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Power

MMD-FUSE

MMDAgg

A.5 Time complexity and runtimes

The time complexity of MMD-FUSE is

where m and n are the sizes of the two samples, K is the number of kernels fused, and B is the number of permutations to estimate the quantile. Note that this is an improvement over the time complexity of MMDAgg (Schrab et al., 2023)

where the extra parameter B ′ corresponds to the number of permutations used to estimate the multiple testing correction, often set as B ′ = B in practice (Schrab et al., 2023, Section 5.2). We indeed observe in Figure 8 that MMD-FUSE runs twice as fast as MMDAgg. 7

While the runtimes of most tests should not depend on the type of data (only on the sample size and on the dimension), we note that the runtimes of tests relying on optimisation ( e.g. ME & SCF) can be affected. In the experiment of Figure 8, we consider samples from multivariate Gaussians centred at zero with covariance matrices I d and σI d with σ = 1 . 1 . We vary both the sample sizes and the dimensions.

250

500

1000

2000

4000

Sample size

10

-

2

10

-

1

10

0

10

1

10

2

Time (seconds)

1

10

100

1000

10000

Dimension

10

-

2

10

-

1

10

0

10

1

10

2

Time (seconds)

MMD-FUSE

MMDAgg

MMDAggInc

ACTT

CTT

ME

SCF

MMD-D

AutoML

MMD-Median

MMD-Split

B Closed Form Mean Kernel

We have observed in the main text that in some cases where the prior has non-finite support, ̂ FUSE 1 still gives a straightforwardly expressed statistic. This happens because for certain kernel choices, the expectation of the kernel with respect to some parameter is still a closed form kernel. We give one example of such a statistic here.

Theorem 4. For any fixed α > 0 , the following is an example of a ̂ FUSE 1 statistic:

where k rq ( α,η ) = (1 + ∥ x -y ∥ 2 / 2 η 2 ) -α is a rational quadratic kernel, and η 0 is the 'prior' bandwidth (which can essentially be absorbed into the data scaling).

Proof of Theorem 4. Firstly, we define a general rational quadratic kernel

Note that k g ( r ) = e -τr 2 / 2 with r = ∥ x -y ∥ 2 is a bounded kernel with parameter τ . If we take the expectation of τ with respect to a Gamma distribution,

where Γ denotes the gamma function. The KL divergence between Gamma distributions is

where R = β/β 0 and ψ ( α ) := Γ ′ ( α ) / Γ( α ) denotes the digamma function. Using the dual form of ̂ FUSE 1 we find that

Restricting α = α 0 , prior β 0 = η 2 0 and setting β = Rη 2 0 gives the result.

C Null and Permutation Concentration Results

In this section we prove the following concentration results under the null (or equivalently, under the permutation distribution). These results show that for appropriate choices of λ , both statistics converge to zero at rate at least O (1 /n ) under the null or permutations. We recall that without loss of generality n ≤ m .

Theorem 5. For bounded kernels k ≤ κ < ∞ and parameter λ , with probability at least 1 -δ over a sample Z from the null,

provided 0 < λ < √ n ( n -1) / 8 √ 2 κ , and

For potentially unbounded kernels and parameter λ , with probability at least 1 -δ over a sample Z from the null,

provided 0 < λ < √ n ( n -1) / 16 √ 2 , and

The above bounds are also valid for ̂ FUSE 1 ( σ Z ) , ̂ FUSE N ( σ Z ) and any fixed Z (potentially nonnull) under permutation by σ ∼ Uniform( S n + m ) .

While the upper bounds depend critically upon our choice of λ , the lower bounds instead hold regardless of λ . Choosing λ ≍ √ n ( n -1) ≍ n gives the desired upper bound rates O (1 /n ) .

C.1 Sub-Gaussian Chaos Theorem

We provided constants for Theorem 5, even though they are not strictly necessary, as we hope they could be useful when adapting FUSE statistics to other settings. In particular, the constants help understand for which values of λ the bounds hold, and to understand the relation between the two different right hand side terms which can be matched by tuning λ . In provide these constants, in this subsection we replicate the proof of a sub-result from Rudelson and Vershynin (2013), but unlike in that paper we explicitly keep track of the constants. To be clear, Rudelson and Vershynin (2013) does not give numerical constants at all (only their existence is proved), and we need then for our statement of Theorem 5. We do not claim this as a contribution, the proof is only included so that we are not stating the numerical constants without justification.

Theorem 6 (Sub-Gaussian Chaos; adapted from Rudelson and Vershynin, 2013) . Let X i be meanzero 1 -sub-Gaussian variables, such that log E X exp( tX ) ≤ 1 2 t 2 for every t ∈ R . Let A ∈ R n × n be a real symmetric matrix with zeros on the diagonal and we define the sub-Gaussian chaos

For all | t | ≤ (4 √ 2 ∥ A ∥ ) -1 ,

Proof. This proof is closely adapted from Rudelson and Vershynin (2013) with some modifications to explicitly track numerical constants.

Let X i be mean-zero 1 -sub-Gaussian variables, such that log E X exp( tX ) ≤ 1 2 t 2 for every t ∈ R . We are considering

where A ij has zeros on the diagonal. We introduce independent Bernoulli random variables δ i ∈ { 0 , 1 } with E δ i = 1 / 2 and define the matrix A δ with entries ( A δ ) ij = δ i (1 -δ j ) A ij . Note that E δ A δ = 1 4 A , since E δ i (1 -δ j ) equals 1 / 4 for i = j .

̸

Writing the set of indices Λ δ = { i ∈ [ n ] : δ i = 1 } , we introduce

By Jensen's inequality,

Conditioned on δ and ( X i ) i ∈ Λ δ , W δ is a linear combination of the mean-zero sub-gaussian random variables X j , j ∈ Λ c δ . It follows that this conditional distribution of W δ is sub-gaussian, and so

Now introduce g = ( g 1 , . . . , g n ) ∼ N (0 , 1) i.i.d. draws from a normal and note that the above can also arise directly as the moment generating function of these Normal variables, so we make the further equality (for fixed X and δ ),

Rearranging the terms, and using that the resulting term is a linear combination of the sub-Gaussian X i , i ∈ Λ δ , we find that

Now, by the rotation invariance of the distribution of g , the random variable ∥ A δ g ∥ 2 2 is distributed identically with ∑ i s 2 i g 2 i where s i denote the singular values of A δ , so by independence

where the last inequality holds if 8 t 2 max i s 2 i ≤ 1 4 and arises since the MGF of a Chi-squared variable, E e tg 2 ≤ e 2 t for 0 ≤ t ≤ 1 4 .

Since max i s i = ∥ A δ ∥ ≤ ∥ A ∥ and ∑ i s 2 i = ∥ A δ ∥ 2 F ≤ ∥ A ∥ F , we combine the above steps to find that

C.2 Permutation Bounds for Two-Sample U-Statistics

The main technical result we use for the null bounds is the following. Here we assume the permutation-invariance property of the U statistic kernel that h ( x 1 , x 2 ; y 1 , y 2 ) = h ( x 2 , x 1 ; y 1 , y 2 ) = h ( x 1 , x 2 ; y 2 , y 1 ) . This can always be ensured by symmetrizing the kernel, and is satisfied by the kernel used by ̂ MMD 2 which we consider in this paper.

Theorem 7. Fix combined sample Z of size n + m with n ≤ m and let h ( x 1 , x 2 ; y 1 , y 2 ) be a two-sample U-statistic kernel as described above. Define

and

If σ ∈ Uniform( S n + m ) is a random permutation of the dataset, then then for all | t | < (4 √ 2 ¯ U k ( Z )) -1 ,

Theorem 8 (Normaliser Bound) . If h ( x, x ′ ; y, y ′ ) = k ( x, x ′ ) + k ( y, y ′ ) -k ( x, y ′ ) -k ( x ′ , y ) then where n ≤ m and

Additionally, if 0 ≤ k ( x, x ′ ) ≤ κ for all x, x ′ , then

Proof of Theorem 7. Firstly, we make the following definitions based on the work of Kim et al. (2022): let L = { l 1 , . . . , l n } be an n -tuple drawn uniformly without replacement from [ m ] . Then for any fixed Z

where we define

Note that in the above we have used the invariance h ( x 1 , x 2 ; y 1 , y 2 ) = h ( x 1 , x 2 ; y 2 , y 1 ) .

Now additionally define ζ i as i.i.d. Rademacher variables, and let σ ∼ Uniform( S n + m ) . For any fixed Z

where we have defined

This works because we can first define ˜ Z i = Z i or ˜ Z i = Z n + l i , each with probability 1 2 . The distribution of ˜ U L k ( σ ˜ Z ) = d ˜ U L k ( σ Z ) , and then using the symmetry of k ( x, y ) in its arguments gives the equivalence Equation (8) ( c.f. eq. 28, Kim et al., 2022).

Now for fixed Z , combining Equations (7) and (8) and Jensen's inequality we gives

In the final step we have defined the matrix A σ , with entries A σ ij = ( n ( n -1)) -1 E L [ h ( Z σ ( i ) , Z σ ( j ) ; Z σ ( n + l i ) , Z σ ( n + l j ) ) ∣ ∣ σ ] for i = j and A ii = 0 for all i .

We note that ζ is independent of σ and satisfies the conditions of Theorem 6, so applying this theorem we obtain that

̸

where in the second line we have used that ∥ M ∥ ≤ ∥ M ∥ F for any matrix M . Now note that

We complete the proof by noting that σ ∈ S n + m ∥ ∥ F Proof of Theorem 8. We have

where the first inequality holds by Jensen's inequality, the second by convexity (so that ( a + b + c + d ) 2 ≤ 4( a 2 + b 2 + c 2 + d 2 ) ) and the third inequality using n ≤ m to upper bound each of the four scalars inside the parentheses by m ( m -1) , and we also upper bounded each of sums over subsets of terms by the sum over all possible terms.

For the second part note that since h ∈ [ -2 κ, 2 κ ]

C.3 Bounds for MMD-FUSE under Null: Proof of Theorem 5

Proof of Theorem 5. Note under the null Z is permutation invariant. We will use Theorem 7 with h as the MMD 2-statistic kernel, so that U k ( Z ) = ̂ MMD 2 ( Z ) (for fixed k ). We will use the notation ≤ 1 -δ to denote that the inequality holds with probability at least 1 -δ over the random variables being considered.

Note that ¯ U 2 4 κ 2 /n ( n 1) from Theorem 8, so that where the penultimate result holds using Theorem 7 provided | λ | < (4 √ 2 sup k ¯ U k ( Z )) -1 . Using the bound on ¯ U k from above gives the | λ | < √ n ( n -1) / 8 √ 2 κ in the theorem statement.

By dividing ̂ MMD 2 by the permutation invariant ̂ N k in every one of the above, we also find that

(11)

the first inequality using that ̂ N k is permutation-invariant and holding for | λ | < √ sup k ̂ N k (4 √ 2 sup k ¯ U k ( Z )) -1 . The final step uses that ¯ U 2 k ≤ 16 ̂ N k /n ( n -1) from Theorem 8 which also shows that | λ | < √ n ( n -1) / 16 √ 2 is sufficient.

The upper tail bounds for ̂ FUSE N and ̂ FUSE 1 immediately follow from the above by dividing through λ > 0 and from their definitions.

Lower Bounds. For the lower tails,

where the last line follows from Equation (9) replacing λ by dummy variable -s for 0 < s < √ n ( n -1) / 8 √ 2 κ ; we importantly note that this bound held for negative λ by Theorem 7.

By following the same process for ̂ FUSE N and instead using Equation (10), also holding for potentially negative λ , we obtain

For the small δ we generally use, these are tightest when s is at its maximum permissible value, so we substitute these values for the theorem statement.

Under Permutation. To prove the equivalent result for σ Z under permutations, we replace Z by σ Z and our application of Markov's inequality introducing an expectation over Z with one over σ . This changes nothing else in the derivations.

D Concentration under the alternative

We give the exponential convergence bounds for ̂ FUSE 1 under the alternative and relate it to its mean. In the following we will assume that K is a class of kernels bounded by 0 < κ < ∞ , so that ̂ FUSE 1 ∈ [ -2 κ, 2 κ ] . We also introduce the following quantity (for fixed, data-free prior π ) which is closely related to the expectation of ̂ FUSE 1 ,

Theorem 9. FUSE 1 is bounded in the following ways:

and

Under the null hypothesis FUSE 1 = 0 .

We can now state concentration results for ̂ FUSE 1 in terms of FUSE 1 .

Theorem 10. With probability at least 1 -δ over the sample

and with the same probability,

D.1 Proofs

Note on the proofs. The 'bounded difference lemma' (Theorem 11) we give below is not the same as the bounded difference inequality, though it is closely related. In fact, our result could be used as an intermediate step in proving the latter, but we note that this would not be the usual method ( e.g. Boucheron et al., 2013, use an 'entropy' method instead), and the converse is not true. The fact that we have to prove a variant form of an existing concentration inequality to get concentration for our log-sum-exp statistics is similar to how the same is required in PAC-Bayesian proofs, where e.g. PAC-Bayes Bernstein inequalities also require modified proof techniques that mirror those used to prove the usual Bernstein inequalities.

The usual bounded difference inequality cannot be used to prove Theorem 9, since it is not a concentration bound. It also does not give the concentration we need in Theorem 10, since the FUSE statistics do not have the bounded difference property, only ̂ MMD 2 k does; this is why our proofs cannot use a 'plug-in' concentration bound.

Theorem 11 (Bounded Difference Lemma) . A function f has the bounded difference property there exist constants L ℓ < ∞ , ℓ ∈ [ n ] such that

for any choices of z 1 , . . . , z n , z ′ ℓ , and ℓ ∈ [ n ] .

For such a function and any independent random variables Z 1 , . . . , Z n and t ∈ R (subject to appropriate measurability restrictions)

Proof of Theorem 11. We introduce the Doob construction, defining

This is a martingale difference sequence with

It is shown by (for example) Wainwright (2019, Ch. 2.2, p.37) that D ℓ lies in an interval of length at most L ℓ by the bounded differences assumption. Thus, applying iterated expectation and Hoeffding's lemma for the MGF of bounded random variables,

The MGF of ̂ MMD 2 can then be bounded using the following result, proved via the above.

Theorem 12. For bounded kernel k ≤ κ , t ∈ R and sample sizes m,n ,

Proof of Theorem 12. We show that ̂ MMD 2 k ( x, y ) has the bounded differences property and then apply Theorem 11. Denote by x \ ℓ for ℓ ∈ [ n ] that the ℓ -th example in the x sample is changed. Then

Asimilar process for the y sample gives bounds of 8 κ/m , so that ̂ MMD 2 has the bounded differences property with

Proof of Theorem 9. For the first lower bound, note

For the upper bound, note that

The lower bound on FUSE 1 is obtained by relaxing the supremum with ρ = π . The upper bounds come since the supremum over ρ ∈ M 1 + ( K ) : KL( ρ, π ) < ∞ of MMD 2 is clearly greater than the KL-regularised version. Under the null hypothesis, MMD k ( p, q ) = 0 for every kernel (regardless of them being characteristic or not), so FUSE 1 = 0 .

Proof of Theorem 10. For the upper bound,

For the lower bound, let ρ ∗ be the value of ρ achieving the supremum in the dual form of FUSE 1 (which we note is independent of the sample), so that

Now the above is for fixed kernel K ρ ∗ independent of the data, so by Markov's inequality and Theorem 12, for any s > 0 we find

Optimising for s yields s = κ -1 √ log(1 /δ ) / √ 8(1 /m +1 /n ) which gives the desired result.

E Power Analysis

E.1 General Recipe for Power Analysis of Permutation Tests

The general outline for power analysis consists of the following: First we start with the Type II error probability. Then we attempt to upper bound it by iteratively applying the following simple lemma to the different terms.

Monotonicity in High Probability. Let X,Y be r.v.s, such that X ≤ Y w.p. ≥ 1 -δ . Then P ( X ≥ c ) ≤ P ( Y ≥ c ) + δ . This result can be applied both when X ≤ Y a.s. giving δ = 0 , or if Y = a is constant.

Iteratively applying this lemma gives P ( Type II ) ≤ Nδ , and then we set Nδ = β .

As a first step in the above process we will use the following useful result (adapted from Kim et al., 2022) to convert from the random permutations we use in practice to the full set of permutation. This result shows that when B is taken sufficiently large (roughly Ω( α -2 log( β -1 )) ), the only changes to the final power results will be in constants multiplying α and β .

Theorem 13 (From Randomised to Deterministic Permutations) . Suppose G = ( g 1 , . . . , g B +1 ) consists of B uniformly drawn permutations from G , plus the identity permutation as g B +1 . Then

where 1 -α B = B +1 B (1 -α ) + √ log(2 /δ ) 2 B .

We note that provided B ≥ 8 α -2 log(2 /δ ) , then 1 -α B ≤ 1 -α/ 2 .

Proof. We note the Dvoretzky-Kiefer-Wolfowitz inequality Dvoretzky et al. (1956); Massart (1990) for empirical CDF F n of n samples from original CDF F :

for every t > 0 .

The permutation CDF for a group G take the form and given sample ˜ G = ( g 1 , . . . , g B ) i.i.d. uniformly from G (this excludes the identity permutation added to G ), the empirical CDF is

We can also write ̂ F G ( x ) including the identity permutation as g B +1 , and note that ̂ F G ( x ) = Define the good event

which holds with probability at least 1 -δ by Equation (12). Given A , we have

Proof. We define

From a well-known bound (based on Lee, 1990, Equation 2, p.38; see also Kim et al., 2022, Appendix F, Equation 59 or Schrab et al., 2023, Proposition 3),

where we used that n ≤ m ≤ cn . and the result in red above, and the boundedness of the kernel for the final term. This gives the further bound

since

a similar result for σ 01 , and the simple bound σ 2 11 ≤ κ 2 .

where we have defined α B as above.

E.2 Variance of ̂ MMD 2 k

In proving our power results it is necessary to upper bound the variance of ̂ MMD 2 k .

Theorem 14. For any kernel k upper bounded by κ , if n ≤ m ≤ cn for c ≥ 1 , there exists universal constant C > 0 depending only on c , such that

E.3 Proof of Theorem 2

Proof. Define α B as in Theorem 13, noting that 1 -α B < 1 -α/ 2 under the assumption B ≥ 8 α -2 log(4 /β ) . From Theorem 13 we can consider the full permutation set as

We also recall that from Theorem 5 when λ = cn/κ ,

so that

For any ρ , we define

which we substitute into

After substituting S ρ , we used the dual form of ̂ FUSE 1 and the inequalities sup f ( ρ ′ ) ≥ f ( ρ ) , Chebyshev's, and Theorem 14. This term is upper bounded by β/ 2 if we set

We also note that for a, b, x all non-negative, if x 2 > a 2 +2 b , then x 2 > ax + b . This works because x 2 > ax + b is equivalent to x 2 > ( a 2 + √ a 2 4 + b ) 2 by taking the positive root, and

using Young's inequality 2 AB ≤ A 2 + B 2 .

Combining the above the Type II error rate is controlled by β provided any of the following statements are true for any ρ (with each new result implying the former):

E.4 Proof of Theorem 3

The proof of this statement proceeds similarly to the proof of Theorem 2. Using Theorem 5 with λ = cn and Theorem 13 we find that

For any ρ such that KL( ρ, π ) < ∞ we define

We assumed that n ≤ m ≤ cn for some c ≥ 1 . Based on this, note that N k ≤ κ 2 /C 2 1 for C 1 depending only on m/n ∈ [1 , c ] , and (since MMD 2 ≥ 0 is strictly non-negative, unlike ̂ MMD 2 which can be negative),

Now we introduce these definitions to bound

Thus the type II error will be controlled by β if for any ρ

Provided there is a c > 0 with for all k , this reduces to the condition

Applying Theorem 14, the proof is completed in essentially the same way as in the result for ̂ FUSE 1 (with slightly different β dependence).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n[Update: After intensive discussion with the authors, I changed my score from 6 to 7. I further increase the score for \"Contribution\" from 2 to 3.]\n\nThe paper introduces two new test statistics for permutation-based two-sample tests that are related to the MMD -- called FUSE_N and FUSE_1. These are motivated by selecting/combining good kernels for an MMD-based two sample test *without splitting* the data.\nThe authors theoretically show that FUSE_1 corresponds to a (regularized) supremum of the MMD over the possible kernel combinations. FUSE_N, which empirically performs much stronger, is derived from FUSE_1 but where the kernels are weighted by their normalized (squared) MMD estimates instead of simply by their (squared) MMD estimates. There is no theoretical justification for using FUSE_N.\n\nThe authors provide a theoretical power analysis of the proposed tests. Furthermore they run experiments that show their test has competitive power against a recent SOTA MMDAgg, while having a constant speed-up. The code is provided in a well-structured repository, such that it can be easily reproduced or the community can build on it.\n\nThe paper also gives some general insights into what test selection strategies are generally possible when using a permutation test. Whilst this might not be very new in itself, having a clear presentation of the general thoughts in one place, might be very useful for scientist entering the field.\n\nSTRENGTHS:\n- The delivered experiments are done very thoroughly and structured and the code is provided in a high quality.\n- The paper is very well written and clarifies important (although not really new) aspects of permutation-based tests in an accessible way.\n- The proposed test convinces in terms of power and runtime -- (although I am not 100% convinced by the speed-up over MMDAgg yet).\n- Originality and significance: Two-sample testing is an ongoing research area and improvements and new ideas are published at NeurIPS and similar conferences regularly. The community seems nevertheless relatively small. The test statistic seems new, although its motivation is not really clear.\n\nWEAKNESSES:\n- The motivation for the proposed test statistic is not clear enough for me. FUSE_1 has somewhat a motivation (Sec. 4.1.) but:\n  - it does not perform well and also the motivation does not arise from test power considerations.\n  - It is well-known that selecting the kernel by maximizing the MMD is not a theoretically justified strategy (not scale invariant for example).\n- Sutherland et al (2017) give a criterion for optimizing the kernel in terms of asymptotic power, which results in optimizing a Signal-to-noise ratio. The normalized statistic FUSE_N seems to go in this direction, but there is no motivation for the normalization given.\n- Generally I am not really understanding the starting point for this paper. What motivated you to do this work and what where you trying to solve? In particular what is the similarity/difference to MMDAgg? \n- The test statistic has a parameter $\\lambda$. For the theoretical results, $\\lambda$ depends on the sample size. However, in the experiments (as far as I understood) $\\lambda=1$ irrespectively of the sample size. How was this value determined and where is it discussed? And what is the recommended way to do this in practice?\n\nQUESTIONS:\nPlease also see my points in the weakness section.\n### Questions I would like to discuss / understand:\n- What is the motivation for FUSE_N (except that it works well on your experiments)?\n- In Section A.4. you discuss the time-complexity relative to MMDAgg:\n  - The difference is a factor of 2, so I do not think that the $\\mathcal{O}$ notation is adequate here. \n  - I also experienced that computing the kernel matrix is quite costly, compared to the permutations. But this only needs to be done once in both methods. Could it be that this difference is only visible because you used a very large number of permutations (20,000)?\n- l. 143 \"without incurring Bonferrony-type multiple testing penalties\": I understand that your method does not have such a penalty, but what exactly is it that leads to it?\n- What's the \"N\" or \"1\" in the naming of FUSE_ standing for?\n\n\n### Questions / Remarks that do not require a response:\n- l.57 \"Our unsupervised feature extraction method is extemely powerful\": I do not see any support for this statement. I agree that it is very general, but whether it works well, I do not see evidence in the paper.\n- l.108f: Note that Kübler et al (AutoML Two-Sample Test) show in App. B.1 that the asymptotically optimal kernel for a fixed problem is indeed 1-dimensional.\n- l. 116: I do not think that permuting the samples is exactly the same as sampling from 1/2 (p+q). In the permutation there are always exactly half the samples still from p and half from q. Whilst drawing from 1/2(p+q) can by chance (and most likely will) result unbalanced.\n\nLIMITATIONS:\nnothing specific", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks into the problem of permutation based kernel two-sample testing. The paper is motivated by two main challenges: \n\n1. The first one being the fact that selecting kernel hyper-parameters through standard data splitting often reduces the statistical power due to using less data than we actually have.\n2. The second challenge being that if we don't select a data specific hyperparameter for the test, a poor kernel choice will also lead to a decrease in statistical power.\n\nTo tackle these problems, the authors proposed the MMD-FUSE method that can select kernels adapted to the data without data splitting. They first restate a result from Hemerik and Goeman Theorem 2 to justify/emphasise that any permutation invariant representation/function of the data can be used a statistic that have controlled type 1 error rate. In addition justifying the adequacy of median heuristics. \n\nThe second contribution is that they proposed to combine multiple MMD via a set of kernels, and as such optimise the test power.\n\nSTRENGTHS:\nThe problem is well motivated as the trade off between data splitting to learn optimal kernel v.s. not splitting the data but selecting some heuristic kernel has been a long standing problem in the community. This authors explained their arguments very clearly. In addition, not only did they justify their method by showcasing the optimal MMD separation rate (Typo in line 290 btw), I appreciate they included a discussion with computationally faster sub-optimal tests to highlight between the trade off of test power and computational efficiency.\n\nWEAKNESSES:\nThe authors have discussed the weakness in their conclusion section so I don't think I have much to add. There is some recent work on not using permutation test at all to do two sample test, such as the work of Shekhar et al. 2022. I understand it is not necessary to compare the two types of approach but might be also good to mention non-permutation based approach as a side note to complete your already-detailed background section on testing?\n\n\n---\n\nShekhar et al. 2022 \"A Permutation-free Kernel Two-Sample Test\" arXiv:2211.14908\n\nQUESTIONS:\nAlthough you have this prior over a set of kernels which this set could potentially be infinite, in practice you would sample from this prior to compute the test statistic right? How would your approach compare to standard Bongerroni correction technique or its extension weighted bonferroni correction? Would be interesting to hear about your comments on this.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the question of choice of critical values for the MMD test. The authors introduce a new MMD-based test that allows data-driven critical values based on permutation test, without the need to train a new kernel for every permutation. The main advantage of this new method is that it does not require sample splitting and the whole sample can be utilized for the computation of the test value.\n\npros:\n- Experiments show improvements in some cases compared to sample-splitting methods.\n- The authors provide plenty of theoretical results in the appendix, including concentration results for the test both under null and alternative.\n\ncons:\n- The analysis is very conservative. At no point we can state that the distribution of the permuted test conditional on the observed data approximates that of the original test. Theorem 1 in particular is an upper bound, so there is no guarantees that the type I error of the test will be close to the desired level alpha. It is not clear how crude this upper bound is. I suggest this limitation should be emphasised in the text below.\n- I think theoretical analysis can be simplified. For example, proof of Theorem 6 does not have to reproduce Vershynin and Rudelson's proof, it is sufficient to integrate their bound. Also, it is not clear to me why it is necessary to include a proof of Bounded Differences Inequality in the appendix. Instead, one can refer to Lugosi et al Concentration Inequalities, Theorem 6.7.\n- There is no dependence of prior on the data in the experiments, please correct me if I'm wrong.\n\nother comments:\n\nline 174 m+ m -> n + m\n\nline 222 inconsistent notation MMD_k or MMD(.; k)\n\nline 829, display below: not sure why t appeared in second line and why it disappeared in the third line\n\nline 747 FUSE depends on lambda, what depends on t then?\n\nline 872 sup_y -> sup_rho?\n\nline 873 what is N?\n\nSerious concerns:\n\n- Section E.3 proof of Theorem 2 (which in my understanding one of the main theoretical contributions of the paper). Could the authors please elaborate on what happens below line 926 in detail? The sup over rho disappears in line 2. The \"object\" S_rho appears to be random, depending on the observed data Z, and it appears in the final bound thanks to the Markov inequality. \n- I can't find proof of Theorem 3 in the appendix with Ctrl + F\n\nTo my best understanding if there are mistakes above section E.3 they should be fixable.\n\n### My rating is conditional on whether the authors can fix these two serious concerns. Otherwise, the paper must be rejected.\n\nSTRENGTHS:\n-\n\nWEAKNESSES:\n-\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new statistic that can incorporate with the permutation test where we can learn features of data in an unsupervised way. From the perspective of testing, this paper contributes a new idea and paves a new way to perform two-sample testing without data splitting. If we can safely use all data to find a proper kernel, the test power will increase for sure. Theorems and proofs also verify the proposed methods.\n\nSTRENGTHS:\n1. This paper focuses on an important problem: two-sample testing, which will be very important when generators are like humans nowadays.\n\n2. The selected research direction, testing without data splitting, is very important. As demonstrated in summary, a proper kernel selection in this direction can directly boost the test power.\n\n3. Experiments include simple data and complex data (Galaxy MNIST and CIFAR), verifying the effectiveness of the proposed statistics.\n\nWEAKNESSES:\n1. The presentation can be improved a lot. The major issue is the lack of motivation for many choices. For example, what is the motivation to consider a mean kernel? what is the motivation to have this kind of design: log( E( exp( ) ) )? Can we consider more possibilities? The finding of this paper is exciting, however, it looks like a rush version. A revised version with a better presentation is required during the rebuttal.\n\n2. Section 3 is a little bit long. The authors can consider introducing the theorem in the preliminary (as it is not proposed in this paper). Or the authors can remove the permutation test from Section 2. You can also consider merging Sections 2 and 3, making demonstrations more compact.\n\n3. After removing the necessary parts in Sections 2 and 3, more experimental results can be put into the main body. For example, CIFAR results look very promising and should be moved into the main body.\n\n4. In line 115, there is a notation typo.\n\n5. Marks are recommended in Figure 1.\n\nQUESTIONS:\nPlease address the points in Weakness. The presentation can be improved to make this paper even better.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}