Title: Kernel-Based Tests for Likelihood-Free Hypothesis Testing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Kernel methods

Maximum mean discrepancy

Likelihood-free inference

Hypothesis testing

Minimax statistics
Abstract: Given $n$ observations from two balanced classes, consider the task of labeling an additional $m$ inputs that are known to all belong to \emph{one} of the two classes. 
Special cases of this problem are well-known: with complete
knowledge of class distributions ($n=\infty$) the
problem is solved optimally by the likelihood-ratio test; when
$m=1$ it corresponds to binary classification; and when $m\approx n$ it is equivalent to two-sample testing. The intermediate settings occur in the field of likelihood-free inference, where labeled samples are obtained by running forward simulations and the unlabeled sample is collected experimentally. In recent work it was discovered that there is a fundamental trade-off
between $m$ and $n$: increasing the data sample $m$ reduces the amount $n$ of training/simulation
data needed. In this work we (a) introduce a generalization where unlabeled samples 
come from a mixture of the two classes -- a case often encountered in practice; (b) study the minimax sample complexity for non-parametric classes of densities under \textit{maximum mean
discrepancy} (MMD) separation; and (c) investigate the empirical performance of kernels parameterized by neural networks on two tasks: detection
of the Higgs boson and detection of planted DDPM generated images amidst
CIFAR-10 images. For both problems we confirm the existence of the theoretically predicted asymmetric $m$ vs $n$ trade-off.

Kernel-Based Tests for Likelihood-Free Hypothesis Testing

Abstract

Given n observations from two balanced classes, consider the task of labeling an additional m inputs that are known to all belong to one of the two classes. Special cases of this problem are well-known: with complete knowledge of class distributions ( n = ∞ ) the problem is solved optimally by the likelihood-ratio test; when m = 1 it corresponds to binary classification; and when m ≈ n it is equivalent to two-sample testing. The intermediate settings occur in the field of likelihood-free inference, where labeled samples are obtained by running forward simulations and the unlabeled sample is collected experimentally. In recent work it was discovered that there is a fundamental trade-off between m and n : increasing the data sample m reduces the amount n of training/simulation data needed. In this work we (a) introduce a generalization where unlabeled samples come from a mixture of the two classes - a case often encountered in practice; (b) study the minimax sample complexity for non-parametric classes of densities under maximum mean discrepancy (MMD) separation; and (c) investigate the empirical performance of kernels parameterized by neural networks on two tasks: detection of the Higgs boson and detection of planted DDPM generated images amidst CIFAR-10 images. For both problems we confirm the existence of the theoretically predicted asymmetric m vs n trade-off.

1 Likelihood-Free Inference

The goal of likelihood-free inference (LFI) [8; 12; 17; 24], also called simulation-based inference (SBI), is to perform statistical inference in a setting where the data generating process is a black-box, but can be simulated. Given the ability to generate samples X θ ∼ P ⊗ n θ for any parameter θ , and given real-world data Z ∼ P ⊗ m θ ⋆ , we want to use our simulations to learn about the truth θ ⋆ . LFI is particularly relevant in areas of science where we have precise but complex laws of nature, for which we can do (stochastic) forward simulations, but can not directly compute the (distribution) density P θ . The Bayesian community approached the problem under the name of Approximate Bayesian Computation (ABC) [6; 13; 47]. More recent ML-based methods where regressors and classifiers

are used to summarize data, select regions of interest, approximate likelihoods or likelihood-ratios [14; 29; 30; 42; 49] have also emerged for this challenge.

Despite empirical advances, the theoretical study of frequentist LFI is still in its infancy. We focus on the nonparametric and non-asymptotic setting, which we justify as follows. For applications where tight error control is critical one might be reluctant to rely on asymptotics. More broadly, the nonasymptotic regime can uncover new phenomena and provide insights for algorithm design. Further, parametric models are clearly at odds with the black-box assumption. Recently, [18] proposed likelihood-free hypothesis testing ( LFHT ) as a simplified model and found minimax optimal tests for a range of nonparametric distribution classes, thereby identifying a fundamental simulationexperimentation trade-off between the number of simulated observations n and the size of the experimental data sample m . Here we extend [18], and prior related work [19; 26; 27; 32; 33], to a new setting designed to model experimental setups more truthfully and derive sample complexity (upper and lower bounds) for kernel-based tests over nonparametric classes.

While minimax optimal, the algorithms of [18; 19] are impractical as they rely on discretizing the observations on a regular grid. Thus, both in our theory as well as experiments we turn to kernel methods which provide an empirically more powerful set of algorithms that have shown success in nonparametric testing [20; 21; 22; 31; 39].

Contributions Our contributions are twofold. Theoretically , we introduce mixed likelihood-free hypothesis testing ( mLFHT ), which is a generalization of ( LFHT ) and provides a better model of applications such as the search for new physics [11; 38]. We propose a robust kernel-based test and derive both upper and lower bounds on its minimax sample complexity over a large nonparametric class of densities, generalizing multiple results in [18; 19; 26; 27; 32; 33; 37]. Although the simulationexperimentation ( m vs n ) trade-off has been proven in the minimax sense (that is, for some worst-case data distribution), it is not clear whether it actually occurs in real data. Our second contribution is the empirical confirmation of the existence of an asymmetric trade-off, cf. Figure 1.2. To this end we construct state-of-the-art tests building on ideas of [39; 48] on learning good kernels from the data. We execute this program in two settings: the Higgs boson discovery [5], and detecting diffusion [25] generated images planted in the CIFAR-10 [35] dataset.

1.1 LFHT and the Simulation-Experimentation Trade-off

Suppose we have i.i.d. samples X,Y each of size n from two unknown distributions P X , P Y on a measurable space X , as well as a third i.i.d. sample Z ∼ P Z of size m . In the context of LFI, we may think of the samples X,Y as being generated by our simulator, and Z being the data collected in the real world. The problem we refer to as likelihood-free hypothesis testing is the task of deciding between the hypotheses

This problem originates in [23; 52], where authors study the exponents of error decay for finite X and fixed P X , P Y as n ∼ m → ∞ ; more recently [2; 18; 19; 26; 27; 32; 33; 37] it is studied in the non-asymptotic regime. Assuming that P X , P Y belong to a known nonparametric class of distributions P and are guaranteed to be ϵ -separated with respect to total variation (TV) distance (i.e. TV( P X , P Y ) ≥ ϵ ), [18] characterizes the sample sizes n and m required for the sum of type-I and type-II errors to be small, as a function of ϵ and for several different P 's. Their results show, for three settings of P , that ( i ) testing ( LFHT ) at vanishing error is possible even when n is not large enough to estimate P X and P Y within total variation distance O ( ϵ ) , and that ( ii ) to achieve a fixed level of error, say α , one can trade off m vs. n along a curve of the form { min { n, √ mn } ≳ n TS ( α, ϵ, P ) , m ≳ log(1 /α ) /ϵ 2 } . Here n TS denotes the minimax sample complexity of two-sample testing over P , i.e. the minimum number of observations n needed from P X , P Y ∈ P to distinguish the cases TV( P X , P Y ) ≥ ϵ versus P X = P Y . Here ≳ suppresses dependence on constants and untracked parameters.

It is unclear, however, whether predictions drawn from minimax sample complexities over specified distribution classes can be observed in real-world data. Without the theory, a natural expectation is that the error contour { ( m,n ) : ∃ a test with total error ≤ α } would look similar to that of minimax two-sample testing with unequal sample size, namely { ( m,n ) : min { n, m } ≳ n TS ( α, ϵ, P ) } , i.e. n and m simply need to be above a certain threshold simultaneously (as is the case for e.g. two-sample testing over smooth densities [4; 37]). However, from Figures 2 and 1.2 we see that there is indeed a

non-trivial trade-off between n and m : the contours are not always parallel to the axes and aren't symmetric about the line m = n . The importance of Fig. 1.2 is in demonstrating that said trade-off is not a kink of a theory that arises due to some esoteric worst-case data distribution, but is instead a real effect observed in state-of-the-art LFI algorithms ran on actual data. We remark that the n used in this plot is the total number of simulated samples (most of which are used for choosing a neural-network parameterized kernel) and are not just the n occuring in Theorems 3.1 and 3.2 which apply to a fixed kernel. See Section 4 for details on sample division.

1.2 Mixed Likelihood-Free Hypothesis Testing

A prominent application of likelihood-free inference lies in the field of particle physics. Scientists run sophisticated experiments in the hope of finding a new particle or phenomenon. Often said phenomenon can be predicted from theory, and thus can be simulated, as was the case for the Higgs boson whose existence was verified after nearly 50 years at the Large Hadron Collider (LHC) [3; 9].

Suppose we have n simulations from the background distribution P X and the signal distribution P Y . Further, we also have m (real-world) datapoints from P Z = (1 -ν ) P X + νP Y , i.e. the observed data is a mixture between the background and signal distributions with rate parameter ν . The goal of physicists is to construct confidence intervals for ν , and a discovery corresponds to a 5 σ confidence interval that excludes ν = 0 . We model this problem by testing

for fixed (usually predicted) δ > 0 . See the rigorous definition of ( mLFHT ) in Section 3. In particular, a discovery can be claimed if H 0 is rejected.

2 The Likelihood-Free Test Statistic

This section introduces the testing procedure based on Maximum Mean Discrepancy (MMD) that we study throughout the paper both theoretically and empirically. First, we introduce the necessary background on MMD in Section 2.1. Then, we define our test statistics in Section 2.2.

2.1 Kernel Embeddings and MMD

Given a set X , we call the function K : X 2 → R a kernel if the n × n matrix with ij 'th entry K ( x i , x j ) is symmetric positive semidefinite for all choices of x 1 , . . . , x n ∈ X and n ≥ 1 . There is a unique reproducing kernel Hilbert space (RKHS) H K associated to K . H K consists of functions X ↦→ R and satisfies the reproducing property ⟨ K ( x, · ) , f ⟩ H K = f ( x ) for all f ∈ H k and x ∈ X , in particular K ( x, · ) ∈ H K . Given a probability measure P on X , define its kernel embedding θ P as

Given the kernel embeddings of two probability measures P, Q , we can measure their distance in the RKHS by MMD( P, Q ) ≜ ∥ θ P -θ Q ∥ H K , where MMD stands for maximum mean discrepancy. MMD has a closed form thanks to the reproducing property and linearity:

where ( X,X ′ , Y, Y ′ ) ∼ P ⊗ 2 ⊗ Q ⊗ 2 . In particular, if P, Q

are empirical measures based on observations, we can evaluate the MMD exactly, which is crucial

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

lg(m)

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

lg(n)

Higgs experiment

6.5

10

252

1.0

10

100

1.5

10

40

1.4

10

16

4.9

10

7

3.1

10

3

0.10

0.40

0.69

0.86

0.94

1.0

1.5

2.0

2.5

3.0

3.5

4.0

lg(m)

1.0

1.5

2.0

2.5

3.0

3.5

4.0

lg(n)

CIFAR experiment

0.00

0.15

0.30

0.45

0.60

0.75

0.90

Type I error + Type II error

in practice. Yet another attractive property of MMD is that (under mild integrability conditions) it is an integral probability metric (IPM) where the supremum is over the unit ball of the RKHS H K . See e.g. [41; 46] for references. The following result is a consequence of the fact that self-adjoint compact operators are diagonalizable.

Theorem 2.1 (Hilbert-Schmidt) . Suppose that K ∈ L 2 ( µ ⊗ µ ) is symmetric. Then there exists a sequence ( λ j ) j ≥ 1 ∈ ℓ 2 and an orthonormal basis { e j } j ≥ 1 of L 2 ( µ ) such that K ( x, y ) = ∑ j ≥ 1 λ j e j ( x ) e j ( y ) for all j ≥ 1 , where convergence is in L 2 ( µ ⊗ µ ) .

Assumption 1. Unless specified otherwise, we implicitly assume a choice of a non-negative measure µ and kernel K for which the conditions of Theorem 2.1 hold. Note that λ j ≥ 0 and depend on µ .

Removing the Bias In our proofs we work with the kernel embedding of empirical measures for which we need to modify the inner product ⟨· , ·⟩ H K (and thus MMD ) slightly by removing the diagonal terms. Namely, given i.i.d. samples X,Y of size n, m respectively and corresponding empirical measures ̂ P X , ̂ P Y , we define

̸

̸

̸

We also write ⟨ θ ̂ P X , θ ̂ P X ⟩ u, H K ≜ ∥ θ ̂ P X ∥ 2 u, H K ≜ 1 n ( n -1) ∑ i = j K ( X i , X j ) and extend linearly. The u stands for unbiased, since E MMD 2 u ( ̂ P X , ̂ P Y ) = MMD 2 ( P X , P Y ) = E MMD 2 ( ̂ P X , ̂ P Y ) in general.

2.2 Test Statistic

With Section 2.1 behind us, we are in a position to define the test statistic that we use to tackle ( mLFHT ). Suppose that we have samples X,Y,Z of sizes n, n, m from the probability measures P X , P Y , P Z . Write ̂ P X for the empirical measure of sample X , and analogously for Y, Z . The core of our test statistic for ( mLFHT ) is the following:

Note that T is of the additive form 1 m ∑ m j =1 f ( Z i ) where f ( z ) ≜ θ ̂ P Y ( z ) -θ ̂ P X ( z ) can be interpreted as the witness function of [21; 31]. Given some π ∈ [0 , 1] (taken to be half the predicted signal rate δ/ 2 in our proofs), the output of our test is

✶ The threshold γ gives Ψ π a natural geometric interpretation: it checks whether the projection of θ ̂ P Z -θ ̂ P X onto the vector θ ̂ P Y -θ ̂ P X falls further than π along the segment joining θ ̂ P X to θ ̂ P Y (up to deviations due to the omitted diagonal terms, see Section 2.1).

̸

Setting δ = 1 in ( mLFHT ) recovers ( LFHT ), and the corresponding test output is Ψ δ/ 2 = Ψ 1 / 2 = 1 if and only if MMD u ( ̂ P Z , ̂ P X ) ≥ MMD u ( ̂ P Z , ̂ P Y ) . This very statistic (i.e. MMD u ( ̂ P Z , ̂ P X ) -MMD u ( ̂ P X , ̂ P Y ) ) has been considered in the past for relative goodness-of-fit testing [7] where it's asymptotic properties are established. In the non-asymptotic setting, if MMD is replaced by the L 2 -distance we recover the test statistic studied by [18; 27; 33]. However, we are the first to introduce Ψ δ for δ = 1 and to study MMD-based tests for (m)LFHT in a non-asymptotic setting.

̸

Variance cancellation At first sight it may seem more natural to the reader to threshold the distance MMD u ( ̂ P Z , ̂ P X ) , resulting in rejection if, say, MMD u ( ̂ P Z , ̂ P X ) ≥ MMD u ( ̂ P X , ̂ P Y ) δ/ 2 . The geometric meaning of this would be similar to the one outlined above. However, there is a crucial difference: ( LFHT ) (the case δ = 1 ) is possible with very little experimental data m due to the cancellation of variance . More precisely, the statistic MMD 2 ( ̂ P Z , ̂ P X ) contains the term 1 m ( m -1) ∑ i = j K ( Z i , Z j ) -whose variance is prohibitively large and would inflate the m required for reliable testing - but this can be canceled by subtracting MMD 2 ( ̂ P Z , ̂ P Y ) . Our statistic T ( X,Y,Z ) -γ ( X,Y,π ) simply generalizes this idea to ( mLFHT ).

̸

3 Minimax Rates of Testing

3.1 Upper Bounds on the Minimax Sample Complexity of ( mLFHT )

Let us start by reintroducing ( mLFHT ) in a rigorous fashion. Given C, ϵ, R ≥ 0 , let P µ ( C, ϵ, R ) denote the set of triples ( P X , P Y , P Z ) of distributions such that the following three conditions hold:

( i ) P X , P Y and P Z have µ -densities bounded by C ,

( ii ) MMD( P X , P Y ) ≥ ϵ , and

( iii ) MMD( P Z , (1 -ν ) P X + νP Y ) ≤ R · MMD( P X , P Y ) ,

where we define ν = ν ( P X , P Y , P Z ) = arg min ν ′ ∈ R MMD( P Z , (1 -ν ′ ) P X + ν ′ P Y ) . For some δ > 0 , consider the two hypotheses

which we regard as subsets of probability measures. Notice that R controls the level of misspecification in the direction that is orthogonal to the line connecting the kernel embeddings of P X and P Y . Setting R = 0 simply asserts that P Z is guaranteed to be a mixture of P X and P Y , as is the case for prior works on LFHT. Before presenting our main result on the minimax sample complexity of mLFHT, let us define one final piece of terminology. We say that a test Ψ , which takes some data as input and takes values in { 0 , 1 } , has total error probability less than α for the problem of testing H 0 vs H 1 if

Theorem 3.1. Suppose we observe three i.i.d. samples X,Y,Z from distributions P X , P Y , P Z composed of n, n, m observations respectively and let C ∈ (0 , ∞ ) and R,ϵ ≥ 0 and δ ∈ (0 , 1) . There exists a universal constant c > 0 such that Ψ δ/ 2 defined Section 2.2 tests H 0 vs H 1 , as defined in (5) , at total error α provided

Note that Theorem 3.1 does not place assumptions on the distributions P X , P Y beyond bounded density with respect to the base measure µ . This is different from usual results in statistics, where prior specification of distribution classes is crucial. On the other hand, instead of standard distances such as L p , we assume separation with respect to MMD and the latter is potentially harder to interpret than, say, L 1 i.e. total variation. We do point out that our Theorem 3.1 can be used to derive results in the classical setting; we discuss this further in Section 3.4.

In an appropriate regime of the parameters, the sufficient sample complexity in Theorem 3.1 exhibits a trade-off of the form min { n, √ mn } ≳ ∥ λ ∥ 2 log(1 /α ) / ( δϵ 2 ) between the number of simulation samples n and real observations m . This trade-off is shown in Figure 2 using data from a toy problem. The trade-off is clearly asymmetric and the relationship m · n ≥ const. also seems to appear. In this toy problem we set R = 0 , δ = 1 , ϵ = . 3 , k = 100 and

Type-l error

1e4

- 0.40

4e3

0.35

2e3

0.30

1e3

0.25

4e2

n

0.20

2e2

le2

0.15

0.10

4e1

2e1

0.05

lel

0.00

le1

2e1

4e1

le2

2e2

4e2

1e3

2e3

4e3

1e4

m

P X = P Z , P Y are distributions on { 1 , 2 , . . . , k } with P X ( i ) = (1 + ϵ · (2 · ✶ { i odd } -1)) /k = 2 /k -P Y ( i ) for all i = 1 , 2 , . . . , k . The kernel we take is K ( x, y ) = ∑ k i =1 ✶ { x = y = i } and µ is simply the counting measure; the resulting MMD is simply the L 2 -distance on pmfs.

Figure 1.2 illustrates a larger scale experiment using real data using a trained kernel. Note that we plot the total number n of simulation samples, including those used for training the kernel itself (see Section 4); which ensures that Figure 1.2 gives a realistic picture of data requirements. However, due to the dependence between the kernel and the data, Theorem 3.1 no longer applies. Nevertheless, we observe a trade-off similar to Figure 2.

3.2 Lower Bounds on the Minimax Sample Complexity of ( mLFHT )

In this section we prove a minimax lower bound on the sample complexity of mLFHT, giving a partial converse to Theorem 3.1. Before we can state this results, we must make some technical definitions. Given J ≥ 2 , let ∥ λ ∥ 2 2 J ≜ ∑ J j =2 λ 2 j and define

Theorem 3.2 (Lower Bounds for mLFHT) . Suppose that ∫ X K ( x, y ) µ (d x ) ≡ λ 1 , µ ( X ) = 1 and sup x ∈X K ( x, x ) ≤ 1 . There exists a universal constant c > 0 such that any test of H 0 vs H 1 , as defined in (5) , with total error at most α must use a number ( n, m ) of observations that satisfy

Remark 3.3 . Recall that the eigenvalues λ depend on the choice of µ , so that by choosing a different base measure µ one can optimize the lower bound. However, since P X , P Y , P Z are assumed to have bounded density with respect to µ , this appears rather involved.

Remark 3.4 . The requirements sup x ∈X K ( x, x ) ≤ 1 and µ ( X ) = 1 are is essentially without loss of generality, as µ and K can be rescaled. The condition ∫ X K ( x, y ) µ (d x ) ≡ λ 1 implies that the top eigenfunction e 1 is equal to a constant or equivalently, that y ↦→ K ( x, y ) µ (d x ) defines a Markov kernel up to a normalizing constant.

3.3 Tightness of Theorems 3.1 and 3.2

Dependence on ∥ λ ∥ 2 ∥ λ ∥ 2 ∥ λ ∥ 2 An apparent weakness of Theorem 3.2 is its reliance on the unknown value J ⋆ ϵ , which depends on the specifics of the kernel K and base measure µ . Determining it is potentially highly nontrivial even for simple kernels. Slightly weakening Theorem 3.2 we obtain the following corollary, which shows that the dependence on ∥ λ ∥ 2 is tight, at least for small ϵ .

Corollary 3.5. Suppose J ≥ 2 is such that ∑ J j =2 λ 2 j ≥ c 2 ∥ λ ∥ 2 2 for some c ≤ 1 . Then ∥ λ ∥ 2 J ⋆ ϵ can be replaced by c ∥ λ ∥ 2 in Theorem 3.2 whenever ϵ ≤ c ∥ λ ∥ 2 / (2 √ J -1) .

Dependence on R and α Due to the general nature of our lower bound constructions, it is difficult to capture the dependence on the misspecification parameter R . As for the probability of error α , based on related work [16] we expect the gap of size √ log(1 /α ) to be a shortcoming of Theorem 3.1 and not the lower bound. Closing this gap may require a different approach, however, as tests based on empirical L 2 distances are known to have a sub-optimal concentration [27].

Dependence on δ The correct dependence on the signal rate δ is the most important question left open by our theoretical results. Any method requiring n larger than a function of δ irrespective of m (as in Theorem 3.1) is provably sub-optimal because taking m ≳ 1 / ( δϵ ) 2 and n large enough to estimate both P X , P Y to within accuracy ϵ/ 10 always suffices to reach a fixed level of total error.

3.4 Relation to Prior Results

In this section we discuss some connections of Theorem 3.1 to prior work. Specifically, we discuss how Theorem 3.1 recovers some known results in the literature [4; 18; 37] that are minimax optimal . Details omitted in this section are included in Appendix B.

Binary Hypothesis Testing Suppose the two distributions P X , P Y are known , we are given m i.i.d. observations Z 1 , . . . , Z m ∼ P Z and our task is to decide between the hypotheses H 0 : P X = P Z versus H 1 : P Y = P Z . Then, we may take n = ∞ , R = 0 , δ = 1 in Theorem 3.1 to conclude that

observations suffice to perform the test at total error α .

Two-Sample Testing Suppose we have two i.i.d. samples X and Y , both of size n , from unknown distributions P X , P Y respectively and our task is to decide between H 0 : P X = P Y against H 1 :

MMD( P X , P Y ) ≥ ϵ . We split our Y sample in half resulting in Y (1) and Y (2) and form the statistic Ψ TS ≜ Ψ 1 / 2 ( X,Y (1) , Y (2) ) -Ψ 1 / 2 ( Y (1) , X, Y (2) ) , where Ψ 1 / 2 is defined in Section 2.2. Then | E Ψ TS | is equal to 0 under the null hypothesis and is at least 1 -2 α 1 under the alternative, where α 1 is the target total error probability of Ψ 1 / 2 . Taking α 1 = 5% , by repeated sample splitting and majority voting we may amplify the success probability to α provided

where c ′ > 0 is universal (see Appendix for details). The upper bound (7) partly recovers [37, Theorem 3 and 5] where authors show that thresholding the MMD with Gaussian kernel G σ ( x, y ) = σ -d exp( -∥ x -y ∥ 2 /σ 2 ) achieves the minimax optimal sample complexity n ≍ ϵ -(2 β + d/ 2) /β for the problem of two-sample testing over the class P β,d of d -dimensional ( β, 2) -Sobolev-smooth distributions (defined in Appendix B.3) under ϵ -L 2 -separation. For this, taking σ ≍ ϵ 1 /β ensures that ∥ P -Q ∥ L 2 ≲ MMD( P, Q ) over P, Q ∈ P β,d . Taking e.g. d µ ( x ) = exp( -∥ x ∥ 2 2 )d x as the base measure, (7) recovers the claimed sample complexity since ∥ λ ∥ 2 2 = ∫ G 2 σ ( x, y )d µ ( x )d µ ( y ) = O ( σ -d ) hiding dimension dependent constants. Our result requires a bounded density with respect to a Gaussian.

Likelihood-Free Hypothesis Testing By taking α ≍ R ≍ δ = Θ(1) in Theorem 3.1 we can recover many of the results in [18; 32; 33]. When X is finite, we can take the kernel K ( x, y ) = ∑ z ∈X ✶ { x = y = z } in Theorem 3.1 to obtain the results for bounded discrete distributions (defined in Appendix B.1) which state that under ϵ -TV-separation the minimax optimal sample complexity is given by m ≳ 1 /ϵ 2 ; min { n, √ nm } ≳ √ |X| /ϵ 2 . A similar kernel recovers the optimal result for the class of β -Hölder smooth densities on the hypercube [0 , 1] d (see Appendix B.2).

Curse of Dimensionality Using the Gaussian kernel G σ as for two-sample testing above, one can conclude by Theorem 3.1 that the required number of samples for ( mLFHT ) over the class P β,d under ϵ -L 2 -separation grows at most like Ω ( ( c ϵ ) Ω( d ) 1 δ 2 ) for some c > 1 , instead of the expected Ω ( ( c ϵδ ) Ω( d ) ) . This may be interpreted as theoretical support for the success of LFI in practice where signal and background can be rather different (cf. [5, Figures 2-3]) and the difficulty of the problem stems from the rate of signal events being small (i.e. ϵ ≈ 1 but δ ≪ 1 ).

4 Learning Kernels from Data

Given a fixed kernel K , our Theorems 3.1 and 3.2 show that the sample complexity is heavily dependent on the separation ϵ under the given MMD as well as the spectrum λ = λ ( µ, K ) of the kernel. Thus, to have good test performance we need to use a kernel K that is well-adapted to the problem at hand. In practice, however, instead of using a fixed kernel it would be only natural to use part of the simulation sample to try to learn a good kernel.

In Sections 4 and 5 we report experimental results after training a kernel parameterized by a neural network on part of the simulation data. In particular, due to the dependence between the data and the kernel, Theorem 3.1 doesn't directly apply. Our main contribution here is showing the existence of an asymmetric simulation-experimentation trade-off (cf. Figure 1.2 and also Section 1.1) even in this realistic setting. Figure 1.2 plots the total number n of simulations used, including those used for training, so as to provide a realistic view of the amount of data used. The experiments also illustrate that the (trained-)kernel-based statistic of Section 2.2 achieves state-of-the-art performance.

4.1 Proposed Training Algorithm

Consider splitting the data into three parts: ( X tr , Y tr ) is used for training (optimizing) the kernel; ( X ev , Y ev ) is used to evaluate our test statistic at test time; and ( X cal , Y cal ) is used for calibrating the distribution of the test statistic under the null hypothesis. We write n s = | X s | = | Y s | for s ∈ { tr , ev , cal } . Given the training data X tr , Y tr with empirical measures ̂ P X tr , ̂ P Y tr , we maximize the objective in

which was introduced in [48]. Here ̂ σ 2 is an estimator of the variance of MMD 2 u ( ̂ P X tr , ̂ P Y tr ; K ) and is defined in Appendix F.1.

Algorithm 1 mLFHT with a learned deep kernel

Input: ( X tr , X ev , X cal ) , ( Y tr , Y ev , Y cal ); parametrized kernel K ω ; hyperparameters and initialization.

# Phase 1:

Kernel training (optimization) on

# maximize objective as defined in (8)

# Phase 2: Distributional calibration of test statistic (under null hypothesis).

Z cal ,r ← sample m points without replacement from X cal ;

end for

# Phase 3: Inference with input Z .

Output:

Estimated

p

-value:

1

k

k

i

=1

✶

{

̂

T < T

i

}

.

Intuitively, the objective J aims to separate P X from P Y while keeping variance low. For a heuristic justification of its use for ( mLFHT ) see Appendix.

In Algorithm 1 we describe the training and testing procedure, which produces unbiased p -values for ( mLFHT ) when there is no misspecification ( R = 0 in Theorem 3.1). During training, we use the Adam optimizer [34] with stochastic batches.

Proposition 4.1. When there is no misspecification ( R = 0 in Theorem 3.1), Algorithm 1 outputs an unbiased estimate of the p -value that is consistent as min { n cal , k } → ∞ .

Time complexity Algorithm 1 runs in three separate stages: training, calibration, and inference. The first two take O ( # epochs · B 2 + kn ev m ) total time, where B is the batch size, whereas Phase 3 takes only O ( n ev m ) time, which is generally much faster especially if n ev < < n tr .

Sample usage Empirically, data splitting in Algorithm 1 can have non-trivial effects on performance. Instead of training the kernel on only a fraction of the data ( { X tr , Y tr } ∩ { X ev , Y ev } = ∅ ), we discovered that taking { X ev , Y ev } ⊆ { X tr , Y tr } results in more efficient use of data. The stronger condition { X ev , Y ev } = { X tr , Y tr } can also be applied; we take ⊆ to reduce time complexity. We do, however, crucially require X cal , Y cal in Phase 2 to be independently sampled ('held-out') for consistent p -value estimation. Finally, we remark also that splitting this way is only valid in the context of Algorithm 1. For the test (4) using the data-dependent threshold γ , one needs { X tr , Y tr } ∩ { X ev , Y ev } = ∅ to estimate γ .

4.2 Classifier-Based Tests and Other Benchmarks

Let ϕ : X → [0 , 1] be a classifier, assigning small values to P X and high values to P Y by minimizing cross-entropy loss of a classifier net. There are two natural test statistics based on ϕ :

Scheffé's Test. The first idea, attributed to Scheffé in folklore [15, Section 6], is to take the statistic T ( Z ) = 1 m ∑ m i =1 { ϕ ( Z i ) > t } where t is some (learn-able) threshold.

✶ Approximate Neyman-Pearson / Logit Methods. If ϕ is trained to perfection, then ϕ ( z ) = P ( P X | z ) would be the likelihood and ϕ ( z ) / (1 -ϕ ( z )) would equal precisely the likelihood ratio between P Y and P X at z . This motivates the use of T ( Z ) = 1 m ∑ m i =1 log( ϕ ( Z i ) / (1 -ϕ ( Z i ))) . See also [10].

Let us list the testing procedures that we benchmark against each other in our experiments.

MMD-M : The MMD statistic (3) using K with the mixing architecture

Here G σ is the Gaussian kernel with variance σ 2 ; φ ω , φ ′ ω ′ are NN's (with parameters ω, ω ′ ), and σ, σ 0 , τ, ω, ω ′ are trained.

MMD-G : The MMD statistic (3) using the Gaussian kernel architecture K ( x, y ) = G σ ( φ ω ( x ) , φ ω ( y )) where φ ω is the feature mapping parametrized by a trained network and σ is a trainable parameter.

MMD-O : The MMD statistic (3) using the Gaussian Kernel K ( x, y ) = G σ ( x, y ) with optimized bandwidth σ . First proposed in [7; 39].

∑

X

tr and

Y

tr

.

50

100

150

200

250

300

350

m

0.0

0.2

0.4

0.6

0.8

1.0

50

100

150

200

250

300

350

m

0.0

0.1

0.2

0.3

0.4

0.5

Expected P-Value

50

100

150

200

250

300

350

m

0.0

0.1

0.2

0.3

0.4

0.5

Error Probability At 0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

0.0

0.2

0.4

0.6

0.8

1.0

ROC curves for different m

m=384

m=256

m=192

m=128

m=64

UME : An interpretable model comparison algorithm proposed by [31], which evaluates the kernel mean embedding on a chosen 'witness set'.

SCHE , LBI : Scheffé's test and Logit Based Inference methods [10], based on a binary classifier network ϕ trained via cross-entropy, introduced above.

RFM : Recursive Feature Machines, a recently proposed kernel learning algorithm by [44].

4.3 Additive Statistics and the Thresholding Trick

Given a function f (usually obtained by training) and test data Z = ( Z 1 , . . . , Z m ) , we call a test additive if its output is obtained by thresholding T f ( Z ) ≜ 1 m ∑ m i =1 f ( Z i ) . We point out that all of MMD-M/G/O , SCHE , LBI , UME , RFM are of this form, see the Appendix for further details. Similarly to [39], we observe that any such statistic can be realized by our kernel-based approach.

Proposition 4.2. The kernel-based statistic defined in (3) with the kernel K ( x, y ) = f ( x ) f ( y ) is equal to T f up to a multiplicative and additive constant independent of Z .

Motivated by the Scheffé's test, instead of directly thresholding the additive statistic T f ( Z ) , we found empirically that replacing f by f t ( x ) ≜ ✶ { f ( x ) > t } can yield improved power. We set t by maximizing an estimate of the significance under the null using a normal approximation, i.e. by solving t opt ≜ arg max t T f t ( Y opt ) -T f t ( X opt ) √ T f t ( X opt )(1 -T f t ( X opt )) , where X opt , Y opt satisfy { X opt , Y opt } ∩ ( { X cal , Y cal } ∪ { X ev , Y ev } ) = ∅ . This trick improves the performance of our tester on the Higgs dataset in Section 5.2 but not for the image detection problem in Section 5.1.

5 Experiments

Our code can be found at https://github.com/Sr-11/LFI .

5.1 Image Source Detection

Our first empirical study looks at the task of detecting whether images come from the CIFAR-10 [36] dataset or a SOTA generative model (DDPM) [25; 51]. While source detection is on its own interesting, it turns out that detecting whether a group of images comes from the generative model versus the real dataset can be too 'easy' (see experiments in [31]). Therefore, we consider a mixed alternative , where the alternative hypothesis is not simply the generative model but CIFAR with planted DDPM images. Namely, our n labeled images come from the following distributions:

The goal is to test whether the m unlabeled observations Z have been corrupted with ρ or more fraction of DDPM images (versus uncorrupted CIFAR); this corresponds to ( LFHT ) (or equivalently ( mLFHT ) with δ = 1 ). Figure 3 shows the performance of our approach with this mixed alternative.

Network Architecture With a standard deep CNN, the difference is only at the final layer: for the kernel-based tests it is a feature output; for classifiers, we add an extra linear layer to logits.

We see from Figure 3 that our kernel-based test outperforms other benchmarks at a fixed training set size n tr . One potential cause is that MMD has an 'optimization' subroutine (which it solves in closed form) as it is an IPM. This additional layer of optimization may lead to better performance at small sample sizes. The thresholding trick does not seem to improve power empirically. We omit several benchmarks from this figure for graphic presentation and they do not exhibit good separating power; see the Appendix for the complete results. The bottom plot of Figure 1.2 shows m and n on log -scale against the total probability of error, exhibiting the simulation-experimentation trade-off.

5.2 Higgs-Boson Discovery

The 2012 announcement of the Higgs boson's discovery by the ATLAS and CMS experiments [1; 9] marked a significant milestone in physics. The statistical problem inherent in the experiment is well-modeled by ( mLFHT ), using a signal rate δ predicted by theory and misspecification parameter R = 0 (as was assumed in the original discovery). We consider our algorithm's power against past studies in the physics literature [5] as measured by the significance of discovery . We note an important distinction from Algorithm 1 in this application.

Estimating the Significance In physics, the threshold for claiming a 'discovery' is usually at a significance of 5 σ , corresponding to a p -value of 2 . 87 × 10 -7 . Approximately n cal ∼ (2 . 87) -1 × 10 7 samples would be necessary for Algorithm 1 to reach such a precision. Fortunately the distribution of the test statistic is approximated by a Gaussian customarily. We adopt this approach for our experiment hereby assuming that m is large enough for the CLT to

2.6

2.0

1.4

0.8

0.4

0.1

Training set size 2n/million

3.0

3.5

4.0

4.5

5.0

Significance of discovery/

MMD-M with

t

opt

SCHE with

t

opt

UME

LBI

apply. We use the 'expected significance of discovery' as our metric [5] which, for the additive statistic T f = 1 m ∑ m i =1 f ( Z i ) , is given by δ ( T f ( Y cal ) -T f ( X cal )) √ ̂ var( f ( X cal )) /m . If the thresholding trick (Section 4.3) is applied we use the more precise Binomial tail, in which case the significance is estimated by -Φ -1 ( P ( Bin ( m,T f t ( X cal )) ≥ T f t ( Z ))) , where Φ is the standard normal CDF.

opt opt

Newtowk Architecture The architecture is a 6-layer feedforward net similar for all tests (kernelbased and classifiers) except for the last layer. We leave further details to the Appendix.

As can be seen in Figure 4, Scheffé's test and MMD-M with threshold t opt are the best methods, achieving similar performance as the algorithm of [5]; reaching the significance level of 5 σ on 2.6 million simulated datapoints and a test sample made up of a mixture of 1000 backgrounds and 100 signals. The top plot of Figure 1.2 shows m and n on log -scale against the total probability of error through performing the test (4), exhibiting the asymmetric simulation-experimentation trade-off.

6 Conclusion

In this paper, we introduced ( mLFHT ) as a theoretical model of real-world likelihood-free signal detection problems arising in science. We proposed a kernel-based test statistic and analyzed its minimax sample complexity, obtaining both upper (Theorem 3.1) and lower bounds (Theorem 3.2) in terms of multiple problem parameters, and discussed their tightness (Section 3.3) and connections to prior work (Section 3.4). On the empirical side, we described a method for training a parametrized kernel and proposed a consistent p -value estimate (Algorithm 1 and Proposition 4.1). We examined the performance of our method in two experiments and found that parametrized kernels achieve state-of-the-art performance compared to relevant benchmarks from the literature. Moreover, we confirmed experimentally the existence of the asymmetric simulation-experimentation trade-off (Figure 1.2) which is suggested by minimax analysis. We defer further special cases of Theorem 3.1, all relevant proofs and experimental details to the Appendix.

Acknowledgments and Disclosure of Funding

YP was supported in part by the National Science Foundation under Grant No CCF-2131115. Research was sponsored by the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

References

Aad, G., Abajyan, T., Abbott, B., Abdallah, J., Khalek, S. A., Abdelalim, A. A., Aben, R., Abi, B., Abolins, M., AbouZeid, O., et al. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Physics Letters B , 716(1):1-29, 2012.

Acharya, J., Das, H., Jafarpour, A., Orlitsky, A., Pan, S., and Suresh, A. Competitive classification and closeness testing. In Conference on Learning Theory , pages 22-1. JMLR Workshop and Conference Proceedings, 2012.

Adam-Bourdarios, C., Cowan, G., Germain, C., Guyon, I., Kégl, B., and Rousseau, D. The Higgs boson machine learning challenge. In NIPS 2014 workshop on high-energy physics and machine learning , pages 19-55. PMLR, 2015.

Arias-Castro, E., Pelletier, B., and Saligrama, V. Remember the curse of dimensionality: The case of goodness-of-fit testing in arbitrary dimension. Journal of Nonparametric Statistics , 30 (2):448-471, 2018.

Baldi, P., Sadowski, P., and Whiteson, D. Searching for exotic particles in high-energy physics with deep learning. Nature communications , 5(1):1-9, 2014.

Beaumont, M. A. Approximate bayesian computation. Annual review of statistics and its application , 6:379-403, 2019.

Bounliphone, W., Belilovsky, E., Blaschko, M. B., Antonoglou, I., and Gretton, A. A test of relative similarity for model selection in generative models. In Bengio, Y . and LeCun, Y ., editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016.

Brehmer, J., Louppe, G., Pavez, J., and Cranmer, K. Mining gold from implicit models to improve likelihood-free inference. Proceedings of the National Academy of Sciences , 117(10): 5242-5249, 2020.

Chatrchyan, S., Khachatryan, V., Sirunyan, A. M., Tumasyan, A., Adam, W., Aguilo, E., Bergauer, T., Dragicevic, M., Erö, J., Fabjan, C., et al. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Physics Letters B , 716(1):30-61, 2012.

Cheng, X. and Cloninger, A. Classification logit two-sample testing by neural networks. CoRR , abs/1909.11298, 2019.

Cowan, G., Cranmer, K., Gross, E., and Vitells, O. Asymptotic formulae for likelihood-based tests of new physics. The European Physical Journal C , 71(2):1-19, 2011.

Cranmer, K., Brehmer, J., and Louppe, G. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062, 2020.

Csilléry, K., Blum, M. G., Gaggiotti, O. E., and François, O. Approximate bayesian computation (abc) in practice. Trends in ecology & evolution , 25(7):410-418, 2010.

Dalmasso, N., Izbicki, R., and Lee, A. Confidence sets and hypothesis testing in a likelihoodfree inference setting. In International Conference on Machine Learning , pages 2323-2334. PMLR, 2020.

Devroye, L. and Lugosi, G. Combinatorial Methods in Density Estimation . Springer Series in Statistics. Springer New York, 2012. ISBN 9781461301257.

Diakonikolas, I., Gouleakis, T., Kane, D. M., Peebles, J., and Price, E. Optimal testing of discrete distributions with high probability. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 542-555, 2021.

Diggle, P. J. and Gratton, R. J. Monte carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society: Series B (Methodological) , 46(2):193-212, 1984.

Gerber, P. R. and Polyanskiy, Y. Likelihood-free hypothesis testing. CoRR , abs/2211.01126, 2022. doi: 10.48550/arXiv.2211.01126.

Gerber, P. R., Han, Y., and Polyanskiy, Y. Minimax optimal testing by classification, 2023.

Gretton, A., Fukumizu, K., Harchaoui, Z., and Sriperumbudur, B. K. A fast, consistent kernel two-sample test. Advances in neural information processing systems , 22, 2009.

Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Gretton, A., Sejdinovic, D., Strathmann, H., Balakrishnan, S., Pontil, M., Fukumizu, K., and Sriperumbudur, B. K. Optimal kernel choice for large-scale two-sample tests. Advances in neural information processing systems , 25, 2012.

Gutman, M. Asymptotically optimal classification for multiple tests with empirically observed statistics. IEEE Transactions on Information Theory , 35(2):401-408, 1989.

Gutmann, M. U., Dutta, R., Kaski, S., and Corander, J. Likelihood-free inference via classification. Statistics and Computing , 28:411-425, 2018.

Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Huang, D. Hypothesis Testing and Learning with Small Samples . PhD thesis, Citeseer, 2012.

Huang, D. and Meyn, S. Classification with high-dimensional sparse samples. In 2012 IEEE International Symposium on Information Theory Proceedings , pages 2586-2590. IEEE, 2012.

Ingster, Y. I. Minimax testing of nonparametric hypotheses on a distribution density in the l p metrics. Theory of Probability & Its Applications , 31(2):333-337, 1987. doi: 10.1137/1131042.

Izbicki, R., Lee, A., and Schafer, C. High-dimensional density ratio estimation with extensions to approximate likelihood computation. In Artificial intelligence and statistics , pages 420-429. PMLR, 2014.

Jiang, B., Wu, T.-y., Zheng, C., and Wong, W. H. Learning summary statistic for approximate bayesian computation via deep neural network. Statistica Sinica , pages 1595-1618, 2017.

Jitkrittum, W., Kanagawa, H., Sangkloy, P., Hays, J., Schölkopf, B., and Gretton, A. Informative features for model comparison. Advances in Neural Information Processing Systems , 31, 2018.

Kelly, B. G., Tularak, T., Wagner, A. B., and Viswanath, P. Universal hypothesis testing in the learning-limited regime. In 2010 IEEE International Symposium on Information Theory , pages 1478-1482. IEEE, 2010.

Kelly, B. G., Wagner, A. B., Tularak, T., and Viswanath, P. Classification of homogeneous data with large alphabets. IEEE transactions on information theory , 59(2):782-795, 2012.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Krizhevsky, A. Learning multiple layers of features from tiny images. University of Toronto , 2009.

Krizhevsky, A., Nair, V., and Hinton, G. The cifar-10 dataset (2014). Online: http://www. cs. toronto. edu/kriz/cifar. html , 55, 2020.

Li, T. and Yuan, M. On the optimality of gaussian kernel based nonparametric tests against smooth alternatives. arXiv preprint arXiv:1909.03302 , 2019.

Lista, L. Statistical methods for data analysis in particle physics , volume 941. Springer, 2017.

Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Sutherland, D. J. Learning deep kernels for non-parametric two-sample tests. In International conference on machine learning , pages 6316-6326. PMLR, 2020.

Lopez-Paz, D. and Oquab, M. Revisiting classifier two-sample tests. In International Conference on Learning Representations , 2017.

Muandet, K., Fukumizu, K., Sriperumbudur, B., Schölkopf, B., et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning , 10 (1-2):1-141, 2017.

Papamakarios, G., Sterratt, D., and Murray, I. Sequential neural likelihood: Fast likelihoodfree inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 837-848. PMLR, 2019.

Polyanskiy, Y. and Wu, Y. Information Theory: From Coding to Learning . Cambridge University Press, 2023+.

Radhakrishnan, A., Beaglehole, D., Pandit, P., and Belkin, M. Feature learning in neural networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212.13881 , 2022.

Sackrowitz, H. and Samuel-Cahn, E. P values as random variables-expected p values. The American Statistician , 53(4):326-331, 1999. ISSN 00031305.

Schölkopf, B. and Smola, A. J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond . The MIT Press, 06 2001. ISBN 9780262256933. doi: 10.7551/ mitpress/4175.001.0001.

Sisson, S. A., Fan, Y., and Beaumont, M. Handbook of approximate Bayesian computation . CRC Press, 2018.

Sutherland, D. J., Tung, H., Strathmann, H., De, S., Ramdas, A., Smola, A. J., and Gretton, A. Generative models and model criticism via optimized maximum mean discrepancy. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Thomas, O., Dutta, R., Corander, J., Kaski, S., and Gutmann, M. U. Likelihood-free inference by ratio estimation. Bayesian Analysis , 17(1):1-31, 2022.

Tsybakov, A. B. Introduction to Nonparametric Estimation . Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.

von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/ diffusers , 2022.

Ziv, J. On classification with empirically observed statistics and universal data compression. IEEE Transactions on Information Theory , 34(2):278-286, 1988.

Appendix

A Notation

We use A ≳ B,A ≲ B,A ≍ B to denote A = Ω( B ) , B = Ω( A ) and A = Θ( B ) respectively, where the hidden constants depend on untracked parameters multiplicatively. 2

We write TV , KL , χ 2 for total-variation, KL-divergence and χ 2 -divergence, respectively. We write D ( P Y | X ∥ Q Y | X | P X ) = E X ∼ P X D ( P Y | X ∥ Q Y | X ) as the conditional divergence for any probability measures P, Q on two variables X,Y and divergence D ∈ { TV , KL , χ 2 } .

We write ℓ p for the usual ℓ p sequence space and L p for the usual L p space with respect to the Lebesgue measure. Both the ℓ p norm and the L p norm are written as ∥ · ∥ p if no ambiguity arises.

For real numbers a, b ∈ R we also write max { a, b } as a ∨ b and min { a, b } as a ∧ b .

We use ⃗ 1 d to denote an d -dimensional all 1 's vector.

For an integer k ∈ Z + , we write [ k ] as a short notation for the set { 1 , 2 , . . . , k } .

In the proofs of Theorem 3.1 and Theorem 3.2, we use ! = for an equality that we are trying to prove.

B Applications of Theorem 3.1

Usually, minimax rates of testing are proven under separation assumptions using more traditional measures of distance such as L p , where p ∈ [1 , ∞ ] . In this section we show one example of how Theorem 3.1 can be used to recover known results, and also obtain some novel results under L 2 -separation and L 1 -separation.

B.1 Bounded Discrete Distributions Under L 2 /L 1 -Separation

Sample Complexity Upper Bounds Let P Db ( k, C ) be the set of all discrete distributions P supported on [ k ] = { 1 , 2 , . . . , k } satisfying max 1 ≤ i ≤ k p ( i ) ≤ C/k , where p is the probability mass function of P (here ∑ k i =1 p ( k ) = 1 ). For distributions P X , P Y , P Z we shall write p X , p Y , p Z as their probability mass functions, respectively.

Let us apply Theorem 3.1 with underlying space X = [ k ] and measure µ = 1 k ∑ k i =1 δ i . Take the kernel K ( x, y ) = ✶ { x = y } = ∑ k i =1 ✶ { x = y = i } , and note that for any two distributions P X , P Y we have

where ( X,X ′ , Y, Y ′ ) ∼ P ⊗ 2 X ⊗ P ⊗ 2 Y . So the corresponding MMD is the ℓ 2 -distance on probability mass functions. Note also that K = ∑ k i =1 1 k ( √ k { x = i } )( √ k { y = i } ) , where

of multiplicity k . Suppose that we observe samples X,Y,Z of size n, n, m from P X , P Y , P Z ∈ P Db ( k, C ) , where MMD( P X , P Y ) = √∑ i | p X ( i ) -p Y ( i ) | 2 ≥ ϵ . Plugging into Theorem 3.1 shows that:

Proposition B.1. For any two P X , P Y ∈ P Db ( k, C ) , if the ℓ 2 -distance between p X , p Y is at least ϵ , then testing ( mLFHT ) is possible at total error α using n simulation samples and m real data samples provided that

where R is defined as in the assumption ( iii ) of Section 3.1.

We can convert the above results to measure separation with respect to total variation (recall TV( p, q ) = 1 2 ∑ i | p ( i ) -q ( i ) | = 1 2 ∥ p -q ∥ 1 ) using the AM-QM inequality ∥ p X -p Y ∥ 1 ≤ √ k ∥ p X -p Y ∥ 2 . Then, taking R ≍ α ≍ δ = Θ(1) recovers the minimax optimal results of [18; 32; 33], for LFHT over the class P Db . Note that analogous results for two-sample testing follow from the above using the reduction presented in Section 3.4.

Sample Complexity Lower Bounds Recall the definition of J ⋆ ϵ and note that ∥ λ ∥ 2 2 ,J = min( J -1 ,k ) k 2 for all J ≥ 2 . By Corollary 3.5 we see that J ⋆ ϵ ≳ k as soon as ϵ ≲ 1 /k . Thus, for ϵ ≲ 1 /k the necessity of

follows by Theorem 3.2. Here it is crucial to note that when δ = Θ(1) , we have

and hence the upper bound (10) meets with the lower bound (11) provided R ≍ δ = Θ(1) . Once again, setting R ≍ δ ≍ α = Θ(1) we the optimal lower bounds recovering the results of [18] (in the regime ϵ ≲ 1 /k ). In short we can also recover the following result for LFHT .

Proposition B.2 ([18, Theorem 1, adapted]) . On the class P Db ( k, C ) , using n simulation samples and m real data samples, if

then for any two distributions P X , P Y ∈ P Db ( k, C ) with ∥ p X -p Y ∥ 2 ≥ ϵ , testing ( LFHT ) is possible with a total error of 1% . Conversely, to ensure the existence of a procedure that can test ( LFHT ) with a total error of 1% for any P X , P Y ∈ P Db ( k, C ) with ∥ p X -p Y ∥ 2 ≥ ϵ , the number of observations ( n, m ) must satisfy

The implied constants in (12) and (13) do not depend on k and ϵ , but may differ.

B.2 β -Hölder Smooth Densities on [0 , 1] d Under L 2 /L 1 -Separation

Sample Complexity Upper Bounds Let P H ( β, d, C ) be the set of all distributions on [0 , 1] d with β -Hölder smooth Lebesgue-density p satisfying ∥ p ∥ C β ≤ C for some constant C > 1 , where

̸

where ⌈ β -1 ⌉ is the largest integer strictly smaller than β and | α | = ∑ i α i is the norm of a multiindex α ∈ N d . Abusing notation, we also use P H ( β, d, C ) to denote the set of all corresponding density functions.

We take K ( x, y ) = ∑ j ✶ { x, y ∈ B j } , where { B j } j ∈ [ κ ] d is the j 'th cell of the regular grid of size κ d on [0 , 1] d , i.e., B j = [( j -⃗ 1 d ) /κ, j/κ ] for j ∈ [ κ ] d . Clearly there are κ d nonzero eigenvalues, each equal to 1 . The following approximation result is due to Ingster [28], see also [4, Lemma 7.2].

Lemma B.3. Let f, g ∈ P H ( β, d, C ) with ∥ f -g ∥ 2 ≥ ϵ . Then, there exist constants c, c ′ independent of ϵ such that for any κ ≥ cϵ -1 /β ,

Now, suppose that we have samples X,Y,Z of size n, n, m from P X , P Y , P Z ∈ P H ( β, d, C ) with densities p X , p Y , p Z such that ∥ p X -p Y ∥ 2 ≥ ϵ . Then, Theorem 3.1 combined with Lemma B.3 and the choice κ ≍ ϵ -1 /β shows that

Proposition B.4. Testing ( mLFHT ) on P H ( β, d, C ) at total error α using n simulation and m real data samples is possible provided

where ϵ is an L 2 -distance lower bound between P X , P Y and R is defined as in the assumption ( iii ) of Section 3.1.

Setting R ≍ α ≍ δ = Θ(1) recovers the optimal results of [18] for the class P H . Once again, identical results under L 1 separation follow from Jensen's inequality ∥ · ∥ L 1 ([0 , 1] d ) ≤ ∥ · ∥ L 2 ([0 , 1] d ) . Note that analogous results for two-sample testing follow from the above using the reduction presented in Section 3.4.

Sample Complexity Lower Bounds The kernel defined in the previous paragraph is not suitable for constructing lower bounds over the class P H because its eigenfunctions do not necessarily lie in P H . It would be possible to consider a different kernel that is more adapted to this problem/class but we do not pursue this here.

B.3 ( β, 2) -Sobolev Smooth Densities on R d Under L 2 -Separation

Sample Complexity Upper Bounds Let P S ( β, d, C ) be the class of distributions that are supported on R d and whose Lebesgue density p satisfies ∥ p ∥ β, 2 ≤ C , where

and F denotes the Fourier transform. Again, abusing notation, we write P S ( β, d, C ) both as the set of distributions and the set of density functions.

We take the Gaussian kernel G σ ( x, y ) = σ -d exp( -∥ x -y ∥ 2 2 /σ 2 ) on X = R d with base measure d µ ( x ) = exp( -x 2 )d x . In [37] the authors showed that the two-sample test that thresholds the Gaussian MMD with appropriately chosen variance σ 2 achieves the minimax optimal sample complexity over P S , when separation is measured by L 2 . A key ingredient in their proof is the following inequality.

Lemma B.5 ([37, Lemma 5]) . Let f, g ∈ P S ( β, d, C ) with ∥ f -g ∥ 2 ≥ ϵ . Then, there exist constants c, c ′ independent of ϵ such that for any σ ≤ c ϵ 1 /β , we have

Now, suppose that we have samples X,Y,Z of sizes n, n, m from P X , P Y , P Z ∈ P S ( β, d, C ) for some constant C with densities p X , p Y , p Y satisfying ∥ p X -p Y ∥ 2 ≥ ϵ .

Note that the heat-semigroup is an L 2 -contraction ( ∥ λ ∥ ∞ ≤ 1 ) and that

up to constants depending on the dimension. Theorem 3.1 combined with Lemma B.5 and a choice σ ≍ ϵ 1 /β yields the following result.

Proposition B.6. Testing ( mLFHT ) over the class P S with total error α is possible provided

where ϵ is the lower bound on the L 2 -distance between P X , P Y and R is defined as in the assumption ( iii ) of Section 3.1.

Taking R ≍ δ ≍ α = Θ(1) above, we obtain new results for LFHT and using the reduction from two-sample testing given in Section 3.4 we partly recover [37, Theorem 5]. Only partly, because the above requires bounded density with respect to our base measure d µ ( x ) = exp( -x 2 )d x .

Sample Complexity Lower Bounds Note that our lower bound Theorem 3.2 doesn't apply because the top eigenfunction of the Gaussian kernel is not constant. Once again, a more careful choice of the base measure (or kernel) might lead to a more suitable argument for the lower bound. We leave such pursuit as open.

C Black-box Boosting of Success Probability

In this section we briefly describe how upper bounds on the minimax sample complexity in the constant error probability regime ( α = Θ(1) ) can be used to obtain the dependence log(1 /α ) in the small error probability regime ( α = o (1) ). We will argue abstractly in a way that applies to the setting of Theorem 3.1.

Suppose that from some distributions P 1 , P 2 , . . . , P k we take samples X 1 , X 2 , . . . , X k of size n 1 , n 2 , . . . , n k respectively and are able to decide between two hypotheses H 0 and H 1 (fixed but arbitrary) with total error probability at most 1 / 3 . Call this test as Ψ( X 1 , . . . , X k ) ∈ { 0 , 1 } , so that

Now, to each an error of o (1) , instead, we take 18 n 1 log(2 /α ) , . . . , 18 n k log(2 /α ) observations from P 1 through P k , and split each sample into 18 log(2 /α ) equal sized batches { X i,j } i ∈ [ k ] ,j ∈ [18 log(2 /α )] . Here 18 log(2 /α ) is assumed to be an integer without loss of generality. The split samples form 18 log(2 /α ) independent binary random variables

for j = 1 , 2 , . . . , 18 log(2 /α ) . We claim that the majority voting test

tests H 0 against H 1 with total probability of error at most α , where

Indeed, by Hoeffding's inequality, we have

Therefore, in the remainder of our upper bound proofs, we only focus on achieving a constant probability of error ( α = Θ(1) ) as the logarithmic dependence follows by the above.

Remark C.1 . As mentioned in the discussion succeeding Corollary 3.5, we do conjecture the tight dependence in the upper bound to be √ log( α -1 ) instead of log( α -1 ) shown by this method.

D Proof of Theorem 3.1

D.1 Notation and Technical Tools

We use the expansion

extensively, where λ ≜ ( λ 1 , λ 2 , . . . ) are K 's eigenvalues (regarded as an integral operator on L 2 ( µ ) ) in non-increasing order and e 1 , e 2 , . . . are the corresponding eigenfunctions forming an orthonormal basis for L 2 ( µ ) , and convergence is to be understood in L 2 ( µ ) . We use the notation ⟨ · ⟩ ≜ ∫ · d µ . For all u ∈ L 2 ( µ ) we define

and consequently u = ∑ ℓ u ℓ e ℓ . We also define

where the second equality follows from the orthonormality of { e ℓ } ∞ ℓ =1 . Note that the RKHS embedding satisfies θ u ≜ ∫ K ( x, · ) u ( x )d µ ( x ) = K [ u ] . Now, for P X we write

where p X is the µ -density of P X . The similar notations also apply to P Y , P Z . The following identities will be very useful in our proofs.

Lemma D.1. For each identity below, let f, g, h ∈ L 2 ( µ ) be such that the quantity is well defined. Then,

Suppose that f, g are probability densities with respect to µ that are bounded by C . Then

Proof. We prove each claim, starting with (15). Clearly

as required. The second claim (16) follows immediately from (15) by definition. For (17) by orthogonality we have

For (18) by the definition of K [ · ] we have

For (19) we can write

Finally, for (20) we have

Suppose now that f, g are probability densities with respect to µ that are bounded by C > 0 . Let X,Y be independent random variables following the densities f, g . Then

as claimed, where we used that the e ℓ are orthonormal.

D.2 Mean and Variance Computation

We take π = δ/ 2 . Our statistic reads

Recall that ν = arg min ν ′ ∈ R MMD( P Z , ¯ ν ′ P X + ν ′ P Y ) . Let us write z = ¯ νx + νy + r for 1 -¯ ν = ν , where the residual term is denoted as r ∈ L 2 ( µ ) . Let θ r = ∫ r ( t ) K ( t, · ) µ (d t ) be the mean embedding of r . Under both hypotheses we assume that ∥ θ r ∥ H K ≤ R · MMD( P X , P Y ) , moreover ⟨ θ r , θ P Y -θ P X ⟩ H K = 0 by the definition of ν . We look at each of the 5 + ( 5 2 ) = 15 terms of the variance separately.

For the cross terms we obtain

Note that ( n 2 ) 2 -( n 2 ) -( n 2 )( n 4 ) = n ( n -1) 2 -n ( n -1) . Collecting terms, and simplifying, we get the coefficient of the 1 n term:

After expanding z ℓ as z ℓ = ¯ νx ℓ + νy ℓ + r ℓ , we split the calculation into multiple parts to simplify it. First, we focus on terms that are multiplied by ( x ℓℓ ′ -x ℓ x ℓ ′ ) and do not contain r ℓ or r ℓ ′ . Using Lemma D.1 extensively and the fact that ¯ π = 1 -π, ¯ ν = 1 -ν , we find that the sum of these terms equals

Similarly, the terms involving ( y ℓℓ ′ -y ℓ y ℓ ′ ) but not r ℓ or r ℓ ′ sum up to the quantity

Next, collecting the terms involving both ( x ℓℓ ′ -x ℓ x ℓ ′ ) and r ℓ or r ℓ ′ we get

Finally, collecting the terms involving both ( y ℓℓ ′ -y ℓ y ℓ ′ ) and r ℓ or r ℓ ′ we get

Similarly we get

The remaining coefficients don't rely on subtle cancellations, and simple bounds yield

Summarizing, we've found that

Using that ⟨ θ r , θ P Y -θ P X ⟩ H K = 0 , we compute the expectation to be

Taking π ≜ δ/ 2 and applying Chebyshev's inequality shows that there exists a universal constant c > 0 , such that the testing problem is possible at constant error probability (say α = 5% ), provided that the sample sizes m,n satisfy the following inequalities:

By repeated sample splitting and majority voting (see Appendix C), we can boost the success probability of this test to the desired level 1 -α by incurring a multiplicative Θ(log(1 /α )) factor on the sample sizes n, m , which yields the desired result.

E Proof of Theorem 3.2

E.1 Information theoretic tools

Our lower bounds rely on the method of two fuzzy hypotheses [50]. Given a measurable space S , let M ( S ) denote the set of all probability measures on S . We call subsets H ⊆ M ( S ) hypotheses. The following is the main technical result that our proofs rely on.

Lemma E.1. Take hypotheses H 0 , H 1 ⊆ M ( S ) and P 0 , P 1 ∈ M ( S ) random with P ( P i ∈ H i ) = 1 . Then

̸

where the infimum is over all tests ψ : X → { 0 , 1 } .

Proof. For any ψ

̸

Optimizing over ψ we get that the RHS above is equal to 1 2 (1 -TV( E P 0 , E P 1 )) as required.

Therefore, to prove a lower bound on the minimax sample complexity of testing with total error probability α , we just need to construct two random measures P i ∈ H i such that 1 -TV( E P 0 , E P 1 ) = Ω( α ) . In our proofs we also use the following standard results on f -divergences.

Lemma E.2 ([43, Section 7]) . For any probability distributions P, Q the inequalities

hold.

Lemma E.3 (Chain rule for χ 2 -divergence) . Let P X,Y , Q X,Y be probability measures such that the marginals on X are equal ( P X = Q X ). Then

Proof. Let P X,Y , Q X,Y have densities p, q with respect to some µ . Then, by some abuse of notation, we have

E.2 Constructing hard instances

Recall that in the statement of Theorem 3.2, we assume that µ ( X ) = 1 , sup x ∈X K ( x, x ) ≤ 1 and ∫ K ( x, y ) µ (d x ) ≡ λ 1 . Let f 0 ≡ 1 and for each η ∈ {± 1 } N define

where { ρ j } j ≥ 2 is chosen as ρ j = ✶ { 2 ≤ j ≤ J } √ λ j / ∥ λ ∥ 2 ,J , where we define ∥ λ ∥ 2 ,J = √ ∑ 2 ≤ j ≤ J λ 2 j for some J ≥ 2 . Notice that ∫ f η ( x ) µ (d x ) = µ ( X ) = 1 due to orthogonality of the eigenfunctions. Assume from here on that J is chosen so that for all η we have f η ( x ) ≥ 1 / 2 for all x ∈ X . This makes f η into a valid probability density with respect to the base measure µ . Before continuing, we prove the following Lemma, which gives a lower bound on the maximal J for which f η ≥ 1 / 2 for all η .

Lemma E.4. J ≤ J ⋆ ϵ holds provided 2 ϵ √ J -1 ≤ ∥ λ ∥ 2 J .

Proof of Lemma E.4. Notice that

where we use ∥ K ( x, · ) ∥ H = √ K ( x, x ) . We have

and the result follows.

Note that Lemma E.4 immediately gives us a proof of Corollary 3.5.

Proof of Corollary 3.5. Suppose that J is such that ∑ J j =2 λ 2 j ≥ c 2 ∥ λ ∥ 2 2 . Then, by Lemma E.4, if ϵ ≤ ∥ λ ∥ 2 J / (2 √ J -1) then J ≤ J ⋆ ϵ . By assumption, this is implied by the inequality ϵ ≤ c ∥ λ ∥ 2 / (2 √ J -1) , and the result follows.

Continuing with our proof, note that by construction we have

E.2.1 Lower Bound on m

Again, we apply Lemma E.1 with the new (deterministic) construction

where we write f ✶ = f (1 , 1 ,... ) and similarly for g ✶ . By the data-processing inequality for χ 2 -divergence (also by Lemma E.3), we may drop the first 2 n coordinates and obtain

The lower bound m ≳ λ 2 log(1 /α ) / ( δϵ ) 2 now follows readily.

E.2.2 Lower Bound on n

Once again, we apply Lemma E.1 to the new construction

where we put a uniform prior on η ∈ {± 1 } N as before. Using the subadditivity of total variation under products, we compute

Just as in Appendix E.2.3 we upper bound by the χ 2 -divergence to get

The lower bound n ≳ √ log(1 /α ) ∥ λ ∥ 2 ,J /ϵ 2 now follows readily.

E.2.3 Lower Bound on m · n

We take a uniform prior on η and consider the random measures

Our goal is to apply Lemma E.1 to P 0 , P 1 . Notice that (1 -δ ) f 0 + δf η = 1 + δϵg η . Let us write X,Y,Z for the marginals first n , second n and last m coordinates of P 0 and P 1 . By the data processing inequality and the chain rule Lemma E.3 we have

Notice that the expectation inside the χ 2 -divergence is with respect to η given the variables Y , or in other words, over the posterior of η with uniform prior given n observations from the density 1 + ϵg η = f η . The outer expectation is over Y . Given Y , let η and η ′ be i.i.d. from said posterior. We get the bound

Define the collections of variables η -j = { η j } j ≥ 2 \ { η j } and η ′ -j similarly. We shall prove the following claim:

for some universal constant c > 0 . Assuming that (29) holds, by induction we can show that

Thus, if mn + δ 2 m 2 = o ( ∥ λ ∥ 2 2 ,J / ( δ 2 ϵ 4 ) ) then testing is impossible.

We now prove (29). Since the variable η ′ j η ′ j is either 1 or -1 , we have

Let us write η ± 1 ,j for the vector of signs equal to η but whose j 'th coordinate is ± 1 respectively. Looking at the probability above, and using the independence of η, η ′ given Y , we have

Taking the expectation E [ · | η -j , η ′ -j ] and using the HM-AM inequality ( 1 2 ( x + y )) -1 ≤ 1 2 ( 1 x + 1 y ) valid for all x, y > 0 gives

Note that f η 1 j = f η -1 j + 2 ϵρ j e j . Using the lower bound f η ± 1 j ( x ) ≥ 1 2 for all x ∈ X and the inequality 1 + x ≤ exp( x ) , we get

Recall that ( ⋆ ) is a probability so ( ⋆ ) ≤ 1 , and we obtain

Putting it together and applying Lemma E.5 we get

for universal c = 16 > 0 . Thus, by Lemma E.2 we obtain

The necessity of follows immediately. 3

Lemma E.5. For a, b ≥ 0 , the following inequality holds:

Proof. If b ≥ ln 3 or a ≥ 1 we have:

If b < ln 3 and a < 1 , we have

and then

The result follows from ln 3 > 1 .

F Proofs From Section 4

F.1 Computing ˆ σ

We follow the implementation of ̂ σ 2 in [39]. Given X 1 , . . . , X tr n tr sampled from P X and Y 1 , . . . , Y tr n tr sampled from P Y , denote

Then ̂ σ 2 is computed via

Note that ˆ σ 2 being non-negative follows from the AM-GM inequality.

F.2 Heuristic Justification of the Objective (8)

As usual, let X,Y,Z denotes samples of sizes n, n, m from P X , P Y , P Z respectively. Let us give a heuristic justification for using the training objective defined in (8) for the purpose of obtaining a kernel for LFHT / mLFHT . Note that originally it was proposed as a training objective for kernels to be used in two sample testing. Recall that our test for LFHT can be written as where

Heuristically, to maximize the power of ( mLFHT ), we would like to maximize the following population quantity

where

Let T TS = MMD u ( ˆ P X , ˆ P Y ) be the usual statistic that is thresholded for two-sample testing. Then, a computation analogous to that in Section D.2 show (cf. (22)) that

for some A ( K ) and B ( K ) . Therefore, we have approximately

which only differs from our optimization objective defined in (8) by a constant factor.

Second, notice that MMD( P X ,P Y ; K ) √ var( T TS ) depends only on P X -P Y and that ((1 -δ ) P X + δP Y ) -P X ∝ P Y -P X , therefore it is sensible to use (8) as our training objective for is also sensible for ( mLFHT ), and we don't even need to observe the sample Z .

F.3 Proof of Proposition 4.1

Proof. In this proof we regard D ≜ ( X tr , X ev , Y tr , Y ev ) and the parameters of the kernel ω as fixed. Recall that we are looking at the problem mLFHT with a misspecification parameter R = 0 (see Theorem 3.1). Given a test set { z i } i ∈ [ m ] , our test statistic is T ( { z i } i ∈ [ m ] ) = 1 m ∑ m i =1 f ( z i ) where

In Phase 3 of Algorithm 1, we observe the value ̂ T = T ( Z ) = 1 m ∑ m i =1 f ( Z i ) and reject the null hypothesis for large values of ̂ T . Thus, the p -value is defined as

̸

Phase 2 of our Algorithm 1 produces random variables T 1 , . . . , T k that all have the distribution of T ( { ˜ Z i } i ∈ [ m ] ) , so that ✶ { T r ≥ ̂ T } ( r = 1 , . . . , k ) are unbiased estimates of the p -value. However, the T i are not independent, because they sample from the finite collection of calibration samples X cal . However, as n cal →∞ the covariances between T r 1 , T r 2 for r 1 = r 2 tend to zero, and we obtain a consistent estimate of p .

F.4 Proof of Proposition 4.2

Proof. The test statistic T ( X,Y,Z ) in (3) is given by

This simplifies to (consider K ( x, y ) = f ( x ) f ( y ) )

where C ( X,Y ) does not depend on z . Therefore, for any witness function f , we obtain the desired additive test.

F.5 Additive Test Statistics

In this section we prove accordingly that the test statistics of all of MMD-M/G/O , SCHE , LBI , UME , RFM are of the form T f ( Z ) = 1 m ∑ m i =1 f ( Z i ) (where f might depends on X,Y ). The test is to compare T f ( Z ) with some threshold γ ( X,Y ) .

Note that in the setting of Algorithm 1, the X and Y here correspond to X ev and Y ev .

MMD-M/G/O As described in (3) we have

where

SCHE As described in Section 4.2 we have

LBI As described in Section 4.2 we have

UME As described in [31], the UME statistic evaluates the squared witness function at J q test locations W = { w k } J q k =1 ⊂ X . Formally for any two distributions P, Q we define

̸

However, we note a crucial difference that their result only considers the case of n = m , and their proposed estimator for U 2 ( P Z , P X ) can not be naturally extended to the case of n = m . Here we generalize it to m = n where we (conveniently) use a biased estimate of their distance. Given samples X,Y,Z and a set of witness locations W , the test statistic is a (biased yet) consistent estimator of U 2 ( P Z , P Y ) -U 2 ( P Z , P X ) . Let ψ W ( z ) = 1 √ J q ( K ( z, w 1 ) , . . . , K ( z, w J q )) ∈ R | W | be the 'feature function,' then:

Here ⟨· , ·⟩ denotes the usual inner product. Therefore, the difference between distances is

where F is sum function based only on X,Y . This is clearly an additive statistic for Z .

RFM Algorithm 1 in [44] describes a method for learning a kernel from data given a binary classification task. For convenience lets concatenate the data to X RFM = ( X,Y ) ∈ R 2 n × d and labels y RFM = ( ⃗ 0 n , ⃗ 1 n ) ∈ R 1 × 2 n . Given a learned kernel K , we write the Gram matrix as ( K ( X RFM , X RFM )) i,j = K ( X RFM i , X RFM j ) ( 1 ≤ i, j ≤ 2 n ). Let K ( X RFM , z ) be a column vector with components K ( X RFM i , z ) ( 1 ≤ i ≤ 2 n ). The classifier is then defined as

Though in [44] the kernel learned from RFM is used to construct a classifier as in Equation (32), since RFM is a feature learning method, we also apply the RFM kernel to our MMD test, namely

G Application: Diffusion Models vs CIFAR

We defer a more fine-grained detail to our code submission, which includes executable programs (with PyTorch) once the data-generating script from DDPM has been run (see README in the ./codes/CIFAR folder).

̸

G.1 Dataset Details

We use the CIFAR-10 dataset available online at https://www.cs.toronto.edu/~kriz/cifar. html , which contains 50000 colored images of size 32 × 32 with 10 classes. For the diffusion generated images, we use the SOTA Hugging Face model (DDPM) that can be found at https: //huggingface.co/google/ddpm-CIFAR-10-32 . We generated 10000 artificial images for our experiments. The code can be found at our code supplements.

For dataset balancing, we randomly shuffled the CIFAR-10 dataset and used 10000 images as data in our code. Most of our experiments are conducted with the null P X as CIFAR images, and the alternate as P Y = 2 3 · CIFAR + 1 3 · DDPM. To this end, we matched 20000 images from CIFAR to belong to the alternate hypothesis, and the remaining 30000 images to stay in the null hypothesis. For the alternate dataset, we simply sample without replacement from the 20000 + 10000 mixture. This sampled distribution is almost the same as mixing (so long as the sample bank is large enough compared to the acquired data, so that each item in the alternate has close to 1 / 3 probability of being in DDPM, which is indeed the case).

G.2 Experiment Setup and Benchmarks

We use a standard deep Conv-net [40], which has been employed for SOTA GAN discriminator tasks in similar settings. It has four convolutional layers and one fully connected layer outputting the feature space of size (300 , 1) . For SCHE and LBI, we simply added a linear layer of (300 , 2) after applying ReLU to the 300-dimensional layer and used the cross-entropy loss to train the network. Note that this is equivalent to first fixing the feature space and then performing logistic regression to the feature space. For kernels, we add extra trainable parameters after the 300 -d feature output.

̸

For the MMD -based tests, we simply train the kernel on the neural net and evaluate our objective. For UME, we used a slightly generalized version of the original statistic in [31] which allows for comparison on randomly selected witness locations in the null hypothesis with m = n (see Appendix F.5). The kernel is trained using our heuristic (see (8) and Appendix F.2), with MMD replaced by UME. The formula for UME variance can be found in [31]. For RFM, we use Algorithm 1 in [44] to learn a kernel on (stochastic batched) samples, and then use our MMD test on the trained kernel.

We use 80 training epochs for most of our code from the CNN architecture (for classifiers, this is well after interpolating the training data and roughly when validation loss stops decreasing), and a batch size of 32 which has a slight empirical benefit compared to larger batch sizes. The learning rates are tuned separately in MMD methods for optimality, whereas for classifiers they follow the discriminator's original setting from [40]. In Phase 2 of Algorithm 1, we choose k = 1000 for the

50

100

150

200

250

300

350

m

0.0

0.2

0.4

0.6

0.8

1.0

100

200

300

400

m

0.00

0.05

0.10

0.15

0.20

0.25

False Positive Rate

50

100

150

200

250

300

350

m

0.0

0.1

0.2

0.3

0.4

0.5

Expected P-Value

50

100

150

200

250

300

350

m

0.0

0.1

0.2

0.3

0.4

0.5

Error Probability At 0

desired precision while not compromising runtime. For each task, we run 10 independent models and report their performances as the mean and standard deviation of those 10 runs as estimates. We refer to a full set of hyper-parameters in our code implementation.

Our code is implemented in Python 3.7 (PyTorch 1.1) and was ran on an NVIDIA RTX 3080 GPU equipped with a standard torch library and dataset extensions. Our code setup for feature extraction is similar to that of [39]. For benchmark implementations, our code follows from the original code templated provided by the cited papers.

G.3 Sample Allocation

We make a comment on why (4) is different from just thresholding ̂ MMD 2 ( Z, Y tr ) -̂ MMD 2 ( Z, X tr ) at 0, which was what we did in part (c) of Figure 3 (and hence the difference along the curve of MMD-M vs Figure 1.2). Our theory assumes that the samples are i.i.d. conditioned on the kernel being chosen already. However, in the experiments, the kernel is dependent on the training data. Therefore, to evaluate the MMD estimate (between experimentations), one needs extra data that does not intersect with training.

In fact, it can be experimentally shown by comparing Figure 1.2 and Figure 2(c) that doing so (while reducing the sample complexity on n ev ) hurts performance. Indeed, we found out that when X ev , Y ev are non-intersecting with training, performance is (almost) always better at a cost of hurting the overall sample complexity of n .

G.4 Remarks on Results

Figure 6 lists all of our benchmarks in the setting of Figure 3 (in the main text) on missing benchmarks, where the last figure is replaced by the false positive rate at thresholding at α = 0 . 05 to verify our results. As mentioned in the main text, our MMD-M method consistently outperforms other benchmarks on both the expected p -value (of alternate) and rejection rate at α = 0 . 05 , while all of our tests observe a empirical false positive rate close to α = 0 . 05% (Part (b)), showing the consistency of methods.

H Application: Higgs-Boson Detection

H.1 Dataset Details

We use the Higgs dataset available online at http://archive.ics.uci.edu/ml/datasets/ HIGGS , produced using Monte Carlo simulations [5]. The dataset is nearly balanced, containing 5 , 829 , 122 signal instances and 5 , 170 , 877 background instances. Each instance is a 28 -dimensional vector, consisting of 28 features. The first 21 features are kinematic properties measured by the detectors in the accelerator, such as momentum and energy. The last 7 properties are invariant masses , derived from the first 21 features.

0.4

0.6

0.8

1.0

1.2

1.4

1.6

mWbb

(normalized)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequenc of events

Background

Signal

Above threshold

Below threshold

H.2 Experiment Setup and Training Models

The modified Algorithm 1 is shown in Algorithm 2 and Algorithm 3. Compared with Algorithm 2, we implement the thresholding trick (Section 4.3) in Algorithm 3.

H.2.1 Configuration and Model Architecture

We implement all methods in Python 3.9 and PyTorch 1.13 and run them on an NVIDIA Quadro RTX 8000 GPU.

For all classifier-based methods in this study (SCHE and LBI), we adopt the same architecture as previously proposed in [5]. The classifiers are six-layer neural networks with 300 hidden units in each layer, all employing the tanh activation function. For SCHE, the output layer is a single sigmoid unit and we utilize the binary cross-entropy loss for training. For LBI, the output layer is a linear unit and we utilize the binary cross entropy loss combined with a logit function (which is more numerically stable than simply using a sigmoid layer followed by a cross entropy loss).

For all MMD-based methods (MMD-M, MMD-G, MMD-O, and UME), the networks φ and φ ′ are both six-layer neural networks with 300 ReLU units in each layer. The feature space, which is the output of the neural network φ , is set to be 100 -dimensional. Here UME has the same kernel architecture as MMD-M, and the number of test locations is set to be J q = 4096 . For RFM, we adopt the same architecture as in [44], where the kernel is K M ( x, y ) = exp( -γ ( x -y ) T M ( x -y )) with a constant γ and a learnable positive semi-definite matrix M . We set γ ≡ 1 .

The neural networks are initialized using the default setting in PyTorch, and the bandwidths σ, σ ′ are initialized using the median heuristic [22]. The parameter τ is initially set to 0 . 5 . For UME, the witness locations W are initially randomly sampled from the training set. For RFM, the initial M equals the median bandwidth times an identity matrix.

H.2.2 Training

The size of our training set, denoted as n tr , varies from 1 . 0 × 10 2 to 1 . 6 × 10 6 . For a given n tr , we select the first n tr datapoints from each class of the Higgs dataset to form X tr and Y tr , i.e., | X tr | = | Y tr | = n tr . Subsequently, we randomly select n validation = min( √ 10 n tr , 0 . 1 n tr ) points

from each of X tr , Y tr to constitute the validation set, while the remainder of X tr , Y tr are used for running gradient descent. The optimizer is set to be a minibatch SGD, with a batch size of 1024 , a learning rate of 0 . 001 , and a momentum of 0 . 99 . Training is halted once the validation loss stops to decrease for 10 epochs, then we choose the checkpoint (saved for each epoch) with the smallest validation loss thus far as our trained model. Beyond the general setting above, in RFM a batch size of 1024 doesn't work well and instead we use a batch size of 20 , 000 .

H.3 Evaluating the Performance

H.3.1 Evaluating the p-Value with the Methodology of Algorithm 1

We call the 'witness score' of an instance z ∈ X as

For a vector of instances Z = ( Z 1 , . . . , Z m ) , we write

The testing procedure is summarized in Phases 2, 3 and 4 in Algorithm 2 and Algorithm 3. In the Higgs experiment, we utilize the Gaussian approximation method to determine the p-values when the witness function f is not thresholded, which allows us to reach very small p-values and errors under limited computational resource. In cases where the score function f is thresholded by a value t , using the Binomial distribution as in Algorithm 3 is more precise and also fast enough.

Given a trained kernel K trained on X tr and Y tr , we set X ev = X tr and Y ev = Y tr , and accordingly n ev = n tr . This results in a more efficient use of data (since we reuse X tr , Y tr also as X ev , Y ev ). Then, out of the untouched portion of the data, we randomly choose n cal = 20 , 000 datapoints from both classes to populate X cal and Y cal , i.e., | X cal | = | Y cal | = n cal = 20 , 000 . In addition to the general setting above, for RFM, we need to solve a 2 n ev -dimensional linear equation during inference, which arises from the inverse matrix in Equation (32) (solving K ( X RFM , X RFM ) u = ( y RFM ) T for u ∈ R 2 n ev ). So we set n ev = min( n tr , 10 , 000) that X ev , Y ev are randomly sampled from the training set.

In order to compare different benchmarks, we evaluate the expected significance of discovery on a mixture of 1000 backgrounds and 100 signals. For each benchmark and each n tr , we train 10 independent models. Then for each trained model we proceed through the Phases 2, 3 (and 4) in Algorithm 2 and Algorithm 3 by 10 times for 10 different ( X ev , X cal , X opt , Y ev , Y cal , Y opt ) . The mean and standard deviation from these 100 runs are reported in Figure 8.

We also display in Figure 9 the trade-off b ( m,n ev ) and ( m,n tr ) to reach certain levels of significance of discovery in MMD-M. From the bottom left plot, we see that the (averaged) significance is not sensitive to n ev when lg n ev is large. So taking n ev = 20 , 000 is sufficient.

H.3.2 Evaluating the Error of the Test (4)

We set the parameters to be δ = 0 . 1 and π = 1 2 δ in our experiments. As explained Appendix G.3, here we no longer take X ev = X tr . Empirically, taking X ev = X tr yields a very bad threshold γ ( X ev , Y ev , π ) . 4 Instead, X ev is sampled from untouched datapoints other than X tr , and the same applies for Y . We still take n ev = n tr here, resulting in a total size of n ev + n tr = 2 n tr . Specifically, when n ev ≥ 10 , 000 , computing a n ev × n ev Gram matrix becomes computationally expensive, so we adopt Monte Carlo method to compute γ ( X ev , Y ev , π ) , in which we subsample 10 , 000 points from X ev and Y ev to calculate γ and repeat this process 100 times.

Again, we utilize the Gaussian approximation. Recall that the test is to compare T = 1 ∑ m f ( Z i )

m i =1 with γ . The type 1 and type 2 error are estimated as CDF N (0 , 1) ( -γ ( X ev ,Y ev ,π ) -E [ f | H 0 ] √ var( f | H 0 ) /m ) and

CDF N (0 , 1) ( -E [ f | H 1 ] -γ ( X ev ,Y ev ,π ) √ var( f | H 1 ) /m ) for the witness function f , which can be estimated efficiently using the calibration samples X cal , Y cal .

We consider both the regimes of fixing kernels and varying kernels (training kernel based on n ). The results are shown in the top plot in Figure 1.2 and the top plot in Figure 9. For each point on the plot, we train 30 independent models and test each model 10 times, and report the average of these 300 runs. In both plots, we observe the asymmetric m vs n trade-off.

Algorithm 2 Estimate the significance of discovery of an input Z test, using the original statistic

Algorithm 3 Estimate the significance of discovery of an input Z test, applying the thresholding trick

2.6

2.0

1.4

0.8

0.4

0.1

Training set size 2n/million

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Significance of discovery/

MMD-M with

t

opt

MMD-M

MMD-G

MMD-O

UME

SCHE with

t

opt

SCHE with t=0.5

LBI

RFM as classifier

RFM for MMD

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

lg(m)

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

lg(

nev

)

Type I error + II error

6.0

10

12

2.4

10

6

1.5

10

3

1.0

10

2

0.20

0.44

0.66

0.81

0.90

0.95

0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

lg(m)

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

lg(

nev

)

Significance

0.0

3.0

6.0

9.0

12.0

15.0

18.0

21.0

1.0

1.5

2.0

2.5

3.0

3.5

4.0

lg(m)

3.5

4.0

4.5

5.0

5.5

6.0

lg(

ntr

)

Significance

0.0

1.6

3.2

4.8

6.4

8.0

9.6

11.2

12.8

14.4

I Limitations and Future Directions

Finally, we discuss several limitations of our work and raise open questions that we hope will be addressed in future works. From the theoretical side of our arguments, we point out several aspects. First, our upper bound (on the minimax sample complexity) Theorem 3.1 has a likely sub-optimal dependence on α, δ . Second, it might be possible to improve our lower bound to a more natural form by replacing ∥ λ ∥ 2 ,J ⋆ ϵ by ∥ λ ∥ 2 and removing the constraint that the top eigenfunction has to be constant. Third, it remains open to extend our theory to include data-dependent K , as opposed to fixed K .

Empirically, our proposal Algorithm 1 can be inefficient in Phase 2 (prior works such as [39] have used permutation-based arguments for a more efficient estimate), which we adopted due to its simplicity and universality in all benchmarks. Moreover, one might hope that LFHT / mLFHT can be extended to more complex applications, such as text data or videos. Such questions are important to investigate as a future direction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a problem in the area of likelihood-free testing, where one is given black-box access to samples $X\\sim P_X$, $Y \\sim P_Y$ of size n, as well as \"real-world data\" $Z \\sim P_Z$ of size m. The authors propose a new model (\"Mixed likelihood-free testing\") where it is known that $P_Z = (1 - \\nu) P_X + \\nu P_Y$ for rate parameter $\\nu$, and the goal is to determine whether $\\nu = 0$ or $\\nu \\ge \\delta$. They propose a kernel based test statistic for this problem, and analyze its performance theoretically, and also provide lower bounds. They also perform experiments based on Higgs Boson detection and detecting images produced by a diffusion model to support their theoretical claims.\n\nSTRENGTHS:\nThis paper is well written, and the problem is well motivated from the point of view of practical applications (Higgs Boson detection, diffusion image detection). The paper has an extensive set of experiments on real data, and there is an interesting theory to support empirical findings. Overall, I think this is an interesting paper.\n\nWEAKNESSES:\nThe upper and lower bounds in the theory are a little difficult to interpret, especially the dependence on the eigenvalues of the kernel - it would be useful if the authors could provide some examples with simple kernels to illustrate better what the bounds are saying.\n\nSome of the things done in the experiments, such as training a kernel on the test data (especially overlap between training and evaluation data), are not captured/explained by the theory. Also, while Figure 1 seems like it agrees with the theory, it includes all the samples, including ones used for training, calibration, and evaluation. On the other hand, the plot involving only the evaluation samples (Figure 8 in appendix) seems like it agrees more with the naive expectation of $\\min\\{n, m\\} \\gtrsim n_{TS}(\\alpha, \\epsilon, \\mathcal P)$. Since the theory is supposed to reflect performance relative to evaluation samples for a fixed kernel, it would be useful if the authors could provide a more detailed explanation of both these plots. It would also be useful if the authors could provide a simple synthetic experiment to illustrate the theoretical bounds for the setting that it is supposed to explain (with a fixed kernel)\n\nThe dependence on mis-specification parameter $R$ is not captured by the lower bounds, and Proposition 4.1 (which is about the calibration of the test statistic) assumes that $R=0$. I would be curious to know if proposition 4.1 can be made stronger. In particular, can anything be said about the empirical settings where presumably $R > 0$?\n\nOne very interesting thing the authors do in experiments is to use a fraction of the samples from $P_X$ and $P_Y$ to train a kernel that maximizes the (normalized) expectation gap of the statistic. While this seems to work well, there doesn't seem to be a theoretical explanation for why this is a good idea. Of course, while it may not currently be possible to make theoretical claims about a neural net based kernel, is it possible to make a theoretical claim about some simpler model? Or make some abstract claim given black box access to an algorithm that can take samples and generate a kernel with some good properties?\n\nQUESTIONS:\n1) What is the relationship between this problem/likelihood-free testing more generally, and the distribution testing literature in theoretical CS? Can you include a brief comparison?\n\n2) While Proposition 4.1 says that the p-value estimate is good as n_cal-> infty, what is the finite-sample relationship between n_cal, n_eval and the test statistic? Is there something quantitative that can be said here? What happens if $R > 0$?\n\n3) Is it possible to say anything about the kernel training phase theoretically?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework for likelihood-free hypothesis testing (LFHT), denotes as Mixed-LFHT or mLFHT. The setting of mLFHT is as follows: assume $n$ i.i.d. samples from a background disttribution $P_X$, and the same number of samples from a signal distribution $P_Y$. Also, assume we are given another $m$ i.i.d. samples from a mixed distribution $P_Z=\\nu P_Y + \\left(1-\\nu\\right)P_X$, and the aim is to \"test\" and see if $\\nu$ is greater than some positive threshold $\\delta$. Paper proposes a new kernel-based test statistic for this problem and then analyzes it from both theoretical and experimental perspectives. The main idea is to somehow \"map\" the empirical distributions of observed data into an RKHS (for some kernel of choice) and then measure the discrepancies of the observed data from both $P_X$ and $P_Y$.\n\nFrom a theoretical POV, authors claim to acheieve non-trivial sample complexity bounds for the non-asymptotic regime. Both upper and lower-bounds on the error have been established in Thms 3.2 and 3.3, respectively. From an experimental side, authors propose a new algorithm to learn the kernel from data, and also showcase the applicability of their test statistic and algorithm on a number of real-world tasks.\n\nI am not an expert in this particular field and therefore not much aware of other similar works. However, paper is not well-written and seems a little rushed. Regarding the technical contribution, I have a number of questions (please see questions section) before reaching a final conclusion. For now, my vote is borderline reject, with a relatively low confidence.\n\nSTRENGTHS:\n- Paper investigates an important problem for the ML community in general.\n- The mathematical tools (specially those from functional analysis and RKHS-related areas) used for the theoretical parts are sufficiently sophisticated and might be interesting for many researchers in this field.\n- Paper attempts to propose a new setting which comes with both a) theoretical guarantees and b) applicability on real-world data,\n- I did not completely checked the proofs, however, so far I have not found any miscalculations or mathematical flaws.\n\nWEAKNESSES:\n- IMO paper is not well-written, and this might be the biggest weakness of this work. Introduction seems short and not that much informative. Also, transitions from one part or section to another do not seem to be smooth enough. There are some cases where an equation or formula is referred to, but in fact it appears (for the first time) at least two pages later which makes the paper a little confusing for a general reader. Positioning of the work w.r.t prior papers is lacking in some ways, and some assumptions are not sufficiently justified. Overall, it seems that authors have tried to compress too much information into a 9-page limited manuscript which has degraded its readability. I suggest a major reconsideration in the writing, or even maybe considering this work for a journal paper instead of a conference.\n\n- One of the main theoretical contributions, as far as I have understood, is the claimed \"non-trivial\" behavior of the sample complexity bound which is outlined in Theorem 3.2, where authors show a non-symmetic behaviour between $m$ and $n$. However, in L.79 authors mention a similar relation in an existing work (Ref. [18]), which I assume is achieved for a slightly different problem. If this non-symmetric relation has been already discovered, it might affect the contribution level of this paper (please correct me if I am wrong).\n\n- In Thm. 3.3 authors claim to achieve an error lower-bound for mLFHT which matches (more or less) the error upper-bound in Thm 3.2. However, my understanding is that the bound in Thm 3.3 is mainly for the particular kernel-based solution of this paper, and not for mLFHT in general. The actual error lower-bound might be a lot smaller (or not). Again, please correct me if I am wrong. \n\n- I have some questions regarding a number of assumptions and derivation in the paper to see how they might limit the applicability of this solution (please see Questions section).\n\n-----------------------------------------------------------------\n\nMinor comments:\n\n- Caption of Fig. 1: Eq. (4) appears on page 4, but figures is on page 2.\n\n- L.82 to L.89: explanations are vague, please consider some improvements.\n\n- Section 1.2 is a key part of the technical contribution (introducing mLFHT), but there are no prior announcements or motivations for it. Please highlight it more.\n\n- L.105 to L.109: Explanations seem to be non-informative, and can be removed.\n\n- Transition from Section 1.2 to 2.1 is not smooth at all. Also, the math that comes afterwards is unncessarily intense and some of it can be moved to appendix (Specially the \"Removing the Bias\" part, I guess it won't hurt the paper it this part is moved).\n\n- Again, section 2.2 needs more prior advertisement and explanations.\n\n- L.140 to L.151: explanations seem vague.\n\n- L.154: please consider citing some ref. paper for the theorem, otherwise it might look like it is your contribution.\n\n- L.179: Authors claim that this paper imposes less restrictions on the class of distributions for $P_X$ and $P_Y$ compared to other works. I am not completely sure about this claim (see questions), however, it might be a good idea to advertise it more. For example, authors can bring it up in the Introduction section.\n\nQUESTIONS:\n- Please position your work w.r.t Ref. [18], since they also have achieved a similar non-symmetric and non-aasymptotic relation between $m$ and $n$.\n\n- Please justify the assumptions (i) and (iii) in L.165 and L.167, respectively. Assumption (ii) makes sense, but I assume (i) and (iii) might be limiting. Are they necessary in general (are they fundamentally required)? If yes, then please elaborate. If not, please explain how this will affect the applicability of this method in real-workd tasks.\n\n- In Theorem 3.2, how can we deal with $R$? While reading Section 3.5, I noticed $R$ automatically becomes zero in other existing works since they assume $P_Z=P_X$ or $P_Z=P_Y$. This makes the comparison a little troublesome. Again, is the appearance of $R$ inevitable in your equations? or is it a limitation of this work?\n\n- Related to previous question: In L.101 it has been said that $P_Z\\triangleq (1-\\nu)P_X+\\nu P_Y$. Then, in L.163 we have $\\nu=\\arg \\min_{\\nu'} \\mathrm{MMD}(P_Z,(1-\\nu')P_X+\\nu' P_Y)$. Does this mean $R$ is always zero, since $P_Z$ is already assumed to be a convex combination of $P_X$ and $P_Y$? This part is confusing, please clarify.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a likelihood-free hypothesis testing (LFHT) scheme, where the data distribution is a mixture of two distributions. The null hypothesis is given by one of these distributions, whereas the alternative hypothesis is that the mixture coefficient is lower-bounded by some delta.  The motivation for such a scheme is that it is tied to some applications in particle physics, for example, the Higgs boson discovery. This is a generalization of the original LFHT scheme, which is recovered if delta=1. Goal of the paper is to characterize the tradeoff between n, i.e., the size of the dataset and m, the number of samples available from the mixture data distribution available at test time. Specifically, the authors derive lower and upper bounds on the minimal sample complexity. They also empirically verify this trade-off for kernel based hypothesis testing, where the kernels are parameterized via neural networks.\n\nSTRENGTHS:\nAn interesting result is the dependency between n and m for which the authors derive lower and upper bounds, with a non-trivial trade-off  between these quantities. It seems though that this was already addressed in prior work, but this is not my research area and I may be missing other contributions.\n\nWEAKNESSES:\nIt seems that the applications for the proposed mixture LFHT are quite limited. In particular, the Higgs boson discovery is a very specific application, and therefore it is unclear whether this work extends to other key applications.\n\nThe authors mention in Section 4 that the results from Theorem 3.2 cannot be directly applied because of the dependency of data and kernel. This is not clear to me. Isn’t the main idea of a kernel that it is learned from the data? It also seems to be the reason why their proposed generalization bounds cannot be included in Fig. 1 as a reference, which questions the usefulness of these bounds as fundamental limits of the scenario at hand.\n\nThe statement in line 274 seems like a trivial observation, typically the validation set is always chosen much smaller than the training set.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nDoes not apply.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a test statistic that can be used for likelihood-free hypothesis testing [LFHT] (for distributions $P_X, P_Y$, given a sample from $P_Z$, decide if $P_Z = P_X$ versus $P_Z = P_Y$) and mixed likelihood-free hypothesis testing [MLFHT] (for $P_Z = (1 - \\nu) P_X + \\nu P_Y$, decide if $ \\nu = 0 $ versus $ \\nu \\ge \\delta$). \n\nThe test statistic is defined as the MMD distance between $P_Y$ and $P_X$ for an appropriately chosen kernel $K$. The paper then shows an upper and lower bound for these two testing problems. These depend on a base measure $\\mu$ and a kernel $K$, both of which can be chosen, such that if $e_j$ are eigenfunctions of $\\mu$, then $K(x, y) = \\sum_j \\lambda_j e_j(x) e_j(y) $. The bounds are then stated in terms of $\\lambda$ (either $\\mid\\mid \\lambda \\mid \\mid_\\infty $ or $ \\mid \\mid \\lambda \\mid\\mid_2$).\n\nSTRENGTHS:\n- The results are very general with reasonable assumptions. The authors assume that the distributions $P_x, P_y, P_z$ have bounded densities wrt the base measure $\\mu$, and the sample complexity scales linearly with this bound on the density. This is in contrast to existing work, which assumes more descriptive assumptions on these distributions.\n\n- There is empirical support for the asymmetric sample complexity between the train sample $n$ and the test sample $m$.\n\n- The lower bounds are general, in that although they are minimax, they apply for any choice of $\\mu, K$.\n\n- The upper bound allows for distribution mismatch, i.e., the test sample need not be a perfect mixture of $P_X, P_Y$. The bound holds as long as the true distribution is close in MMD to some mixture of $P_x, P_y$.\n\n- This approach also allows for learning a kernel from the data, although the theory bounds don't hold in this case as it violates independence between choice of the kernel and the training samples.\n\nWEAKNESSES:\n- The bounds are a little non-intuitive / roundabout, as they depend only on the base measure $\\mu$ and the kernel $K$, both of which do not directly depend on the hypothesis class containing $P_x, P_y$. From my understanding, the only place the hypothesis class shows up is that it must have bounded density with respect to the measure $\\mu$.\n\n- The novelty of the results over existing work is not explained fully. Is the main contribution of the work bounds for the mixed-distribution testing? Or is the test statistic and the analysis also entirely new?\n\n- As mentioned by the authors, there is gap between the upper and lower bounds for the training set size $n$ and its dependence on the mixture coefficient $\\delta$. However, I think this is acceptable and a good open question.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nThe authors discuss differences between the upper and lower bounds, but apart from that, there's no discussion of limitations. There should be more discussion about drawbacks of this approach wrt existing approaches -- surely there must be some based on how general these results are in comparison to existing work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}