SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Patrik Róbert Gerber*
Department of Mathematics, MIT
Cambridge, MA 02139
prgerber@mit.edu
Yury Polyanskiy*
Department of EECS, MIT
yp@mit.edu
Tianze Jiang*
tjiang@mit.edu
Rui Sun*
eruisun@mit.edu
Given  $n$  observations from two balanced classes, consider the task of labeling an additional  $m$  inputs that are known to all belong to one of the two classes. Special cases of this problem are well-known: with complete knowledge of class distributions ( $n = \infty$ ) the problem is solved optimally by the likelihood-ratio test; when  $m = 1$  it corresponds to binary classification; and when  $m \approx n$  it is equivalent to two-sample testing. The intermediate settings occur in the field of likelihood-free inference, where labeled samples are obtained by running forward simulations and the unlabeled sample is collected experimentally. In recent work it was discovered that there is a fundamental trade-off between  $m$  and  $n$ : increasing the data sample  $m$  reduces the amount  $n$  of training/simulation data needed. In this work we (a) introduce a generalization where unlabeled samples come from a mixture of the two classes - a case often encountered in practice; (b) study the minimax sample complexity for non-parametric classes of densities under maximum mean discrepancy (MMD) separation; and (c) investigate the empirical performance of kernels parameterized by neural networks on two tasks: detection of the Higgs boson and detection of planted DDPM generated images amidst CIFAR-10 images. For both problems we confirm the existence of the theoretically predicted asymmetric  $m$  vs  $n$  trade-off.
1 Likelihood-Free Inference
The goal of likelihood-free inference (LFI) [8; 12; 17; 24], also called simulation-based inference (SBI), is to perform statistical inference in a setting where the data generating process is a black-box, but can be simulated. Given the ability to generate samples  $X_{\theta} \sim P_{\theta}^{\otimes n}$  for any parameter  $\theta$ , and given real-world data  $Z \sim P_{\theta^{\star}}^{\otimes m}$ , we want to use our simulations to learn about the truth  $\theta^{\star}$ . LFI is particularly relevant in areas of science where we have precise but complex laws of nature, for which we can do (stochastic) forward simulations, but can not directly compute the (distribution) density  $P_{\theta}$ . The Bayesian community approached the problem under the name of Approximate Bayesian Computation (ABC) [6; 13; 47]. More recent ML-based methods where regressors and classifiers
are used to summarize data, select regions of interest, approximate likelihoods or likelihood-ratios [14; 29; 30; 42; 49] have also emerged for this challenge.
Despite empirical advances, the theoretical study of frequentist LFI is still in its infancy. We focus on the nonparametric and non-asymptotic setting, which we justify as follows. For applications where tight error control is critical one might be reluctant to rely on asymptotics. More broadly, the non-asymptotic regime can uncover new phenomena and provide insights for algorithm design. Further, parametric models are clearly at odds with the black-box assumption. Recently, [18] proposed likelihood-free hypothesis testing (LFHT) as a simplified model and found minimax optimal tests for a range of nonparametric distribution classes, thereby identifying a fundamental simulation-experimentation trade-off between the number of simulated observations  $n$  and the size of the experimental data sample  $m$ . Here we extend [18], and prior related work [19; 26; 27; 32; 33], to a new setting designed to model experimental setups more truthfully and derive sample complexity (upper and lower bounds) for kernel-based tests over nonparametric classes.
While minimax optimal, the algorithms of [18; 19] are impractical as they rely on discretizing the observations on a regular grid. Thus, both in our theory as well as experiments we turn to kernel methods which provide an empirically more powerful set of algorithms that have shown success in nonparametric testing [20; 21; 22; 31; 39].
Contributions Our contributions are twofold. Theoretically, we introduce mixed likelihood-free hypothesis testing (mLFHT), which is a generalization of (LFHT) and provides a better model of applications such as the search for new physics [11; 38]. We propose a robust kernel-based test and derive both upper and lower bounds on its minimax sample complexity over a large nonparametric class of densities, generalizing multiple results in [18; 19; 26; 27; 32; 33; 37]. Although the simulation-experimentation  $(m\text{vs} n)$  trade-off has been proven in the minimax sense (that is, for some worst-case data distribution), it is not clear whether it actually occurs in real data. Our second contribution is the empirical confirmation of the existence of an asymmetric trade-off, cf. Figure 1.2. To this end we construct state-of-the-art tests building on ideas of [39; 48] on learning good kernels from the data. We execute this program in two settings: the Higgs boson discovery [5], and detecting diffusion [25] generated images planted in the CIFAR-10 [35] dataset.
1.1 LFHT and the Simulation-Experimentation Trade-off
Suppose we have i.i.d. samples  $X, Y$  each of size  $n$  from two unknown distributions  $P_X, P_Y$  on a measurable space  $\mathcal{X}$ , as well as a third i.i.d. sample  $Z \sim P_Z$  of size  $m$ . In the context of LFI, we may think of the samples  $X, Y$  as being generated by our simulator, and  $Z$  being the data collected in the real world. The problem we refer to as likelihood-free hypothesis testing is the task of deciding between the hypotheses
$$
H _ {0}: P _ {Z} = P _ {X} \quad \text {v e r s u s} \quad H _ {1}: P _ {Z} = P _ {Y}. \tag {LFHT}
$$
This problem originates in [23; 52], where authors study the exponents of error decay for finite  $\mathcal{X}$  and fixed  $P_{X}, P_{Y}$  as  $n \sim m \to \infty$ ; more recently [2; 18; 19; 26; 27; 32; 33; 37] it is studied in the non-asymptotic regime. Assuming that  $P_{X}, P_{Y}$  belong to a known nonparametric class of distributions  $\mathcal{P}$  and are guaranteed to be  $\epsilon$ -separated with respect to total variation (TV) distance (i.e.  $\mathrm{TV}(P_X, P_Y) \geq \epsilon$ ), [18] characterizes the sample sizes  $n$  and  $m$  required for the sum of type-I and type-II errors to be small, as a function of  $\epsilon$  and for several different  $\mathcal{P}$ 's. Their results show, for three settings of  $\mathcal{P}$ , that (i) testing (LFHT) at vanishing error is possible even when  $n$  is not large enough to estimate  $P_{X}$  and  $P_{Y}$  within total variation distance  $\mathcal{O}(\epsilon)$ , and that (ii) to achieve a fixed level of error, say  $\alpha$ , one can trade off  $m$  vs.  $n$  along a curve of the form  $\{\min\{n, \sqrt{mn}\} \gtrsim n_{\mathrm{TS}}(\alpha, \epsilon, \mathcal{P}), m \gtrsim \log(1 / \alpha) / \epsilon^2\}$ . Here  $n_{\mathrm{TS}}$  denotes the minimax sample complexity of two-sample testing over  $\mathcal{P}$ , i.e. the minimum number of observations  $n$  needed from  $P_{X}, P_{Y} \in \mathcal{P}$  to distinguish the cases  $\mathrm{TV}(P_X, P_Y) \geq \epsilon$  versus  $P_{X} = P_{Y}$ . Here  $\gtrsim$  suppresses dependence on constants and untracked parameters.
It is unclear, however, whether predictions drawn from minimax sample complexities over specified distribution classes can be observed in real-world data. Without the theory, a natural expectation is that the error contour  $\{(m,n):\exists$  a test with total error  $\leq \alpha \}$  would look similar to that of minimax two-sample testing with unequal sample size, namely  $\{(m,n):\min \{n,m\} \gtrsim n_{\mathrm{TS}}(\alpha ,\epsilon ,\mathcal{P})\}$ , i.e.  $n$  and  $m$  simply need to be above a certain threshold simultaneously (as is the case for e.g. two-sample testing over smooth densities [4; 37]). However, from Figures 2 and 1.2 we see that there is indeed a
non-trivial trade-off between  $n$  and  $m$ : the contours are not always parallel to the axes and aren't symmetric about the line  $m = n$ . The importance of Fig. 1.2 is in demonstrating that said trade-off is not a kink of a theory that arises due to some esoteric worst-case data distribution, but is instead a real effect observed in state-of-the-art LFI algorithms ran on actual data. We remark that the  $n$  used in this plot is the total number of simulated samples (most of which are used for choosing a neural-network parameterized kernel) and are not just the  $n$  occurring in Theorems 3.1 and 3.2 which apply to a fixed kernel. See Section 4 for details on sample division.
1.2 Mixed Likelihood-Free Hypothesis Testing
A prominent application of likelihood-free inference lies in the field of particle physics. Scientists run sophisticated experiments in the hope of finding a new particle or phenomenon. Often said phenomenon can be predicted from theory, and thus can be simulated, as was the case for the Higgs boson whose existence was verified after nearly 50 years at the Large Hadron Collider (LHC) [3; 9].
Suppose we have  $n$  simulations from the background distribution  $P_{X}$  and the signal distribution  $P_{Y}$ . Further, we also have  $m$  (real-world) datapoints from  $P_{Z} = (1 - \nu)P_{X} + \nu P_{Y}$ , i.e. the observed data is a mixture between the background and signal distributions with rate parameter  $\nu$ . The goal of physicists is to construct confidence intervals for  $\nu$ , and a discovery corresponds to a 5σ confidence interval that excludes  $\nu = 0$ . We model this problem by testing
$$
H _ {0}: \nu = 0 \quad \text {v e r s u s} \quad H _ {1}: \nu \geq \delta \quad (\mathrm {m L F H T})
$$
for fixed (usually predicted)  $\delta > 0$ . See the rigorous definition of (mLFHT) in Section 3. In particular, a discovery can be claimed if  $H_0$  is rejected.
2 The Likelihood-Free Test Statistic
This section introduces the testing procedure based on Maximum Mean Discrepancy (MMD) that we study throughout the paper both theoretically and empirically. First, we introduce the necessary background on MMD in Section 2.1. Then, we define our test statistics in Section 2.2.
2.1 Kernel Embeddings and MMD
Given a set  $\mathcal{X}$ , we call the function  $K:\mathcal{X}^2\to \mathbb{R}$  a kernel if the  $n\times n$  matrix with  $ij$ 'th entry  $K(x_{i},x_{j})$  is symmetric positive semidefinite for all choices of  $x_{1},\ldots ,x_{n}\in \mathcal{X}$  and  $n\geq 1$ . There is a unique reproducing kernel Hilbert space (RKHS)  $\mathcal{H}_K$  associated to  $K$ .  $\mathcal{H}_K$  consists of functions  $\mathcal{X}\mapsto \mathbb{R}$  and satisfies the reproducing property  $\langle K(x,\cdot),f\rangle_{\mathcal{H}_K} = f(x)$  for all  $f\in \mathcal{H}_k$  and  $x\in \mathcal{X}$ , in particular  $K(x,\cdot)\in \mathcal{H}_K$ . Given a probability measure  $P$  on  $\mathcal{X}$ , define its kernel embedding  $\theta_P$  as
$$
\theta_ {P} \triangleq \mathbb {E} _ {X \sim P} K (X, \cdot) = \int_ {\mathcal {X}} K (x, \cdot) P (\mathrm {d} x). \tag {1}
$$
Given the kernel embeddings of two probability measures  $P, Q$ , we can measure their distance in the RKHS by  $\mathrm{MMD}(P, Q) \triangleq \| \theta_P - \theta_Q \|_{\mathcal{H}_K}$ , where MMD stands for maximum mean discrepancy. MMD has a closed form thanks to the reproducing property and linearity:
$$
\operatorname {M M D} ^ {2} (P, Q) = \mathbb {E} \left[ K (X, X ^ {\prime}) + K (Y, Y ^ {\prime}) - 2 K (X, Y) \right]
$$
Figure 1:  $n$  versus  $m$  trade-off for the Higgs and CIFAR experiments using our test in Section 2. Error probabilities are estimated by normal approximation for Higgs and simulated for CIFAR.
where  $(X,X^{\prime},Y,Y^{\prime})\sim P^{\otimes 2}\otimes Q^{\otimes 2}$ . In particular, if  $P,Q$  are empirical measures based on observations, we can evaluate the MMD exactly, which is crucial
in practice. Yet another attractive property of MMD is that (under mild integrability conditions) it is an integral probability metric (IPM) where the supremum is over the unit ball of the RKHS  $\mathcal{H}_K$ . See e.g. [41; 46] for references. The following result is a consequence of the fact that self-adjoint compact operators are diagonalizable.
Theorem 2.1 (Hilbert-Schmidt). Suppose that  $K \in L^{2}(\mu \otimes \mu)$  is symmetric. Then there exists a sequence  $(\lambda_j)_{j \geq 1} \in \ell^2$  and an orthonormal basis  $\{e_j\}_{j \geq 1}$  of  $L^2(\mu)$  such that  $K(x, y) = \sum_{j \geq 1} \lambda_j e_j(x) e_j(y)$  for all  $j \geq 1$ , where convergence is in  $L^2(\mu \otimes \mu)$ .
Assumption 1. Unless specified otherwise, we implicitly assume a choice of a non-negative measure  $\mu$  and kernel  $K$  for which the conditions of Theorem 2.1 hold. Note that  $\lambda_{j} \geq 0$  and depend on  $\mu$ .
Removing the Bias In our proofs we work with the kernel embedding of empirical measures for which we need to modify the inner product  $\langle \cdot, \cdot \rangle_{\mathcal{H}_K}$  (and thus MMD) slightly by removing the diagonal terms. Namely, given i.i.d. samples  $X, Y$  of size  $n, m$  respectively and corresponding empirical measures  $\widehat{P}_X, \widehat{P}_Y$ , we define
$$
\mathrm {M M D} _ {u} ^ {2} \left(\widehat {P} _ {X}, \widehat {P} _ {Y}\right) \triangleq \sum_ {i \neq j} \frac {K \left(X _ {i} , X _ {j}\right)}{n (n - 1)} + \sum_ {i \neq j} \frac {K \left(Y _ {i} , Y _ {j}\right)}{m (m - 1)} - 2 \sum_ {i, j} \frac {K \left(X _ {i} , Y _ {j}\right)}{m n}. \tag {2}
$$
We also write  $\langle \theta_{\widehat{P}_X},\theta_{\widehat{P}_X}\rangle_u,\mathcal{H}_K\triangleq \| \theta_{\widehat{P}_X}\|_u^2$  and extend linearly. The  $u$  stands for unbiased, since  $\mathbb{E}\mathrm{MMD}_u^2 (\widehat{P}_X,\widehat{P}_Y) = \mathrm{MMD}^2 (P_X,P_Y)\neq \mathbb{E}\mathrm{MMD}^2 (\widehat{P}_X,\widehat{P}_Y)$  in general.
2.2 Test Statistic
With Section 2.1 behind us, we are in a position to define the test statistic that we use to tackle (mLFHT). Suppose that we have samples  $X, Y, Z$  of sizes  $n, n, m$  from the probability measures  $P_{X}, P_{Y}, P_{Z}$ . Write  $\hat{P}_{X}$  for the empirical measure of sample  $X$ , and analogously for  $Y, Z$ . The core of our test statistic for (mLFHT) is the following:
$$
T (X, Y, Z) \triangleq \left\langle \theta_ {\widehat {P} _ {Z}}, \theta_ {\widehat {P} _ {Y}} - \theta_ {\widehat {P} _ {X}} \right\rangle_ {u, \mathcal {H} _ {K}} = \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \left\{K \left(Z _ {j}, Y _ {i}\right) - K \left(Z _ {j}, X _ {i}\right) \right\}. \tag {3}
$$
Note that  $T$  is of the additive form  $\frac{1}{m}\sum_{j = 1}^{m}f(Z_i)$  where  $f(z)\triangleq \theta_{\widehat{P}_Y}(z) - \theta_{\widehat{P}_X}(z)$  can be interpreted as the witness function of [21; 31]. Given some  $\pi \in [0,1]$  (taken to be half the predicted signal rate  $\delta /2$  in our proofs), the output of our test is
$$
\Psi_ {\pi} = \mathbb {1} \left\{T (X, Y, Z) \geq \gamma (X, Y, \pi) \right\}, \text {w h e r e} \gamma (X, Y, \pi) = \pi \operatorname {M M D} _ {u} ^ {2} (\widehat {P} _ {X}, \widehat {P} _ {Y}) + T (X, Y, X). \tag {4}
$$
The threshold  $\gamma$  gives  $\Psi_{\pi}$  a natural geometric interpretation: it checks whether the projection of  $\theta_{\widehat{P}_Z} - \theta_{\widehat{P}_X}$  onto the vector  $\theta_{\widehat{P}_Y} - \theta_{\widehat{P}_X}$  falls further than  $\pi$  along the segment joining  $\theta_{\widehat{P}_X}$  to  $\theta_{\widehat{P}_Y}$  (up to deviations due to the omitted diagonal terms, see Section 2.1).
Setting  $\delta = 1$  in (mLFHT) recovers (LFHT), and the corresponding test output is  $\Psi_{\delta /2} = \Psi_{1 / 2} = 1$  if and only if  $\mathrm{MMD}_u(\widehat{P}_Z,\widehat{P}_X)\geq \mathrm{MMD}_u(\widehat{P}_Z,\widehat{P}_Y)$ . This very statistic (i.e.  $\mathrm{MMD}_u(\widehat{P}_Z,\widehat{P}_X) - \mathrm{MMD}_u(\widehat{P}_X,\widehat{P}_Y)$ ) has been considered in the past for relative goodness-of-fit testing [7] where it's asymptotic properties are established. In the non-asymptotic setting, if MMD is replaced by the  $L^2$ -distance we recover the test statistic studied by [18; 27; 33]. However, we are the first to introduce  $\Psi_{\delta}$  for  $\delta \neq 1$  and to study MMD-based tests for (m)LFHT in a non-asymptotic setting.
Variance cancellation At first sight it may seem more natural to the reader to threshold the distance  $\mathrm{MMD}_u(\widehat{P}_Z,\widehat{P}_X)$ , resulting in rejection if, say,  $\mathrm{MMD}_u(\widehat{P}_Z,\widehat{P}_X) \geq \mathrm{MMD}_u(\widehat{P}_X,\widehat{P}_Y)\delta / 2$ . The geometric meaning of this would be similar to the one outlined above. However, there is a crucial difference: (LFHT) (the case  $\delta = 1$ ) is possible with very little experimental data  $m$  due to the cancellation of variance. More precisely, the statistic  $\mathrm{MMD}^2 (\widehat{P}_Z,\widehat{P}_X)$  contains the term  $\frac{1}{m(m - 1)}\sum_{i\neq j}K(Z_i,Z_j)$  — whose variance is prohibitively large and would inflate the  $m$  required for reliable testing — but this can be canceled by subtracting  $\mathrm{MMD}^2 (\widehat{P}_Z,\widehat{P}_Y)$ . Our statistic  $T(X,Y,Z) - \gamma (X,Y,\pi)$  simply generalizes this idea to (mLFHT).
3 Minimax Rates of Testing
3.1 Upper Bounds on the Minimax Sample Complexity of (mLFHT)
Let us start by reintroducing (mLFHT) in a rigorous fashion. Given  $C, \epsilon, R \geq 0$ , let  $\mathcal{P}_{\mu}(C, \epsilon, R)$  denote the set of triples  $(P_X, P_Y, P_Z)$  of distributions such that the following three conditions hold:
(i)  $P_{X}, P_{Y}$  and  $P_{Z}$  have  $\mu$ -densities bounded by  $C$
(ii)  $\mathrm{MMD}(P_X, P_Y) \geq \epsilon$ , and
(iii)  $\mathrm{MMD}(P_Z,(1 - \nu)P_X + \nu P_Y)\leq R\cdot \mathrm{MMD}(P_X,P_Y),$
where we define  $\nu = \nu(P_X, P_Y, P_Z) = \arg \min_{\nu' \in \mathbb{R}} \mathrm{MMD}(P_Z, (1 - \nu') P_X + \nu' P_Y)$ . For some  $\delta > 0$ , consider the two hypotheses
$$
H _ {0} (C, \epsilon , \delta , R): \left(P _ {X}, P _ {Y}, P _ {Z}\right) \in \mathcal {P} _ {\mu} (C, \epsilon , R) \text {a n d} \nu = 0
$$
$$
\begin{array}{l} H _ {0} (C, \epsilon , \delta , R): (P _ {X}, P _ {Y}, P _ {Z}) \in \mathcal {P} _ {\mu} (C, \epsilon , R) \text {a n d} \nu = 0 \\ H _ {1} (C, \epsilon , \delta , R): (P _ {X}, P _ {Y}, P _ {Z}) \in \mathcal {P} _ {\mu} (C, \epsilon , R) \text {a n d} \nu \geq \delta , \end{array} \tag {5}
$$
which we regard as subsets of probability measures. Notice that  $R$  controls the level of misspecification in the direction that is orthogonal to the line connecting the kernel embeddings of  $P_{X}$  and  $P_{Y}$ . Setting  $R = 0$  simply asserts that  $P_{Z}$  is guaranteed to be a mixture of  $P_{X}$  and  $P_{Y}$ , as is the case for prior works on LFHT. Before presenting our main result on the minimax sample complexity of mLFHT, let us define one final piece of terminology. We say that a test  $\Psi$ , which takes some data as input and takes values in  $\{0,1\}$ , has total error probability less than  $\alpha$  for the problem of testing  $H_{0}$  vs  $H_{1}$  if
$$
\sup  _ {P \in H _ {0}} P (\Psi = 1) + \sup  _ {Q \in H _ {1}} Q (\Psi = 0) \leq \alpha . \tag {6}
$$
Theorem 3.1. Suppose we observe three i.i.d. samples  $X, Y, Z$  from distributions  $P_{X}, P_{Y}, P_{Z}$  composed of  $n, n, m$  observations respectively and let  $C \in (0, \infty)$  and  $R, \epsilon \geq 0$  and  $\delta \in (0, 1)$ . There exists a universal constant  $c > 0$  such that  $\Psi_{\delta/2}$  defined in Section 2.2 tests  $H_{0}$  vs  $H_{1}$ , as defined in (5), at total error  $\alpha$  provided
$$
\min \{m, n \} \geq c \frac {C \| \lambda \| _ {\infty} \log (1 / \alpha)}{(\epsilon \delta / (1 + R)) ^ {2}} \quad a n d \quad \min \{n, \sqrt {n m} \} \geq c \frac {C \| \lambda \| _ {2} \log (1 / \alpha)}{\delta \epsilon^ {2}}.
$$
Note that Theorem 3.1 does not place assumptions on the distributions  $P_{X}, P_{Y}$  beyond bounded density with respect to the base measure  $\mu$ . This is different from usual results in statistics, where prior specification of distribution classes is crucial. On the other hand, instead of standard distances such as  $L^{p}$ , we assume separation with respect to MMD and the latter is potentially harder to interpret than, say,  $L^{1}$  i.e. total variation. We do point out that our Theorem 3.1 can be used to derive results in the classical setting; we discuss this further in Section 3.4.
In an appropriate regime of the parameters, the sufficient sample complexity in Theorem 3.1 exhibits a trade-off of the form  $\min \{n,\sqrt{mn}\} \gtrsim \| \lambda \|_2\log (1 / \alpha) / (\delta \epsilon^2)$  between the number of simulation samples  $n$  and real observations  $m$ . This trade-off is shown in Figure 2 using data from a toy problem. The trade-off is clearly asymmetric and the relationship  $m\cdot n\geq const.$  also seems to appear. In this toy problem we set  $R = 0$ ,  $\delta = 1$ ,  $\epsilon = .3$ ,  $k = 100$  and
Figure 2:  $n$  versus  $m$  trade-off for the toy experiment, verifying Theorem 3.1. Probabilities estimated over  $10^{4}$  runs, and smoothed using Gaussian noise.
$P_{X} = P_{Z}, P_{Y}$  are distributions on  $\{1,2,\ldots,k\}$  with  $P_{X}(i) = (1 + \epsilon \cdot (2 \cdot \mathbb{1}\{i \text{ odd}\} - 1)) / k = 2 / k - P_{Y}(i)$  for all  $i = 1,2,\ldots,k$ . The kernel we take is  $K(x,y) = \sum_{i=1}^{k} \mathbb{1}\{x = y = i\}$  and  $\mu$  is simply the counting measure; the resulting MMD is simply the  $L^2$ -distance on pmfs.
Figure 1.2 illustrates a larger scale experiment using real data using a trained kernel. Note that we plot the total number  $n$  of simulation samples, including those used for training the kernel itself (see Section 4); which ensures that Figure 1.2 gives a realistic picture of data requirements. However, due to the dependence between the kernel and the data, Theorem 3.1 no longer applies. Nevertheless, we observe a trade-off similar to Figure 2.
3.2 Lower Bounds on the Minimax Sample Complexity of (mLFHT)
In this section we prove a minimax lower bound on the sample complexity of mLFHT, giving a partial converse to Theorem 3.1. Before we can state this results, we must make some technical definitions. Given  $J \geq 2$ , let  $\| \lambda \|_{2J}^2 \triangleq \sum_{j=2}^{J} \lambda_j^2$  and define
$$
J _ {\epsilon} ^ {\star} \triangleq \max  \left\{J: \sup  _ {\eta_ {j} = \pm 1} \left\| \sum_ {j = 2} ^ {J} \eta_ {j} \sqrt {\lambda_ {j}} e _ {j} \right\| _ {\infty} \leq \frac {\| \lambda \| _ {2 J}}{2 \epsilon} \right\}.
$$
Theorem 3.2 (Lower Bounds for mLFHT). Suppose that  $\int_{\mathcal{X}} K(x, y) \mu(\mathrm{d}x) \equiv \lambda_1$ ,  $\mu(\mathcal{X}) = 1$  and  $\sup_{x \in \mathcal{X}} K(x, x) \leq 1$ . There exists a universal constant  $c > 0$  such that any test of  $H_0$  vs  $H_1$ , as defined in (5), with total error at most  $\alpha$  must use a number  $(n, m)$  of observations that satisfy
$$
m \geq c \frac {\lambda_ {2} \log (1 / \alpha)}{\epsilon^ {2} \delta^ {2}} \quad a n d \quad n \geq c \frac {\| \lambda \| _ {2 J _ {\epsilon} ^ {\star}} \sqrt {\log (1 / \alpha)}}{\epsilon^ {2}} \quad a n d \quad \delta m + \sqrt {m n} \geq c \frac {\| \lambda \| _ {2 J _ {\epsilon} ^ {\star}} \sqrt {\log (1 / \alpha)}}{\epsilon^ {2} \delta}.
$$
Remark 3.3. Recall that the eigenvalues  $\lambda$  depend on the choice of  $\mu$ , so that by choosing a different base measure  $\mu$  one can optimize the lower bound. However, since  $P_{X}, P_{Y}, P_{Z}$  are assumed to have bounded density with respect to  $\mu$ , this appears rather involved.
Remark 3.4. The requirements  $\sup_{x\in \mathcal{X}}K(x,x)\leq 1$  and  $\mu (\mathcal{X}) = 1$  are is essentially without loss of generality, as  $\mu$  and  $K$  can be rescaled. The condition  $\int_{\mathcal{X}}K(x,y)\mu (\mathrm{d}x)\equiv \lambda_1$  implies that the top eigenfunction  $e_1$  is equal to a constant or equivalently, that  $y\mapsto K(x,y)\mu (\mathrm{d}x)$  defines a Markov kernel up to a normalizing constant.
3.3 Tightness of Theorems 3.1 and 3.2