SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Aad, G., Abajyan, T., Abbott, B., Abdallah, J., Khalek, S. A., Abdelalim, A. A., Aben, R., Abi, B., Abolins, M., AbouZeid, O., et al. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Physics Letters B, 716(1):1-29, 2012.
[2] Acharya, J., Das, H., Jafarpour, A., Orlitsky, A., Pan, S., and Suresh, A. Competitive classification and closeness testing. In Conference on Learning Theory, pages 22-1. JMLR Workshop and Conference Proceedings, 2012.
[3] Adam-Boundarios, C., Cowan, G., Germain, C., Guyon, I., Kégl, B., and Rousseau, D. The Higgs boson machine learning challenge. In NIPS 2014 workshop on high-energy physics and machine learning, pages 19-55. PMLR, 2015.
[4] Arias-Castro, E., Pelletier, B., and Saligrama, V. Remember the curse of dimensionality: The case of goodness-of-fit testing in arbitrary dimension. Journal of Nonparametric Statistics, 30 (2):448-471, 2018.
[5] Baldi, P., Sadowski, P., and Whiteson, D. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
[6] Beaumont, M. A. Approximate bayesian computation. Annual review of statistics and its application, 6:379-403, 2019.
[7] Bounliphone, W., Belilovsky, E., Blaschko, M. B., Antonoglou, I., and Gretton, A. A test of relative similarity for model selection in generative models. In Bengio, Y. and LeCun, Y., editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[8] Brehmer, J., Louppe, G., Pavez, J., and Cranmer, K. Mining gold from implicit models to improve likelihood-free inference. Proceedings of the National Academy of Sciences, 117(10): 5242-5249, 2020.
[9] Chatrchyan, S., Khachatryan, V., Sirunyan, A. M., Tumasyan, A., Adam, W., Aguilo, E., Bergauer, T., Dragicevic, M., Ero, J., Fabjan, C., et al. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Physics Letters B, 716(1):30-61, 2012.
[10] Cheng, X. and Cloninger, A. Classification logit two-sample testing by neural networks. CoRR, abs/1909.11298, 2019.
[11] Cowan, G., Cranmer, K., Gross, E., and Vitells, O. Asymptotic formulae for likelihood-based tests of new physics. The European Physical Journal C, 71(2):1-19, 2011.
[12] Cranmer, K., Brehmer, J., and Louppe, G. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48):30055-30062, 2020.
[13] Csilléry, K., Blum, M. G., Gaggiotti, O. E., and François, O. Approximate bayesian computation (abc) in practice. Trends in ecology & evolution, 25(7):410-418, 2010.
[14] Dalmasso, N., Izbicki, R., and Lee, A. Confidence sets and hypothesis testing in a likelihood-free inference setting. In International Conference on Machine Learning, pages 2323-2334. PMLR, 2020.
[15] Devroye, L. and Lugosi, G. Combinatorial Methods in Density Estimation. Springer Series in Statistics. Springer New York, 2012. ISBN 9781461301257.
[40] Lopez-Paz, D. and Oquab, M. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017.
[41] Muandet, K., Fukumizu, K., Striperumbudur, B., Scholkopf, B., et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning, 10 (1-2):1-141, 2017.
[42] Papamakarios, G., Sterratt, D., and Murray, I. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 837-848. PMLR, 2019.
[43] Polyanskiy, Y. and Wu, Y. Information Theory: From Coding to Learning. Cambridge University Press, 2023+.
[44] Radhakrishnan, A., Beaglehole, D., Pandit, P., and Belkin, M. Feature learning in neural networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212.13881, 2022.
[45] Sackrowitz, H. and Samuel-Cahn, E. P values as random variables-expected p values. The American Statistician, 53(4):326–331, 1999. ISSN 00031305.
[46] Scholkopf, B. and Smola, A. J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press, 06 2001. ISBN 9780262256933. doi: 10.7551/mitpress/4175.001.0001.
[47] Sisson, S. A., Fan, Y., and Beaumont, M. Handbook of approximate Bayesian computation. CRC Press, 2018.
[48] Sutherland, D. J., Tung, H., Strathmann, H., De, S., Ramdas, A., Smola, A. J., and Gretton, A. Generative models and model criticism via optimized maximum mean discrepancy. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[49] Thomas, O., Dutta, R., Corander, J., Kaski, S., and Gutmann, M. U. Likelihood-free inference by ratio estimation. Bayesian Analysis, 17(1):1-31, 2022.
[50] Tsybakov, A. B. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.
[51] von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
[52] Ziv, J. On classification with empirically observed statistics and universal data compression. IEEE Transactions on Information Theory, 34(2):278-286, 1988.
We use  $A \gtrsim B, A \lesssim B, A \asymp B$  to denote  $A = \Omega(B), B = \Omega(A)$  and  $A = \Theta(B)$  respectively, where the hidden constants depend on untracked parameters multiplicatively.
We write TV, KL,  $\chi^2$  for total-variation, KL-divergence and  $\chi^2$ -divergence, respectively. We write  $D(P_{Y|X} \| Q_{Y|X} | P_X) = \mathbb{E}_{X \sim P_X} D(P_{Y|X} \| Q_{Y|X})$  as the conditional divergence for any probability measures  $P, Q$  on two variables  $X, Y$  and divergence  $D \in \{\mathrm{TV}, \mathrm{KL}, \chi^2\}$ .
We write  $\ell^p$  for the usual  $\ell^p$  sequence space and  $L^p$  for the usual  $L^p$  space with respect to the Lebesgue measure. Both the  $\ell^p$  norm and the  $L^p$  norm are written as  $\| \cdot \|_p$  if no ambiguity arises.
For real numbers  $a, b \in \mathbb{R}$  we also write  $\max \{a, b\}$  as  $a \vee b$  and  $\min \{a, b\}$  as  $a \wedge b$ .
We use  $\vec{1}_d$  to denote an  $d$ -dimensional all 1's vector.
For an integer  $k \in \mathbb{Z}^+$ , we write  $[k]$  as a short notation for the set  $\{1, 2, \dots, k\}$ .
In the proofs of Theorem 3.1 and Theorem 3.2, we use  $\frac{!}{\text{一}}$  for an equality that we are trying to prove.
B Applications of Theorem 3.1
Usually, minimax rates of testing are proven under separation assumptions using more traditional measures of distance such as  $L^p$ , where  $p \in [1,\infty]$ . In this section we show one example of how Theorem 3.1 can be used to recover known results, and also obtain some novel results under  $L^2$ -separation and  $L^1$ -separation.
B.1 Bounded Discrete Distributions Under  $L^2 / L^1$ -Separation
Sample Complexity Upper Bounds Let  $\mathcal{P}_{\mathrm{Db}}(k,C)$  be the set of all discrete distributions  $P$  supported on  $[k] = \{1,2,\dots ,k\}$  satisfying  $\max_{1\leq i\leq k}p(i)\leq C / k$ , where  $p$  is the probability mass function of  $P$  (here  $\sum_{i = 1}^{k}p(k) = 1$ ). For distributions  $P_{X},P_{Y},P_{Z}$  we shall write  $p_X,p_Y,p_Z$  as their probability mass functions, respectively.
Let us apply Theorem 3.1 with underlying space  $\mathcal{X} = [k]$  and measure  $\mu = \frac{1}{k}\sum_{i=1}^{k}\delta_{i}$ . Take the kernel  $K(x,y) = \mathbb{1}\{x = y\} = \sum_{i=1}^{k}\mathbb{1}\{x = y = i\}$ , and note that for any two distributions  $P_{X}, P_{Y}$  we have
$$
\mathrm {M M D} ^ {2} (P _ {X}, P _ {Y}) = \mathbb {E} \left[ K (X, X ^ {\prime}) + K (Y, Y ^ {\prime}) - 2 K (X, Y) \right] = \sum_ {i} | p _ {X} (i) - p _ {Y} (i) | ^ {2}
$$
where  $(X,X^{\prime},Y,Y^{\prime})\sim P_{X}^{\otimes 2}\otimes P_{Y}^{\otimes 2}$ . So the corresponding MMD is the  $\ell^2$ -distance on probability mass functions. Note also that  $K = \sum_{i = 1}^{k}\frac{1}{k}\left(\sqrt{k}\mathbb{1}\{x = i\}\right)\left(\sqrt{k}\mathbb{1}\{y = i\}\right)$ , where  $\left\{\sqrt{k}\mathbb{1}\{x = i\}\right\}_{i = 1}^{k}$  forms an orthonormal basis of  $L^2 (\mu)$ . So  $K$  has only one nonzero eigenvalue, namely
$$
\lambda_ {1} = \lambda_ {2} = \ldots = \lambda_ {k} = 1 / k,
$$
of multiplicity  $k$ . Suppose that we observe samples  $X, Y, Z$  of size  $n, n, m$  from  $P_X, P_Y, P_Z \in \mathcal{P}_{\mathrm{Db}}(k, C)$ , where  $\mathrm{MMD}(P_X, P_Y) = \sqrt{\sum_i |p_X(i) - p_Y(i)|^2} \geq \epsilon$ . Plugging into Theorem 3.1 shows that:
2For example, the first equation in (10) means that there exists a constant  $c$  independent of  $\alpha ,k,\epsilon ,\delta ,R$  such that  $\min \{m,n\} \geq c\frac{\log(1 / \alpha)(1 + R)^2}{k\epsilon^2\delta^2}$
Proposition B.1. For any two  $P_{X}, P_{Y} \in \mathcal{P}_{\mathrm{Db}}(k, C)$ , if the  $\ell^2$ -distance between  $p_{X}, p_{Y}$  is at least  $\epsilon$ , then testing (mLFHT) is possible at total error  $\alpha$  using  $n$  simulation samples and  $m$  real data samples provided that
$$
\min  \{m, n \} \gtrsim \frac {C \| \lambda \| _ {\infty} \log (1 / \alpha) (1 + R) ^ {2}}{\delta^ {2} \epsilon^ {2}} \asymp \frac {\log (1 / \alpha) (1 + R) ^ {2}}{k \epsilon^ {2} \delta^ {2}},
$$
$$
\min  \{n, \sqrt {m n} \} \gtrsim \frac {C \| \lambda \| _ {2} \sqrt {\log (1 / \alpha)}}{\epsilon^ {2} \delta} \asymp \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2} \delta}.
$$
where  $R$  is defined as in the assumption (iii) of Section 3.1.
We can convert the above results to measure separation with respect to total variation (recall  $\mathrm{TV}(p,q) = \frac{1}{2}\sum_{i}|p(i) - q(i)| = \frac{1}{2}\| p - q\| _1)$  using the AM-QM inequality  $\| p_X - p_Y\| _1\leq \sqrt{k}\| p_X - p_Y\| _2$  . Then, taking  $R\asymp \alpha \asymp \delta = \Theta (1)$  recovers the minimax optimal results of [18; 32; 33], for LFHT over the class  $\mathcal{P}_{\mathrm{Db}}$  . Note that analogous results for two-sample testing follow from the above using the reduction presented in Section 3.4.
Sample Complexity Lower Bounds Recall the definition of  $J_{\epsilon}^{\star}$  and note that  $\| \lambda \|_{2,J}^{2} = \frac{\min(J - 1,k)}{k^{2}}$  for all  $J \geq 2$ . By Corollary 3.5 we see that  $J_{\epsilon}^{\star} \gtrsim k$  as soon as  $\epsilon \lesssim 1 / k$ . Thus, for  $\epsilon \lesssim 1 / k$  the necessity of
$$
m \gtrsim \frac {\log (1 / \alpha)}{k \epsilon^ {2} \delta^ {2}}, n \gtrsim \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2}} \text {a n d} m + \sqrt {m n} \gtrsim \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2} \delta} \tag {11}
$$
follows by Theorem 3.2. Here it is crucial to note that when  $\delta = \Theta(1)$ , we have
$$
m + \sqrt {m n} \gtrsim \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2}} \text {a n d} n \gtrsim \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2}} \iff \sqrt {m n} \gtrsim \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2}} \text {a n d} n \gtrsim \frac {\sqrt {\log (1 / \alpha)}}{\sqrt {k} \epsilon^ {2}}
$$
and hence the upper bound (10) meets with the lower bound (11) provided  $R \asymp \delta = \Theta(1)$ . Once again, setting  $R \asymp \delta \asymp \alpha = \Theta(1)$  we the optimal lower bounds recovering the results of [18] (in the regime  $\epsilon \lesssim 1/k$ ). In short we can also recover the following result for LFHT.
Proposition B.2 ([18, Theorem 1, adapted]). On the class  $P_{\mathrm{Db}}(k,C)$ , using  $n$  simulation samples and  $m$  real data samples, if
$$
n \gtrsim \frac {1}{\sqrt {k} \epsilon^ {2}}, \quad m \gtrsim \frac {1}{k \epsilon^ {2}}, \quad \sqrt {m n} \gtrsim \frac {1}{\sqrt {k} \epsilon^ {2}}, \tag {12}
$$
then for any two distributions  $P_{X}, P_{Y} \in \mathcal{P}_{\mathrm{Db}}(k, C)$  with  $\| p_{X} - p_{Y} \|_{2} \geq \epsilon$ , testing (LFHT) is possible with a total error of  $1\%$ . Conversely, to ensure the existence of a procedure that can test (LFHT) with a total error of  $1\%$  for any  $P_{X}, P_{Y} \in \mathcal{P}_{\mathrm{Db}}(k, C)$  with  $\| p_{X} - p_{Y} \|_{2} \geq \epsilon$ , the number of observations  $(n, m)$  must satisfy
$$
n \gtrsim \frac {1}{\sqrt {k} \epsilon^ {2}}, \quad m \gtrsim \frac {1}{k \epsilon^ {2}}, \quad \sqrt {m n} \gtrsim \frac {1}{\sqrt {k} \epsilon^ {2}}. \tag {13}
$$
The implied constants in (12) and (13) do not depend on  $k$  and  $\epsilon$ , but may differ.
B.2  $\beta$ -Holder Smooth Densities on  $[0,1]^d$  Under  $L^2 / L^1$ -Separation
Sample Complexity Upper Bounds Let  $\mathcal{P}_{\mathrm{H}}(\beta, d, C)$  be the set of all distributions on  $[0, 1]^d$  with  $\beta$ -Hölder smooth Lebesgue-density  $p$  satisfying  $\| p \|_{\mathcal{C}^{\beta}} \leq C$  for some constant  $C > 1$ , where
$$
\| p \| _ {\mathcal {C} ^ {\beta}} \triangleq \max  _ {0 \leq | \alpha | \leq \lceil \beta - 1 \rceil} \| f ^ {(\alpha)} \| _ {\infty} + \sup  _ {x \neq y \in [ 0, 1 ] ^ {d}, | \alpha | = \lceil \beta - 1 \rceil} \frac {| f ^ {(\alpha)} (x) - f ^ {(\alpha)} (y) |}{\| x - y \| _ {2} ^ {\beta - \lceil \beta - 1 \rceil}},
$$
where  $\lceil \beta -1\rceil$  is the largest integer strictly smaller than  $\beta$  and  $|\alpha | = \sum_{i}\alpha_{i}$  is the norm of a multi-index  $\alpha \in \mathbb{N}^d$ . Abusing notation, we also use  $\mathcal{P}_{\mathsf{H}}(\beta ,d,C)$  to denote the set of all corresponding density functions.
We take  $K(x,y) = \sum_{j} \mathbb{1}\{x,y \in B_j\}$ , where  $\{B_j\}_{j \in [\kappa]^d}$  is the  $j$ 'th cell of the regular grid of size  $\kappa^d$  on  $[0,1]^d$ , i.e.,  $B_j = [(j - \vec{1}_d) / \kappa, j / \kappa]$  for  $j \in [\kappa]^d$ . Clearly there are  $\kappa^d$  nonzero eigenvalues, each equal to 1. The following approximation result is due to Ingster [28], see also [4, Lemma 7.2].
Lemma B.3. Let  $f, g \in \mathcal{P}_{\mathsf{H}}(\beta, d, C)$  with  $\| f - g \|_2 \geq \epsilon$ . Then, there exist constants  $c, c'$  independent of  $\epsilon$  such that for any  $\kappa \geq c \epsilon^{-1/\beta}$ ,
$$
\operatorname {M M D} (f, g) \geq c ^ {\prime} \| f - g \| _ {2}.
$$
Now, suppose that we have samples  $X,Y,Z$  of size  $n,n,m$  from  $P_{X},P_{Y},P_{Z}\in \mathcal{P}_{\mathsf{H}}(\beta ,d,C)$  with densities  $p_X,p_Y,p_Z$  such that  $\| p_X - p_Y\| _2\geq \epsilon$ . Then, Theorem 3.1 combined with Lemma B.3 and the choice  $\kappa \asymp \epsilon^{-1 / \beta}$  shows that
Proposition B.4. Testing (mLFHT) on  $\mathcal{P}_{\mathrm{H}}(\beta, d, C)$  at total error  $\alpha$  using  $n$  simulation and  $m$  real data samples is possible provided
$$
\min  \{m, n \} \gtrsim \frac {C \| \lambda \| _ {\infty} \log (1 / \alpha) (1 + R) ^ {2}}{\delta^ {2} \epsilon^ {2}} \asymp \frac {\log (1 / \alpha) (1 + R) ^ {2}}{\delta^ {2} \epsilon^ {2}},
$$
$$
\min \{n, \sqrt {n m} \} \gtrsim \frac {C \| \lambda \| _ {2} \sqrt {\log (1 / \alpha)}}{\epsilon^ {2} \delta} \asymp \frac {\sqrt {\log (1 / \alpha)}}{\epsilon^ {(2 \beta + d / 2) / \beta} \delta},
$$
where  $\epsilon$  is an  $L^2$ -distance lower bound between  $P_X, P_Y$  and  $R$  is defined as in the assumption (iii) of Section 3.1.
Setting  $R \asymp \alpha \asymp \delta = \Theta(1)$  recovers the optimal results of [18] for the class  $\mathcal{P}_{\mathsf{H}}$ . Once again, identical results under  $L^1$  separation follow from Jensen's inequality  $\| \cdot \|_{L^1([0,1]^d)} \leq \| \cdot \|_{L^2([0,1]^d)}$ . Note that analogous results for two-sample testing follow from the above using the reduction presented in Section 3.4.
Sample Complexity Lower Bounds The kernel defined in the previous paragraph is not suitable for constructing lower bounds over the class  $\mathcal{P}_{\mathrm{H}}$  because its eigenfunctions do not necessarily lie in  $\mathcal{P}_{\mathrm{H}}$ . It would be possible to consider a different kernel that is more adapted to this problem/class but we do not pursue this here.
B.3  $(\beta, 2)$ -Sobolev Smooth Densities on  $\mathbb{R}^d$  Under  $L^2$ -Separation
Sample Complexity Upper Bounds Let  $\mathcal{P}_{\mathsf{S}}(\beta, d, C)$  be the class of distributions that are supported on  $\mathbb{R}^d$  and whose Lebesgue density  $p$  satisfies  $\| p \|_{\beta, 2} \leq C$ , where
$$
\left\| p \right\| _ {\beta , 2} \triangleq \left\| (1 + \| \cdot \|) ^ {\beta} \mathcal {F} [ p ] \right\| _ {2} \tag {14}
$$
and  $\mathcal{F}$  denotes the Fourier transform. Again, abusing notation, we write  $\mathcal{P}_{\mathsf{S}}(\beta ,d,C)$  both as the set of distributions and the set of density functions.
We take the Gaussian kernel  $G_{\sigma}(x,y) = \sigma^{-d}\exp (-\| x - y\| _2^2 /\sigma^2)$  on  $\mathcal{X} = \mathbb{R}^{d}$  with base measure  $\mathrm{d}\mu (x) = \exp (-x^2)\mathrm{d}x$ . In [37] the authors showed that the two-sample test that thresholds the Gaussian MMD with appropriately chosen variance  $\sigma^2$  achieves the minimax optimal sample complexity over  $\mathcal{P}_{\mathsf{S}}$ , when separation is measured by  $L^2$ . A key ingredient in their proof is the following inequality.
Lemma B.5 ([37, Lemma 5]). Let  $f,g\in \mathcal{P}_{\mathsf{S}}(\beta ,d,C)$  with  $\| f - g\| _2\geq \epsilon$  . Then, there exist constants  $c,c^{\prime}$  independent of  $\epsilon$  such that for any  $\sigma \leq c\epsilon^{1 / \beta}$  , we have
Now, suppose that we have samples  $X,Y,Z$  of sizes  $n,n,m$  from  $P_{X},P_{Y},P_{Z}\in \mathcal{P}_{\mathsf{S}}(\beta ,d,C)$  for some constant  $C$  with densities  $p_X,p_Y,p_Y$  satisfying  $\| p_X - p_Y\| _2\geq \epsilon$
Note that the heat-semigroup is an  $L^2$ -contraction  $(\| \lambda \|_{\infty} \leq 1)$  and that
$$
\| \lambda \| _ {2} ^ {2} = \int G _ {\sigma} (x, y) ^ {2} \mathrm {d} \mu (x) \mathrm {d} \mu (y) \asymp \sigma^ {- d}
$$
up to constants depending on the dimension. Theorem 3.1 combined with Lemma B.5 and a choice  $\sigma \asymp \epsilon^{1 / \beta}$  yields the following result.
Proposition B.6. Testing (mLFHT) over the class  $\mathcal{P}_{\mathsf{S}}$  with total error  $\alpha$  is possible provided
$$
\min  \{m, n \} \gtrsim \frac {C \| \lambda \| _ {\infty} \log (1 / \alpha) (1 + R) ^ {2}}{\delta^ {2} \epsilon^ {2}} \asymp \frac {\log (1 / \alpha) (1 + R) ^ {2}}{\delta^ {2} \epsilon^ {2}}
$$
where  $\epsilon$  is the lower bound on the  $L^2$ -distance between  $P_X, P_Y$  and  $R$  is defined as in the assumption (iii) of Section 3.1.
Taking  $R \asymp \delta \asymp \alpha = \Theta(1)$  above, we obtain new results for LFHT and using the reduction from two-sample testing given in Section 3.4 we partly recover [37, Theorem 5]. Only partly, because the above requires bounded density with respect to our base measure  $\mathrm{d}\mu(x) = \exp(-x^2)\mathrm{d}x$ .
Sample Complexity Lower Bounds Note that our lower bound Theorem 3.2 doesn't apply because the top eigenfunction of the Gaussian kernel is not constant. Once again, a more careful choice of the base measure (or kernel) might lead to a more suitable argument for the lower bound. We leave such pursuit as open.
C Black-box Boosting of Success Probability
In this section we briefly describe how upper bounds on the minimax sample complexity in the constant error probability regime ( $\alpha = \Theta(1)$ ) can be used to obtain the dependence  $\log(1 / \alpha)$  in the small error probability regime ( $\alpha = o(1)$ ). We will argue abstractly in a way that applies to the setting of Theorem 3.1.
Suppose that from some distributions  $P_{1}, P_{2}, \ldots, P_{k}$  we take samples  $X^{1}, X^{2}, \ldots, X^{k}$  of size  $n_{1}, n_{2}, \ldots, n_{k}$  respectively and are able to decide between two hypotheses  $H_{0}$  and  $H_{1}$  (fixed but arbitrary) with total error probability at most  $1/3$ . Call this test as  $\Psi(X^{1}, \ldots, X^{k}) \in \{0, 1\}$ , so that
$$
\mathbb {P} \left(\Psi \left(X ^ {1}, \dots , X ^ {k}\right) = 0 \mid H _ {0}\right) \geq 2 / 3 \quad \text {a n d} \quad \mathbb {P} \left(\Psi \left(X ^ {1}, \dots , X ^ {k}\right) = 1 \mid H _ {1}\right) \geq 2 / 3.
$$
Now, to each an error of  $o(1)$ , instead, we take  $18n_{1}\log (2 / \alpha),\ldots ,18n_{k}\log (2 / \alpha)$  observations from  $P_{1}$  through  $P_{k}$ , and split each sample into  $18\log (2 / \alpha)$  equal sized batches  $\{X^{i,j}\}_{i\in [k],j\in [18\log (2 / \alpha)]}$ . Here  $18\log (2 / \alpha)$  is assumed to be an integer without loss of generality. The split samples form  $18\log (2 / \alpha)$  independent binary random variables
$$
A _ {j} \triangleq \Psi \left(X ^ {1, j}, \dots , X ^ {k, j}\right)
$$
for  $j = 1,2,\ldots ,18\log (2 / \alpha)$ . We claim that the majority voting test
$$
\Psi_ {\alpha} (\{X ^ {i, j} \} _ {i, j}) = \left\{ \begin{array}{l l} 1 & \text {i f} \bar {A} \geq 1 / 2 \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
tests  $H_0$  against  $H_{1}$  with total probability of error at most  $\alpha$ , where
$$
\bar {A} \triangleq \frac {1}{1 8 \log (2 / \alpha)} \sum_ {j = 1} ^ {1 8 \log (2 / \alpha)} A _ {j}.
$$
Indeed, by Hoeffding's inequality, we have
$$
\mathbb {P} \left(\bar {A} \geq 1 / 2 \mid H _ {0}\right) \leq \alpha / 2
$$
$$
\mathbb {P} \left(\bar {A} \leq 1 / 2 \mid H _ {1}\right) \leq \alpha / 2.
$$
Therefore, in the remainder of our upper bound proofs, we only focus on achieving a constant probability of error  $(\alpha = \Theta(1))$  as the logarithmic dependence follows by the above.
Remark C.1. As mentioned in the discussion succeeding Corollary 3.5, we do conjecture the tight dependence in the upper bound to be  $\sqrt{\log(\alpha^{-1})}$  instead of  $\log (\alpha^{-1})$  shown by this method.
D.1 Notation and Technical Tools
We use the expansion
$$
K (x, y) = \sum_ {\ell} \lambda_ {\ell} e _ {\ell} (x) e _ {\ell} (y)
$$
extensively, where  $\lambda \triangleq (\lambda_1,\lambda_2,\ldots)$  are  $K$ 's eigenvalues (regarded as an integral operator on  $L^2 (\mu)$ ) in non-increasing order and  $e_1,e_2,\ldots$  are the corresponding eigenfunctions forming an orthonormal basis for  $L^2 (\mu)$ , and convergence is to be understood in  $L^2 (\mu)$ . We use the notation  $\langle \cdot \rangle \triangleq \int \cdot \mathrm{d}\mu$ . For all  $u\in L^{2}(\mu)$  we define
$$
u _ {\ell} \triangleq \langle u e _ {\ell} \rangle , \quad u _ {\ell \ell^ {\prime}} \triangleq \langle u e _ {\ell} e _ {\ell^ {\prime}} \rangle , \quad \ell = 1, 2, \dots
$$
and consequently  $u = \sum_{\ell} u_{\ell} e_{\ell}$ . We also define
$$
K [ u ] (\cdot) \triangleq \int K (t, \cdot) u (t) \mu (\mathrm {d} t) = \sum_ {\ell} \lambda_ {\ell} u _ {\ell} e _ {\ell} (\cdot),
$$
where the second equality follows from the orthonormality of  $\{e_{\ell}\}_{\ell = 1}^{\infty}$ . Note that the RKHS embedding satisfies  $\theta_{u} \triangleq \int K(x, \cdot) u(x) \mathrm{d}\mu(x) = K[u]$ . Now, for  $P_{X}$  we write
$$
x _ {\ell} \triangleq (p _ {X}) _ {\ell} = \left\langle p _ {X} e _ {\ell} \right\rangle , \quad x _ {\ell \ell^ {\prime}} \triangleq (p _ {X}) _ {\ell \ell^ {\prime}} = \left\langle p _ {X} e _ {\ell} e _ {\ell^ {\prime}} \right\rangle , \quad \ell , \ell^ {\prime} = 1, 2, \dots
$$
where  $p_X$  is the  $\mu$ -density of  $P_X$ . The similar notations also apply to  $P_Y, P_Z$ . The following identities will be very useful in our proofs.
Lemma D.1. For each identity below, let  $f, g, h \in L^{2}(\mu)$  be such that the quantity is well defined. Then,
$$
\left\| \theta_ {f} \right\| _ {\mathcal {H} _ {K}} ^ {2} = \sum_ {\ell} \lambda_ {\ell} f _ {\ell} ^ {2} \tag {15}
$$
$$
\mathrm {M M D} ^ {2} (f, g) = \sum_ {\ell} \lambda_ {\ell} \left(f _ {\ell} - g _ {\ell}\right) ^ {2} \tag {16}
$$
$$
\| K [ f ] \| _ {2} ^ {2} = \sum_ {\ell} \lambda_ {\ell} ^ {2} f _ {\ell} ^ {2} \tag {17}
$$
$$
\sum_ {\ell} \lambda_ {\ell} f _ {\ell} g _ {\ell} = \langle f K [ g ] \rangle = \langle K [ f ] g \rangle \tag {18}
$$
$$
\sum_ {\ell \ell^ {\prime}} \lambda_ {\ell} \lambda_ {\ell^ {\prime}} h _ {\ell \ell^ {\prime}} f _ {\ell} g _ {\ell^ {\prime}} = \langle h K [ f ] K [ g ] \rangle \tag {19}
$$