SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| g _ {\eta} \| _ {\infty} = \epsilon \| \sum_ {j \geq 2} \rho_ {j} \eta_ {j} e _ {j} \| _ {\infty} = \epsilon \sup _ {x \in \mathcal {X}} \langle K (x, \cdot), \sum_ {j \geq 2} \rho_ {j} \eta_ {j} e _ {j} \rangle_ {\mathcal {H}} \\ \leq \epsilon \| \sum_ {j \geq 2} \rho_ {j} \eta_ {j} e _ {j} \| _ {\mathcal {H}} = \epsilon \sqrt {\sum_ {j \geq 2} \rho_ {j} ^ {2} / \lambda_ {j}} = \frac {\epsilon \sqrt {J - 1}}{\| \lambda \| _ {2 , J}}, \\ \end{array}
$$
Note that Lemma E.4 immediately gives us a proof of Corollary 3.5.
Proof of Corollary 3.5. Suppose that  $J$  is such that  $\sum_{j=2}^{J} \lambda_j^2 \geq c^2 \| \lambda \|_2^2$ . Then, by Lemma E.4, if  $\epsilon \leq \| \lambda \|_{2J} / (2\sqrt{J - 1})$  then  $J \leq J_\epsilon^\star$ . By assumption, this is implied by the inequality  $\epsilon \leq c \| \lambda \|_2 / (2\sqrt{J - 1})$ , and the result follows.
Continuing with our proof, note that by construction we have
$$
\mathrm {M M D} ^ {2} \left(f _ {0}, f _ {\eta}\right) = \sum_ {j \geq 2} \lambda_ {j} \rho_ {j} ^ {2} = \epsilon^ {2}, \quad \forall \eta \in \{\pm 1 \} ^ {\mathbb {N}}. \tag {25}
$$
E.2.1 Lower Bound on  $m$
Again, we apply Lemma E.1 with the new (deterministic) construction
$$
P _ {0} = f _ {0} ^ {\otimes n} \otimes \left(1 + \epsilon e _ {2} / \sqrt {\lambda_ {2}}\right) ^ {n} \otimes \left(1 + \delta \epsilon e _ {2} / \sqrt {\lambda_ {2}}\right) ^ {\otimes m}, \quad P _ {1} = f _ {0} ^ {\otimes n} \otimes \left(1 + \epsilon e _ {2} / \sqrt {\lambda_ {2}}\right) ^ {n} \otimes f _ {0} ^ {\otimes m}, \tag {26}
$$
where we write  $f_{\mathbb{1}} = f_{(1,1,\dots)}$  and similarly for  $g_{\mathbb{1}}$ . By the data-processing inequality for  $\chi^2$ -divergence (also by Lemma E.3), we may drop the first  $2n$  coordinates and obtain
$$
\begin{array}{l} \chi^ {2} (\mathbb {E} P _ {0}, \mathbb {E} P _ {1}) = \chi^ {2} ((1 + \delta \epsilon e _ {2} / \sqrt {\lambda_ {2}}) ^ {\otimes m} \| f _ {0} ^ {\otimes m}) \\ = \left(1 + \delta^ {2} \epsilon^ {2} / \lambda_ {2}\right) ^ {m} - 1 \\ \leq \exp \left(\frac {\delta^ {2} \epsilon^ {2} m}{\lambda_ {2}}\right) - 1. \\ \end{array}
$$
By Lemma E.2 we
$$
1 - \mathrm {T V} (\mathbb {E} P _ {0}, \mathbb {E} P _ {1}) \gtrsim \frac {1}{\chi^ {2} (\mathbb {E} P _ {0} , \mathbb {E} P _ {1}) - 1} \geq \exp (- \delta^ {2} \epsilon^ {2} m) \stackrel {!} {=} \Omega (\alpha).
$$
The lower bound  $m \gtrsim \lambda_2 \log (1 / \alpha) / (\delta \epsilon)^2$  now follows readily.
E.2.2 Lower Bound on  $n$
Once again, we apply Lemma E.1 to the new construction
$$
P _ {0} = f _ {0} ^ {\otimes n} \otimes f _ {\eta} ^ {\otimes n} \otimes f _ {0} ^ {\otimes m}, \quad P _ {1} = f _ {\eta} ^ {\otimes n} \otimes f _ {0} ^ {\otimes n} \otimes f _ {0} ^ {\otimes m}, \tag {27}
$$
where we put a uniform prior on  $\eta \in \{\pm 1\}^{\mathbb{N}}$  as before. Using the subadditivity of total variation under products, we compute
$$
\begin{array}{l} \operatorname {T V} \left(\mathbb {E} P _ {0}, \mathbb {E} P _ {1}\right) = \operatorname {T V} \left(f _ {0} ^ {\otimes n} \otimes \mathbb {E} f _ {\eta} ^ {\otimes n}, \mathbb {E} \left[ f _ {\eta} ^ {\otimes n} \right] \otimes f _ {0} ^ {\otimes n}\right) \\ \leq 2 \operatorname {T V} \left(\mathbb {E} f _ {\eta} ^ {\otimes n}, f _ {0} ^ {\otimes n}\right). \\ \end{array}
$$
Just as in Appendix E.2.3 we upper bound by the  $\chi^2$ -divergence to get
$$
\begin{array}{l} \chi^ {2} \left(\mathbb {E} f _ {\eta} ^ {\otimes n} \| f _ {0} ^ {\otimes n}\right) = - 1 + \mathbb {E} _ {\eta \eta^ {\prime}} \int \prod_ {i = 1} ^ {n} \left(f _ {\eta} \left(x _ {i}\right) f _ {\eta^ {\prime}} \left(x _ {i}\right)\right) \mu (\mathrm {d} x _ {1}) \dots \mu (\mathrm {d} x _ {n}) \\ \leq - 1 + \mathbb {E} \exp \left(n \epsilon^ {2} \sum_ {j \geq 2} \rho_ {j} ^ {2} \eta_ {j} \eta_ {j} ^ {\prime}\right) \\ = - 1 + \prod_ {j \geq 2} \cosh (n \epsilon^ {2} \rho_ {j} ^ {2}) \\ \leq - 1 + \exp (n ^ {2} \epsilon^ {4} \sum_ {j \geq 2} \rho_ {j} ^ {4}) \\ = - 1 + \exp \left(n ^ {2} \epsilon^ {4} / \| \lambda \| _ {2, J} ^ {2}\right). \\ \end{array}
$$
Again, by Lemma E.2 we obtain
$$
1 - \operatorname {T V} (\mathbb {E} P _ {0}, \mathbb {E} P _ {1}) \gtrsim \frac {1}{\chi^ {2} (\mathbb {E} P _ {0} \| \mathbb {E} P _ {1}) - 1} \geq \exp (- n ^ {2} \epsilon^ {4} / \| \lambda \| _ {2, J} ^ {2}) \stackrel {!} {=} \Omega (\alpha).
$$
The lower bound  $n \gtrsim \sqrt{\log(1 / \alpha)}\|\lambda\|_{2,J} / \epsilon^2$  now follows readily.
E.2.3 Lower Bound on  $m \cdot n$
We take a uniform prior on  $\eta$  and consider the random measures
$$
P _ {0} = f _ {0} ^ {\otimes n} \otimes f _ {\eta} ^ {\otimes n} \otimes ((1 - \delta) f _ {0} + \delta f _ {\eta}) ^ {\otimes m} \quad \text {a n d} \quad P _ {1} = f _ {0} ^ {\otimes n} \otimes f _ {\eta} ^ {\otimes n} \otimes f _ {0} ^ {\otimes m}. \tag {28}
$$
Our goal is to apply Lemma E.1 to  $P_0, P_1$ . Notice that  $(1 - \delta)f_0 + \delta f_\eta = 1 + \delta \epsilon g_\eta$ . Let us write  $X, Y, Z$  for the marginals first  $n$ , second  $n$  and last  $m$  coordinates of  $P_0$  and  $P_1$ . By the data processing inequality and the chain rule Lemma E.3 we have
$$
\begin{array}{l} \chi^ {2} (\mathbb {E} P _ {0} \| \mathbb {E} P _ {1}) = \chi^ {2} ((\mathbb {E} P _ {0}) _ {Y, Z} \| (\mathbb {E} P _ {1}) _ {Y, Z}) \\ = \chi^ {2} \left(\left(\mathbb {E} P _ {0}\right) _ {Z | Y} \| \left(\mathbb {E} P _ {1}\right) _ {Z | Y} \mid \left(\mathbb {E} P _ {0}\right) _ {Y}\right) \\ = \mathbb {E} \chi^ {2} \left(\mathbb {E} \left[ (1 + \delta \epsilon g _ {\eta}) ^ {\otimes m} \mid Y \right] \| f _ {0} ^ {\otimes m}\right) =: (\dagger). \\ \end{array}
$$
Notice that the expectation inside the  $\chi^2$ -divergence is with respect to  $\eta$  given the variables  $Y$ , or in other words, over the posterior of  $\eta$  with uniform prior given  $n$  observations from the density  $1 + \epsilon g_{\eta} = f_{\eta}$ . The outer expectation is over  $Y$ . Given  $Y$ , let  $\eta$  and  $\eta'$  be i.i.d. from said posterior. We get the bound
$$
\begin{array}{l} (\dagger) + 1 \leq \mathbb {E} \int \prod_ {i = 1} ^ {m} (1 + \delta \epsilon g _ {\eta} (x _ {i})) (1 + \delta \epsilon g _ {\eta^ {\prime}} (x _ {i})) \mu (\mathrm {d} x _ {i}) \\ = \mathbb {E} \left(1 + \delta^ {2} \epsilon^ {2} \sum_ {j \geq 2} \rho_ {j} ^ {2} \eta_ {j} \eta_ {j} ^ {\prime}\right) ^ {m} \\ \leq \mathbb {E} \exp \left(\delta^ {2} \epsilon^ {2} m \sum_ {j \geq 2} \rho_ {j} ^ {2} \eta_ {j} \eta_ {j} ^ {\prime}\right). \\ \end{array}
$$
Define the collections of variables  $\eta_{-j} = \{\eta_j\}_{j\geq 2}\setminus \{\eta_j\}$  and  $\eta_{-j}^{\prime}$  similarly. We shall prove the following claim:
$$
\mathbb {E} \left[ \exp \left(\delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2} \eta_ {j} \eta_ {j} ^ {\prime}\right) \mid \eta_ {- j} \eta_ {- j} ^ {\prime} \right] \leq \exp \left(c \delta^ {2} \epsilon^ {4} \left(\delta^ {2} m ^ {2} + m n\right) \rho_ {j} ^ {4}\right) \tag {29}
$$
for some universal constant  $c > 0$ . Assuming that (29) holds, by induction we can show that
$$
\begin{array}{l} (\dagger) + 1 \leq \exp (c \delta^ {2} (\delta^ {2} m ^ {2} + m n) \epsilon^ {4} \sum_ {j \geq 2} \rho_ {j} ^ {4}) \\ = \exp (c \delta^ {2} (\delta^ {2} m ^ {2} + m n) \epsilon^ {4} / \| \lambda \| _ {2, J} ^ {2}). \\ \end{array}
$$
Thus, if  $mn + \delta^2 m^2 = o\left(\| \lambda \|_{2,J}^2 / (\delta^2 \epsilon^4)\right)$  then testing is impossible.
We now prove (29). Since the variable  $\eta_j^\prime \eta_j^\prime$  is either 1 or  $-1$ , we have
$$
\mathbb {E} \left[ \exp (\delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2} \eta_ {j} \eta_ {j} ^ {\prime}) | \eta_ {- j} \eta_ {- j} ^ {\prime} \right] = (e ^ {\delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2}} - e ^ {- \delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2}}) \cdot \mathbb {P} (\eta_ {j} \eta_ {j} ^ {\prime} = 1 | \eta_ {- j} \eta_ {- j} ^ {\prime}) + e ^ {- \delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2}}.
$$
Let us write  $\eta_{\pm 1,j}$  for the vector of signs equal to  $\eta$  but whose  $j$ 'th coordinate is  $\pm 1$  respectively. Looking at the probability above, and using the independence of  $\eta, \eta'$  given  $Y$ , we have
$$
\begin{array}{l} \mathbb {P} (\eta_ {j} \eta_ {j} ^ {\prime} = 1 | Y, \eta_ {- j}, \eta_ {- j} ^ {\prime}) = \mathbb {P} (\eta_ {j} = 1 | Y, \eta_ {- j}) ^ {2} + \mathbb {P} (\eta_ {j} = - 1 | Y, \eta_ {- j}) ^ {2} \\ = \frac {1}{4} \frac {(f _ {\eta_ {1 j}} ^ {\otimes n} (Y)) ^ {2} + (f _ {\eta_ {- 1 j}} ^ {\otimes n} (Y)) ^ {2}}{\left(\frac {1}{2} f _ {\eta_ {1 j}} ^ {\otimes n} (Y) + \frac {1}{2} f _ {\eta_ {- 1 j}} ^ {\otimes n} (Y)\right) ^ {2}}. \\ \end{array}
$$
Taking the expectation  $\mathbb{E}[\cdot |\eta_{-j},\eta_{-j}^{\prime}]$  and using the HM-AM inequality  $(\frac{1}{2} (x + y))^{-1}\leq \frac{1}{2} (\frac{1}{x} +\frac{1}{y})$  valid for all  $x,y > 0$  gives
$$
\begin{array}{l} \mathbb {P} (\eta_ {j} \eta_ {j} ^ {\prime} = 1 | \eta_ {- j}, \eta_ {- j} ^ {\prime}) = \frac {1}{4} \int \frac {(\prod_ {i = 1} ^ {n} f _ {\eta_ {1 j}} (x _ {i})) ^ {2} + (\prod_ {i = 1} ^ {n} f _ {\eta_ {- 1 j}} (x _ {i})) ^ {2}}{\frac {1}{2} \prod_ {i = 1} ^ {n} f _ {\eta_ {1 j}} (x _ {i}) + \frac {1}{2} \prod_ {i = 1} ^ {n} f _ {\eta_ {- 1 j}} (x _ {i})} \mu (\mathrm {d} x _ {1}) \dots \mu (\mathrm {d} x _ {n}) \\ \leq \frac {1}{4} + \frac {1}{8} \int \left(\frac {(\prod_ {i = 1} ^ {n} f _ {\eta_ {1 j}} (x _ {i})) ^ {2}}{\prod_ {i = 1} ^ {n} f _ {\eta_ {- 1 j}} (x _ {i})} + \frac {(\prod_ {i = 1} ^ {n} f _ {\eta_ {- 1 j}} (x _ {i})) ^ {2}}{\prod_ {i = 1} ^ {n} f _ {\eta_ {1 j}} (x _ {i})}\right) \mu (\mathrm {d} x _ {1}) \ldots \mu (\mathrm {d} x _ {n}) = (\star). \\ \end{array}
$$
Note that  $f_{\eta_{1j}} = f_{\eta_{-1j}} + 2\epsilon \rho_j e_j$ . Using the lower bound  $f_{\eta_{\pm 1j}}(x) \geq \frac{1}{2}$  for all  $x \in \mathcal{X}$  and the inequality  $1 + x \leq \exp(x)$ , we get
$$
\begin{array}{l} (\star) \leq \frac {1}{4} + \frac {1}{8} \left[ \left(1 + \int \frac {4 \epsilon^ {2} \rho_ {j} ^ {2} e _ {j} ^ {2} (x)}{f _ {\eta - 1 j} (x)} \mu (\mathrm {d} x)\right) ^ {n} + \left(1 + \int \frac {4 \epsilon^ {2} \rho_ {j} ^ {2} e _ {j} ^ {2} (x)}{f _ {\eta 1 j} (x)} \mu (\mathrm {d} x)\right) ^ {n} \right] \\ \leq \frac {1}{4} (1 + e ^ {8 \epsilon^ {2} n \rho_ {j} ^ {2}}). \\ \end{array}
$$
Recall that  $(\star)$  is a probability so  $(\star) \leq 1$ , and we obtain
$$
(\star) \leq \frac {1}{4} (1 + e ^ {8 \epsilon^ {2} n \rho_ {j} ^ {2} \wedge \ln 3}).
$$
Putting it together and applying Lemma E.5 we get
$$
\begin{array}{l} \text {L H S} (2 9) \leq \left(e ^ {\delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2}} - e ^ {- \delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2}}\right) \frac {1}{4} \left(1 + e ^ {8 \epsilon^ {2} n \rho_ {j} ^ {2} \wedge \ln 3}\right) + e ^ {- \delta^ {2} \epsilon^ {2} m \rho_ {j} ^ {2}} \\ \leq e ^ {c \delta^ {2} \epsilon^ {4} \rho_ {j} ^ {4} (\delta^ {2} m ^ {2} + m n)} \\ \end{array}
$$
for universal  $c = 16 > 0$ . Thus, by Lemma E.2 we obtain
$$
1 - \mathrm {T V} (\mathbb {E} P _ {0}, \mathbb {E} P _ {1}) \gtrsim \frac {1}{\chi^ {2} (\mathbb {E} P _ {0} , \mathbb {E} P _ {1}) + 1} \geq \exp (- c \delta^ {2} \epsilon^ {4} (\delta^ {2} m ^ {2} + m n) / \| \lambda \| _ {2, J} ^ {2}) \stackrel {!} {=} \Omega (\alpha).
$$
The necessity of
$$
m n + \delta^ {2} m ^ {2} \gtrsim \frac {\log (1 / \alpha) \| \lambda \| _ {2 , J} ^ {2}}{\delta^ {2} \epsilon^ {4}}
$$
follows immediately.3
Lemma E.5. For  $a, b \geq 0$ , the following inequality holds:
$$
\frac {1}{4} (e ^ {a} - e ^ {- a}) (1 + e ^ {b \wedge \ln 3}) + e ^ {- a} \leq e ^ {2 (a b + a ^ {2})}.
$$
Proof. If  $b \geq \ln 3$  or  $a \geq 1$  we have:
$$
\mathrm {L H S} \leq \frac {1}{4} (e ^ {a} - e ^ {- a}) (1 + e ^ {\ln 3}) + e ^ {- a} = e ^ {a} \leq e ^ {\frac {b}{\ln 3} a + a ^ {2}}.
$$
If  $b < \ln 3$  and  $a < 1$ , we have
$$
e ^ {b} \leq 1 + \frac {2}{\ln 3} b \leq 1 + 2 b, \quad \frac {e ^ {a} + e ^ {- a}}{2} \leq e ^ {a ^ {2}}, \quad \frac {e ^ {a} - e ^ {- a}}{2} \leq \frac {e - e ^ {- 1}}{2} a \leq 2 a,
$$
3We have  $mn + m^2\leq (\sqrt{mn} +m)^2\leq 2(mn + m^2)$  , so  $\sqrt{mn} +m\succ \sqrt{mn + m^2}$
$$
\begin{array}{l} \frac {1}{4} \left(e ^ {a} - e ^ {- a}\right) \left(1 + e ^ {b}\right) + e ^ {- a} = \frac {1}{2} \left(e ^ {a} + e ^ {- a}\right) + \frac {e ^ {b} - 1}{4} \left(e ^ {a} - e ^ {- a}\right) \\ \leq e ^ {a ^ {2}} + 2 a b \\ \leq e ^ {a ^ {2}} (1 + 2 a b) \\ \leq e ^ {a ^ {2} + 2 a b} \\ \end{array}
$$
The result follows from  $\ln 3 > 1$ .
F Proofs From Section 4
F.1 Computing  $\hat{\sigma}$
We follow the implementation of  $\widehat{\sigma}^2$  in [39]. Given  $X_{1},\ldots ,X_{n_{\mathrm{tr}}}^{\mathrm{tr}}$  sampled from  $P_{X}$  and  $Y_{1},\ldots ,Y_{n_{\mathrm{tr}}}^{\mathrm{tr}}$  sampled from  $P_{Y}$ , denote
$$
H _ {i j} := K \left(X _ {i} ^ {\mathrm {t r}}, X _ {j} ^ {\mathrm {t r}}\right) + K \left(Y _ {i} ^ {\mathrm {t r}}, Y _ {j} ^ {\mathrm {t r}}\right) - K \left(X _ {i} ^ {\mathrm {t r}}, Y _ {j} ^ {\mathrm {t r}}\right) - K \left(Y _ {i} ^ {\mathrm {t r}}, X _ {j} ^ {\mathrm {t r}}\right), \quad i, j \in [ n _ {\mathrm {t r}} ]. \tag {30}
$$
Then  $\widehat{\sigma}^2$  is computed via
$$
\hat {\sigma} ^ {2} \left(X ^ {n _ {\mathrm {t r}}}, Y ^ {n _ {\mathrm {t r}}}; K\right) = \frac {4}{n _ {\mathrm {t r}} ^ {3}} \sum_ {i = 1} ^ {n _ {\mathrm {t r}}} \left(\sum_ {j = 1} ^ {n _ {\mathrm {t r}}} H _ {i j}\right) ^ {2} - \frac {4}{n _ {\mathrm {t r}} ^ {4}} \left(\sum_ {i = 1} ^ {n _ {\mathrm {t r}}} \sum_ {j = 1} ^ {n _ {\mathrm {t r}}} H _ {i j}\right) ^ {2}. \tag {31}
$$
Note that  $\hat{\sigma}^2$  being non-negative follows from the AM-GM inequality.
F.2 Heuristic Justification of the Objective (8)
As usual, let  $X, Y, Z$  denotes samples of sizes  $n, n, m$  from  $P_X, P_Y, P_Z$  respectively. Let us give a heuristic justification for using the training objective defined in (8) for the purpose of obtaining a kernel for LFHT/mLFHT. Note that originally it was proposed as a training objective for kernels to be used in two sample testing. Recall that our test for LFHT can be written as
$$
\Psi_ {1 / 2} (X, Y, Z) = \mathbb {1} \left\{T _ {\mathrm {L F}} \geq 0 \right\}
$$
$$
T _ {\mathsf {L F}} = \mathrm {M M D} _ {u} ^ {2} (\widehat {P} _ {Z}, \widehat {P} _ {Y}; K) - \mathrm {M M D} _ {u} ^ {2} (\widehat {P} _ {Z}, \widehat {P} _ {X}; K),
$$
Heuristically, to maximize the power of (mLFHT), we would like to maximize the following population quantity
$$
J _ {\mathrm {L F}} \triangleq \frac {\mathbb {E} _ {0} \left[ T _ {\mathrm {L F}} \right] - \mathbb {E} _ {1} \left[ T _ {\mathrm {L F}} \right]}{\sqrt {\operatorname {v a r} _ {0} \left(T _ {\mathrm {L F}}\right)}}
$$
$$
\mathbb {E} _ {0} \left[ T _ {\mathrm {L F}} \right] = \mathbb {E} _ {X, Y, Z} \left[ T _ {\mathrm {L F}} \mid P _ {Z} = P _ {X} \right] = + \operatorname {M M D} ^ {2} \left(P _ {X}, P _ {Y}; K\right),
$$
$$
\mathbb {E} _ {1} \left[ T _ {\mathrm {L F}} \right] = \mathbb {E} _ {X, Y, Z} \left[ T _ {\mathrm {L F}} \mid P _ {Z} = P _ {Y} \right] = - \operatorname {M M D} ^ {2} \left(P _ {X}, P _ {Y}; K\right).
$$
Let  $T_{\mathrm{TS}} = \mathrm{MMD}_u(\hat{P}_X, \hat{P}_Y)$  be the usual statistic that is thresholded for two-sample testing. Then, a computation analogous to that in Section D.2 show (cf. (22)) that
$$
\operatorname {v a r} _ {0} \left(T _ {\mathrm {L F}}\right) \approx \frac {A (K , P _ {X} , P _ {Y})}{n} + \frac {A (K , P _ {X} , P _ {Y})}{m} + \frac {B (K , P _ {X} , P _ {Y})}{n ^ {2}} + \frac {B (K , P _ {X} , P _ {Y})}{m n},
$$
$$
\operatorname {v a r} _ {0} \left(T _ {\mathsf {T S}}\right) \approx \frac {A (K , P _ {X} , P _ {Y})}{n} + \frac {B (K , P _ {X} , P _ {Y})}{n ^ {2}}
$$
for some  $A(K)$  and  $B(K)$ . Therefore, we have approximately
$$
J _ {\mathrm {L F}} \approx \frac {2 \mathrm {M M D} ^ {2} (P _ {X} , P _ {Y} ; K)}{\sqrt {1 + \frac {n}{m}} \sqrt {\mathrm {v a r} _ {0} (T _ {\mathsf {T S}})}} \approx 2 \sqrt {\frac {m}{m + n}} \widehat {J} (X, Y; K)
$$
which only differs from our optimization objective defined in (8) by a constant factor.
Second, notice that  $\frac{\mathrm{MMD}(P_X,P_Y;K)}{\sqrt{\mathrm{var}(T_{\mathrm{TS}})}}$  depends only on  $P_{X} - P_{Y}$  and that  $((1 - \delta)P_{X} + \delta P_{Y}) - P_{X}\propto$ $P_{Y} - P_{X}$ , therefore it is sensible to use (8) as our training objective for is also sensible for (mLFHT), and we don't even need to observe the sample  $Z$ .
F.3 Proof of Proposition 4.1
Proof. In this proof we regard  $\mathcal{D} \triangleq (X^{\mathrm{tr}}, X^{\mathrm{ev}}, Y^{\mathrm{tr}}, Y^{\mathrm{ev}})$  and the parameters of the kernel  $\omega$  as fixed. Recall that we are looking at the problem mLFHT with a misspecification parameter  $R = 0$  (see Theorem 3.1). Given a test set  $\{z_i\}_{i \in [m]}$ , our test statistic is  $T(\{z_i\}_{i \in [m]}) = \frac{1}{m} \sum_{i=1}^{m} f(z_i)$  where
$$
f (z _ {i}) = \frac {1}{n _ {\mathrm {e v}}} \sum_ {j = 1} ^ {n _ {\mathrm {e v}}} \Bigl (K _ {\omega} (z _ {i}, Y _ {j} ^ {\mathrm {e v}}) - K _ {\omega} (z _ {i}, X _ {j} ^ {\mathrm {e v}}) \Bigr).
$$
In Phase 3 of Algorithm 1, we observe the value  $\widehat{T} = T(Z) = \frac{1}{m}\sum_{i=1}^{m}f(Z_i)$  and reject the null hypothesis for large values of  $\widehat{T}$ . Thus, the  $p$ -value is defined as
$$
p = p (Z, \mathcal {D}) \triangleq \mathbb {P} _ {\widetilde {Z} \sim P _ {X} ^ {\otimes m}} (T (\widetilde {Z}) > \widehat {T}).
$$
Phase 2 of our Algorithm 1 produces random variables  $T_{1}, \ldots, T_{k}$  that all have the distribution of  $T(\{\widetilde{Z}_{i}\}_{i \in [m]})$ , so that  $\mathbb{1}\{T_{r} \geq \widehat{T}\}$  ( $r = 1, \ldots, k$ ) are unbiased estimates of the  $p$ -value. However, the  $T_{i}$  are not independent, because they sample from the finite collection of calibration samples  $X^{\mathrm{cal}}$ . However, as  $n_{\mathrm{cal}} \to \infty$  the covariances between  $T_{r_1}, T_{r_2}$  for  $r_1 \neq r_2$  tend to zero, and we obtain a consistent estimate of  $p$ .
F.4 Proof of Proposition 4.2
Proof. The test statistic  $T(X,Y,Z)$  in (3) is given by
$$
T (X, Y, Z) = \frac {1}{m} \sum_ {i = 1} ^ {m} f _ {K} (Z _ {i})
$$
$$
f _ {K} (z) = \theta_ {\widehat {P} _ {Y}} (z) - \theta_ {\widehat {P} _ {X}} (z).
$$
This simplifies to (consider  $K(x,y) = f(x)f(y)$ )
$$
f _ {K} (z) = \left(\frac {1}{n} \sum_ {j = 1} ^ {n} f (Y _ {j}) - \frac {1}{n} \sum_ {j = 1} ^ {n} f (X _ {j})\right) f (z) = C (X, Y) f (z).
$$
where  $C(X,Y)$  does not depend on  $z$ . Therefore, for any witness function  $f$ , we obtain the desired additive test.
F.5 Additive Test Statistics
In this section we prove accordingly that the test statistics of all of MMD-M/G/O, SCHE, LBI, UME, RFM are of the form  $T_{f}(Z) = \frac{1}{m}\sum_{i = 1}^{m}f(Z_{i})$  (where  $f$  might depends on  $X,Y$ ). The test is to compare  $T_{f}(Z)$  with some threshold  $\gamma (X,Y)$ .
Note that in the setting of Algorithm 1, the  $X$  and  $Y$  here correspond to  $X^{\mathrm{ev}}$  and  $Y^{\mathrm{ev}}$ .
MMD-M/G/O As described in (3) we have
$$
T _ {f} (Z) = \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\frac {1}{n} \sum_ {j = 1} ^ {n} \left(K \left(Z _ {i}, Y _ {j}\right) - K \left(Z _ {i}, X _ {j}\right)\right)\right).
$$
SCHE As described in Section 4.2 we have
$$
T _ {f} (Z) = \frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {1} \{\phi (Z _ {i}) > t \}.
$$
LBI As described in Section 4.2 we have
$$
T _ {f} (Z) = \frac {1}{m} \sum_ {i = 1} ^ {m} \log \left(\frac {\phi (Z _ {i})}{1 - \phi (Z _ {i})}\right).
$$
UME As described in [31], the UME statistic evaluates the squared witness function at  $J_{q}$  test locations  $W = \{w_{k}\}_{k = 1}^{J_{q}}\subset \mathcal{X}$ . Formally for any two distributions  $P,Q$  we define
$$
U ^ {2} (P, Q) = \| \theta_ {Q} - \theta_ {P} \| _ {L ^ {2} (W)} ^ {2} = \frac {1}{J _ {q}} \sum_ {k = 1} ^ {J _ {q}} (\theta_ {Q} (w _ {k}) - \theta_ {P} (w _ {k})) ^ {2}.
$$
However, we note a crucial difference that their result only considers the case of  $n = m$ , and their proposed estimator for  $U^{2}(P_{Z}, P_{X})$  can not be naturally extended to the case of  $n \neq m$ . Here we generalize it to  $m \neq n$  where we (conveniently) use a biased estimate of their distance. Given samples  $X, Y, Z$  and a set of witness locations  $W$ , the test statistic is a (biased yet) consistent estimator of  $U^{2}(P_{Z}, P_{Y}) - U^{2}(P_{Z}, P_{X})$ . Let  $\psi_{W}(z) = \frac{1}{\sqrt{J_{q}}} (K(z, w_{1}), \ldots, K(z, w_{J_{q}})) \in \mathbb{R}^{|W|}$  be the "feature function," then:
$$
\begin{array}{l} \widehat {U} ^ {2} (Z, X) = \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \psi_ {W} (Z _ {i}) - \frac {1}{n} \sum_ {j = 1} ^ {n} \psi_ {W} (X _ {i}) \right\| _ {2} ^ {2} \\ = \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \psi_ {W} (Z _ {i}) \right\| _ {2} ^ {2} + \left\| \frac {1}{n} \sum_ {j = 1} ^ {n} \psi_ {W} (X _ {i}) \right\| _ {2} ^ {2} - \frac {2}{m n} \sum_ {1 \leq i \leq m, 1 \leq j \leq n} \langle \psi_ {W} (Z _ {i}), \psi_ {W} (X _ {j}) \rangle \\ \end{array}
$$
Here  $\langle \cdot ,\cdot \rangle$  denotes the usual inner product. Therefore, the difference between distances is
$$
\widehat {U} ^ {2} (Z, Y) - \widehat {U} ^ {2} (Z, X) = \frac {1}{m} \sum_ {i = 1} ^ {m} \left\langle \psi_ {W} \left(Z _ {i}\right), \frac {2}{n} \sum_ {j = 1} ^ {n} \left(\psi_ {W} \left(X _ {j}\right) - \psi_ {W} \left(Y _ {j}\right)\right) \right\rangle + F (X, Y)
$$
where  $F$  is sum function based only on  $X, Y$ . This is clearly an additive statistic for  $Z$ .
RFM Algorithm 1 in [44] describes a method for learning a kernel from data given a binary classification task. For convenience let's concatenate the data to  $X^{\mathrm{RFM}} = (X,Y) \in \mathbb{R}^{2n \times d}$  and labels  $y^{\mathrm{RFM}} = (\vec{0}_n, \vec{1}_n) \in \mathbb{R}^{1 \times 2n}$ . Given a learned kernel  $K$ , we write the Gram matrix as  $(K(X^{\mathrm{RFM}}, X^{\mathrm{RFM}}))_{i,j} = K(X_i^{\mathrm{RFM}}, X_j^{\mathrm{RFM}})$  ( $1 \leq i,j \leq 2n$ ). Let  $K(X^{\mathrm{RFM}}, z)$  be a column vector with components  $K(X_i^{\mathrm{RFM}}, z)$  ( $1 \leq i \leq 2n$ ). The classifier is then defined as
$$
f ^ {\mathrm {R F M}} (z) = y ^ {\mathrm {R F M}} \cdot K \left(X ^ {\mathrm {R F M}}, X ^ {\mathrm {R F M}}\right) ^ {- 1} \cdot K \left(X ^ {\mathrm {R F M}}, z\right). \tag {32}
$$
Though in [44] the kernel learned from RFM is used to construct a classifier as in Equation (32), since RFM is a feature learning method, we also apply the RFM kernel to our MMD test, namely
$$
f ^ {\mathrm {R F M} \text {t o M M D}} (z) = \frac {1}{n} \sum_ {j = 1} ^ {n} \left(K \left(z, Y _ {j}\right) - K \left(z, X _ {j}\right)\right).
$$
G Application: Diffusion Models vs CIFAR
We defer a more fine-grained detail to our code submission, which includes executable programs (with PyTorch) once the data-generating script from DDPM has been run (see README in the .codes/CIFAR folder).
Figure 5: Data visualization for CIFAR-10 (left) vs DDPM diffusion generated images (right)
G.1 Dataset Details
We use the CIFAR-10 dataset available online at https://www.cs.toronto.edu/~kriz/cifar.html, which contains 50000 colored images of size  $32 \times 32$  with 10 classes. For the diffusion generated images, we use the SOTA Hugging Face model (DDPM) that can be found at https://huggingface.co/google/ddpm-CIFAR-10-32. We generated 10000 artificial images for our experiments. The code can be found at our code supplements.
For dataset balancing, we randomly shuffled the CIFAR-10 dataset and used 10000 images as data in our code. Most of our experiments are conducted with the null  $P_{X}$  as CIFAR images, and the alternate as  $P_{Y} = \frac{2}{3} \cdot \mathrm{CIFAR} + \frac{1}{3} \cdot \mathrm{DDPM}$ . To this end, we matched 20000 images from CIFAR to belong to the alternate hypothesis, and the remaining 30000 images to stay in the null hypothesis. For the alternate dataset, we simply sample without replacement from the  $20000 + 10000$  mixture. This sampled distribution is almost the same as mixing (so long as the sample bank is large enough compared to the acquired data, so that each item in the alternate has close to  $1/3$  probability of being in DDPM, which is indeed the case).
G.2 Experiment Setup and Benchmarks