SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C.3 Bounds for MMD-FUSE under Null: Proof of Theorem 5
Proof of Theorem 5. Note under the null  $\mathbf{Z}$  is permutation invariant. We will use Theorem 7 with  $h$  as the MMD 2-statistic kernel, so that  $U_{k}(\mathbf{Z}) = \widehat{\mathrm{MMD}}^{2}(\mathbf{Z})$  (for fixed  $k$ ). We will use the notation  $\leq_{1 - \delta}$  to denote that the inequality holds with probability at least  $1 - \delta$  over the random variables being considered.
Note that  $\bar{U}_k^2\leq 4\kappa^2 /n(n - 1)$  from Theorem 8, so that
$$
\begin{array}{l} \log \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \exp \left(\widehat {\lambda \operatorname {M M D} _ {k} ^ {2}} (\boldsymbol {X}, \boldsymbol {Y})\right) \\ \leq_ {1 - \delta} \log \mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \exp \left(\widehat {\lambda \operatorname {M M D}} _ {k} ^ {2} (\boldsymbol {X}, \boldsymbol {Y})\right) + \log \frac {1}{\delta} \quad \text {M a r k o v} \\ = \log \mathbb {E} _ {\sigma} \mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \exp \left(\widehat {\lambda \operatorname {M M D}} _ {k} ^ {2} (\sigma \boldsymbol {Z})\right) + \log \frac {1}{\delta} \quad \text {N u l l P e r m u t a t i o n - F r e e} \\ = \log \mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \mathbb {E} _ {\sigma} \exp \left(\widehat {\lambda \operatorname {M M D}} _ {k} ^ {2} (\sigma \boldsymbol {Z})\right) + \log \frac {1}{\delta} \quad \text {P r i o r P e r m u t a t i o n - F r e e} \\ \leq \log \mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \mathbb {E} _ {\sigma} \exp \left(\lambda^ {2} \bar {U} _ {k} ^ {2}\right) + \log \frac {1}{\delta} \\ \leq \frac {4 \kappa^ {2} \lambda^ {2}}{n (n - 1)} + \log \frac {1}{\delta}, \tag {9} \\ \end{array}
$$
where the penultimate result holds using Theorem 7 provided  $|\lambda| < (4\sqrt{2}\sup_k \bar{U}_k(\mathbf{Z}))^{-1}$ . Using the bound on  $\bar{U}_k$  from above gives the  $|\lambda| < \sqrt{n(n - 1)} / 8\sqrt{2}\kappa$  in the theorem statement.
By dividing  $\widehat{\mathrm{MMD}}^2$  by the permutation invariant  $\widehat{N}_k$  in every one of the above, we also find that
$$
\log \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \exp \left(\widehat {\lambda \operatorname {M M D}} _ {k} ^ {2} (\boldsymbol {X}, \boldsymbol {Y}) / \sqrt {\widehat {N} _ {k} (\langle \boldsymbol {Z} \rangle)}\right) \leq_ {1 - \delta} \log \mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \mathbb {E} _ {\sigma} \exp \left(\frac {\lambda^ {2} \bar {U} _ {k} ^ {2}}{\widehat {N} _ {k} (\langle \boldsymbol {Z} \rangle)}\right) + \log \frac {1}{\delta}.
$$
$$
\leq \frac {1 6 \lambda^ {2}}{n (n - 1)} + \log \frac {1}{\delta} \tag {10}
$$
the first inequality using that  $\widehat{N}_k$  is permutation-invariant and holding for  $|\lambda| < \sqrt{\sup_k \widehat{N}_k} (4\sqrt{2} \sup_k \bar{U}_k(\mathbf{Z}))^{-1}$ . The final step uses that  $\bar{U}_k^2 \leq 16\widehat{N}_k / n(n - 1)$  from Theorem 8 which also shows that  $|\lambda| < \sqrt{n(n - 1)} / 16\sqrt{2}$  is sufficient.
The upper tail bounds for  $\widehat{\mathrm{FUSE}}_N$  and  $\widehat{\mathrm{FUSE}}_1$  immediately follow from the above by dividing through  $\lambda > 0$  and from their definitions.
Lower Bounds. For the lower tails,
$$
\begin{array}{l} - \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) = \frac {1}{\lambda} \log \mathbb {E} _ {k \sim \pi (\langle \boldsymbol {Z} \rangle)} \exp \left(\lambda \widehat {\mathrm {M M D}} _ {k} ^ {2} (\boldsymbol {Z})\right) \\ \leq - \mathbb {E} _ {k \sim \pi (\langle Z \rangle)} [ \widehat {\operatorname {M M D}} _ {k} ^ {2} (Z) ] \quad \text {J e s s e n} \\ = \frac {1}{s} \log \circ \exp \left(\mathbb {E} _ {k \sim \pi (\langle Z \rangle)} [ - s \widehat {\operatorname {M M D}} _ {k} ^ {2} (Z) ]\right) \quad \text {i n t r o d u c e d u m m y} s > 0 \\ \leq \frac {1}{s} \log \circ \exp \left(\mathbb {E} _ {k \sim \pi (\langle Z \rangle)} - s \widehat {\operatorname {M M D}} _ {k} ^ {2} (Z)\right) \quad \text {J e s s e n o n} \\ \leq_ {1 - \delta} \frac {4 \kappa^ {2} s}{n (n - 1)} + \frac {1}{s} \log \frac {1}{\delta}, \\ \end{array}
$$
where the last line follows from Equation (9) replacing  $\lambda$  by dummy variable  $-s$  for  $0 < s < \sqrt{n(n - 1)} / 8\sqrt{2}\kappa$ ; we importantly note that this bound held for negative  $\lambda$  by Theorem 7.
By following the same process for  $\widehat{\mathrm{FUSE}}_N$  and instead using Equation (10), also holding for potentially negative  $\lambda$ , we obtain
$$
- \widehat {\mathrm {F U S E}} _ {N} (\mathbf {Z}) \leq_ {1 - \delta} \frac {1 6 s}{n (n - 1)} + \frac {1}{s} \log \frac {1}{\delta}
$$
for  $0 < s < \sqrt{n(n - 1)} / 16\sqrt{2}$ .
For the small  $\delta$  we generally use, these are tightest when  $s$  is at its maximum permissible value, so we substitute these values for the theorem statement.
Under Permutation. To prove the equivalent result for  $\sigma Z$  under permutations, we replace  $Z$  by  $\sigma Z$  and our application of Markov's inequality introducing an expectation over  $Z$  with one over  $\sigma$ . This changes nothing else in the derivations.
D Concentration under the alternative
We give the exponential convergence bounds for  $\widehat{\mathrm{FUSE}}_1$  under the alternative and relate it to its mean. In the following we will assume that  $\mathcal{K}$  is a class of kernels bounded by  $0 < \kappa < \infty$ , so that  $\widehat{\mathrm{FUSE}}_1 \in [-2\kappa, 2\kappa]$ . We also introduce the following quantity (for fixed, data-free prior  $\pi$ ) which is closely related to the expectation of  $\widehat{\mathrm{FUSE}}_1$ ,
$$
\mathrm {F U S E} _ {1} := \sup  _ {\rho \in \mathcal {M} _ {+} ^ {1} (\mathcal {K})} \mathrm {M M D} _ {K _ {\rho}} ^ {2} (p, q) - \frac {\mathrm {K L} (\rho , \pi)}{\lambda}.
$$
Theorem 9.  $\mathrm{FUSE}_1$  is bounded in the following ways:
$$
\mathrm {F U S E} _ {1} \leq \mathbb {E} _ {\boldsymbol {Z}} \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) \leq \mathrm {F U S E} _ {1} + 8 \kappa^ {2} \lambda \left(\frac {1}{n} + \frac {1}{m}\right),
$$
$$
\mathrm{MMD}_{K_{\pi}}^{2}(p,q)\leq \mathrm{FUSE}_{1}\leq \sup_{\rho \in \mathcal{M}^{1}_{+}(\mathcal{K}):\mathrm{KL}(\rho ,\pi) <   \infty}\mathrm{MMD}_{K_{\rho}}^{2}(p,q)\leq \sup_{k\in \operatorname {supp}(\pi)}\mathrm{MMD}_{k}^{2}(p,q).
$$
Under the null hypothesis  $\mathrm{FUSE}_1 = 0$
We can now state concentration results for  $\widehat{\mathrm{FUSE}}_1$  in terms of  $\mathrm{FUSE}_1$ .
Theorem 10. With probability at least  $1 - \delta$  over the sample
$$
\widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) - \mathrm {F U S E} _ {1} \leq 8 \kappa^ {2} \lambda \left(\frac {1}{n} + \frac {1}{m}\right) + \frac {\log \delta^ {- 1}}{\lambda}
$$
and with the same probability,
$$
\mathrm {F U S E} _ {1} - \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) \leq 2 \kappa \sqrt {8 \left(\frac {1}{n} + \frac {1}{m}\right) \log \delta^ {- 1}}.
$$
Note on the proofs. The "bounded difference lemma" (Theorem 11) we give below is not the same as the bounded difference inequality, though it is closely related. In fact, our result could be used as an intermediate step in proving the latter, but we note that this would not be the usual method (e.g. Boucheron et al., 2013, use an "entropy" method instead), and the converse is not true. The fact that we have to prove a variant form of an existing concentration inequality to get concentration for our log-sum-exp statistics is similar to how the same is required in PAC-Bayesian proofs, where e.g. PAC-Bayes Bernstein inequalities also require modified proof techniques that mirror those used to prove the usual Bernstein inequalities.
The usual bounded difference inequality cannot be used to prove Theorem 9, since it is not a concentration bound. It also does not give the concentration we need in Theorem 10, since the FUSE statistics do not have the bounded difference property, only  $\widehat{\mathrm{MMD}}_k^2$  does; this is why our proofs cannot use a "plug-in" concentration bound.
Theorem 11 (Bounded Difference Lemma). A function  $f$  has the bounded difference property there exist constants  $L_{\ell} < \infty, \ell \in [n]$  such that
$$
\left| f \left(z _ {1}, \dots , z _ {k}, \dots , z _ {n}\right) - f \left(z _ {1}, \dots , z _ {\ell}, \dots , z _ {n}\right) \right| \leq L _ {\ell}
$$
for any choices of  $z_{1},\ldots ,z_{n},z^{\prime}_{\ell}$  , and  $\ell \in [n]$
For such a function and any independent random variables  $Z_{1},\ldots ,Z_{n}$  and  $t\in \mathbb{R}$  (subject to appropriate measurability restrictions)
$$
\mathbb {E} \exp (t (f (Z _ {1}, \dots , Z _ {n}) - \mathbb {E} f (Z _ {1}, \dots , Z _ {n}))) \leq \exp \left(\frac {1}{8} t ^ {2} \sum_ {\ell = 1} ^ {n} L _ {\ell} ^ {2}\right).
$$
Proof of Theorem 11. We introduce the Doob construction, defining
$$
D _ {\ell} = \mathbb {E} [ f (Z _ {1}, \dots , Z _ {n}) | Z _ {1}, \dots , Z _ {\ell} ] - \mathbb {E} [ f (Z _ {1}, \dots , Z _ {n}) | Z _ {1}, \dots , Z _ {\ell - 1} ]
$$
This is a martingale difference sequence with
$$
\sum_ {\ell = 1} ^ {n} D _ {\ell} = f (Z _ {1}, \dots , Z _ {n}) - \mathbb {E} f (Z _ {1}, \dots , Z _ {n}).
$$
It is shown by (for example) Wainwright (2019, Ch. 2.2, p.37) that  $D_{\ell}$  lies in an interval of length at most  $L_{\ell}$  by the bounded differences assumption. Thus, applying iterated expectation and Hoeffding's lemma for the MGF of bounded random variables,
$$
\begin{array}{l} \mathbb {E} \exp (t (f (Z _ {1}, \dots , Z _ {n}) - \mathbb {E} f (Z _ {1}, \dots , Z _ {n}))) = \mathbb {E} \exp \left(\lambda \sum_ {\ell = 1} ^ {n} D _ {\ell}\right) \\ = \mathbb {E} \left[ \exp \left(\lambda \sum_ {\ell = 1} ^ {n - 1} D _ {\ell}\right) \mathbb {E} \left[ \exp \left(\lambda D _ {n}\right) \mid Z _ {1}, \dots , Z _ {n - 1} \right] \right] \\ \leq \mathbb {E} \left[ \exp \left(\lambda \sum_ {\ell = 1} ^ {n - 1} D _ {\ell}\right) \right] \exp \left(\frac {1}{8} \lambda^ {2} L _ {n} ^ {2} D _ {n}\right) \\ \leq \exp \left(\frac {1}{8} t ^ {2} \sum_ {\ell = 1} ^ {n} L _ {\ell} ^ {2}\right). \\ \end{array}
$$
The MGF of  $\widehat{\mathrm{MMD}^2}$  can then be bounded using the following result, proved via the above.
Theorem 12. For bounded kernel  $k \leq \kappa$ ,  $t \in \mathbb{R}$  and sample sizes  $m, n$ ,
$$
\mathbb {E} \exp (t (\widehat {\operatorname {M M D}} _ {k} ^ {2} (\boldsymbol {X}, \boldsymbol {Y}) - \operatorname {M M D} _ {k} ^ {2} (p, q))) \leq \exp \left(8 t ^ {2} \kappa^ {2} \left(\frac {1}{m} + \frac {1}{n}\right)\right).
$$
Proof of Theorem 12. We show that  $\widehat{\mathrm{MMD}}_k^2 (x,y)$  has the bounded differences property and then apply Theorem 11. Denote by  $x^{\setminus \ell}$  for  $\ell \in [n]$  that the  $\ell$ -th example in the  $x$  sample is changed. Then
$$
\begin{array}{l} \left| \widehat {\operatorname {M M D}} _ {k} ^ {2} (x, y) - \widehat {\operatorname {M M D}} _ {k} ^ {2} \left(x ^ {\backslash \ell}, y\right) \right| \\ = \left| \frac {2}{n (n - 1)} \sum_ {i \in [ n ] \backslash \{\ell \}} \left(k \left(x _ {\ell}, x _ {i}\right) - k \left(x _ {\ell} ^ {\prime}, x _ {i}\right)\right) - \frac {2}{m n} \sum_ {j = 1} ^ {m} \left(k \left(x _ {\ell}, y _ {j}\right) - k \left(x _ {\ell} ^ {\prime}, y _ {j}\right)\right) \right| \\ \leq \frac {2}{n (n - 1)} \sum_ {i \in [ n ] \backslash \{\ell \}} | k (x _ {\ell}, x _ {i}) - k (x _ {\ell} ^ {\prime}, x _ {i}) | + \frac {2}{m n} \sum_ {j = 1} ^ {m} | k (x _ {\ell}, y _ {j}) - k (x _ {\ell} ^ {\prime}, y _ {j}) | \\ \leq \frac {2}{n (n - 1)} (n - 1) \cdot 2 \kappa + \frac {2}{m n} m \cdot 2 \kappa \\ = \frac {8 \kappa}{n}. \\ \end{array}
$$
A similar process for the  $y$  sample gives bounds of  $8\kappa /m$ , so that  $\widehat{\mathrm{MMD}^2}$  has the bounded differences property with
$$
\sum_ {\ell = 1} ^ {n + m} L _ {\ell} ^ {2} \leq n \cdot \left(\frac {8 \kappa}{n}\right) ^ {2} + m \cdot \left(\frac {8 \kappa}{m}\right) ^ {2} = 6 4 \kappa^ {2} \left(\frac {1}{n} + \frac {1}{m}\right).
$$
Proof of Theorem 9. For the first lower bound, note
$$
\mathbb {E} _ {\boldsymbol {Z}} \widehat {\operatorname {F U S E}} _ {1} (\boldsymbol {Z}) = \mathbb {E} _ {\boldsymbol {Z}} \sup  _ {\rho} \mathbb {E} _ {\rho} [ \widehat {\operatorname {M M D}} ^ {2} ] - \frac {\mathrm {K L}}{\lambda} \geq \sup  _ {\rho} \mathbb {E} _ {\boldsymbol {Z}, \rho} [ \widehat {\operatorname {M M D}} ^ {2} ] - \frac {\mathrm {K L}}{\lambda} = \operatorname {F U S E} _ {1}.
$$
For the upper bound, note that
$$
\mathbb {E} _ {\boldsymbol {Z}} \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) \leq \left[ \frac {1}{\lambda} \log \left(\mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi} \left[ e ^ {\lambda \overline {{\mathrm {M M D}}} _ {k} ^ {2}} \right]\right) \right]
$$
$$
\leq \left[ \frac {1}{\lambda} \log \left(\mathbb {E} _ {k \sim \pi} \mathbb {E} _ {\mathcal {Z}} \left[ e ^ {\lambda \overline {{\mathrm {M M D}}} _ {k} ^ {2}} \right]\right) \right]
$$
Independence of  $\pi$  from  $Z$
$$
\leq \frac {1}{\lambda} \log \left(\mathbb {E} _ {k \sim \pi} \left[ e ^ {\lambda \operatorname {M M D} _ {k} ^ {2} + 8 \lambda^ {2} \kappa^ {2} (n ^ {- 1} + m ^ {- 1})} \right]\right)
$$
Theorem 12
$$
= \mathrm {F U S E} _ {1} + 8 \kappa^ {2} \lambda \left(\frac {1}{n} + \frac {1}{m}\right).
$$
The lower bound on  $\mathrm{FUSE}_1$  is obtained by relaxing the supremum with  $\rho = \pi$ . The upper bounds come since the supremum over  $\rho \in \mathcal{M}_{+}^{1}(\mathcal{K}) : \mathrm{KL}(\rho, \pi) < \infty$  of  $\mathrm{MMD}^2$  is clearly greater than the KL-regularised version. Under the null hypothesis,  $\mathrm{MMD}_k(p, q) = 0$  for every kernel (regardless of them being characteristic or not), so  $\mathrm{FUSE}_1 = 0$ .
Proof of Theorem 10. For the upper bound,
$$
\begin{array}{l} \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) = \left[ \frac {1}{\lambda} \log \left(\mathbb {E} _ {k \sim \pi} \left[ e ^ {\lambda \widehat {\mathrm {M M D}} _ {k} ^ {2}} \right]\right) \right] \\ \leq_ {1 - \delta} \left[ \frac {1}{\lambda} \log \left(\mathbb {E} _ {\boldsymbol {Z}} \mathbb {E} _ {k \sim \pi} \left[ e ^ {\widehat {\lambda \operatorname {M M D} _ {k} ^ {2}}} \right]\right) \right] + \frac {\log \delta^ {- 1}}{\lambda} \\ \leq \left[ \frac {1}{\lambda} \log \left(\mathbb {E} _ {k \sim \pi} \mathbb {E} _ {\boldsymbol {Z}} \left[ e ^ {\widehat {\lambda \operatorname {M M D}} _ {k} ^ {2}} \right]\right) \right] + \frac {\log \delta^ {- 1}}{\lambda} \\ \leq \frac {1}{\lambda} \log \left(\mathbb {E} _ {k \sim \pi} \left[ e ^ {\lambda \operatorname {M M D} _ {k} ^ {2} + 8 \lambda^ {2} \kappa^ {2} (m ^ {- 1} + n ^ {- 1})} \right]\right) + \frac {\log \delta^ {- 1}}{\lambda} \\ = \mathrm {F U S E} _ {1} + 8 \kappa^ {2} \lambda \left(\frac {1}{n} + \frac {1}{m}\right) + \frac {\log \delta^ {- 1}}{\lambda}. \\ \end{array}
$$
For the lower bound, let  $\rho^{*}$  be the value of  $\rho$  achieving the supremum in the dual form of  $\mathrm{FUSE}_1$  (which we note is independent of the sample), so that
$$
\begin{array}{l} \mathrm {F U S E} _ {1} - \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) = \sup _ {\rho} \left\{\mathbb {E} _ {\rho^ {*}} [ \mathrm {M M D} _ {k} ^ {2} ] - \frac {\mathrm {K L} (\rho , \pi)}{\lambda} \right\} - \sup _ {\rho} \left\{\mathbb {E} _ {\rho} [ \widehat {\mathrm {M M D}} _ {k} ^ {2} ] - \frac {\mathrm {K L} (\rho , \pi)}{\lambda} \right\} \\ = \mathbb {E} _ {\rho^ {*}} \left[ \mathrm {M M D} _ {k} ^ {2} \right] - \frac {\mathrm {K L} (\rho^ {*} , \pi)}{\lambda} - \sup  _ {\rho} \left\{\mathbb {E} _ {\rho} [ \widehat {\mathrm {M M D}} _ {k} ^ {2} ] - \frac {\mathrm {K L} (\rho , \pi)}{\lambda} \right\} \\ \leq \mathbb {E} _ {\rho^ {*}} \left[ \mathrm {M M D} _ {k} ^ {2} \right] - \frac {\mathrm {K L} (\rho^ {*} , \pi)}{\lambda} - \left(\mathbb {E} _ {\rho^ {*}} \widehat {\left[ \mathrm {M M D} _ {k} ^ {2} \right]} - \frac {\mathrm {K L} (\rho^ {*} , \pi)}{\lambda}\right) \\ = \mathbb {E} _ {\rho^ {*}} \left[ \mathrm {M M D} _ {k} ^ {2} - \widehat {\mathrm {M M D}} _ {k} ^ {2} \right] \\ = \operatorname {M M D} _ {K _ {\rho^ {*}}} ^ {2} (p, q) - \widehat {\operatorname {M M D}} _ {K _ {\rho^ {*}}} ^ {2} (\boldsymbol {Z}). \\ \end{array}
$$
Now the above is for fixed kernel  $K_{\rho^*}$  independent of the data, so by Markov's inequality and Theorem 12, for any  $s > 0$  we find
$$
\begin{array}{l} \mathrm {M M D} _ {K _ {\rho^ {*}}} ^ {2} (p, q) - \widehat {\mathrm {M M D}} _ {K _ {\rho^ {*}}} ^ {2} (\boldsymbol {Z}) \\ \leq_ {1 - \delta} s ^ {- 1} \log \mathbb {E} _ {Z} \exp \left(s \left(\operatorname {M M D} _ {K _ {\rho^ {*}}} ^ {2} (p, q) - \widehat {\operatorname {M M D}} _ {K _ {\rho^ {*}}} ^ {2} (\boldsymbol {Z})\right)\right) + s ^ {- 1} \log \delta^ {- 1} \\ \leq 8 s \kappa^ {2} \left(\frac {1}{m} + \frac {1}{n}\right) + s ^ {- 1} \log \delta^ {- 1}. \\ \end{array}
$$
Optimising for  $s$  yields  $s = \kappa^{-1}\sqrt{\log(1 / \delta)} /\sqrt{8(1 / m + 1 / n)}$  which gives the desired result.
E Power Analysis
E.1 General Recipe for Power Analysis of Permutation Tests
The general outline for power analysis consists of the following: First we start with the Type II error probability. Then we attempt to upper bound it by iteratively applying the following simple lemma to the different terms.
Monotonicity in High Probability. Let  $X, Y$  be r.v.s, such that  $X \leq Y$  w.p.  $\geq 1 - \delta$ . Then  $\mathbb{P}(X \geq c) \leq \mathbb{P}(Y \geq c) + \delta$ . This result can be applied both when  $X \leq Y$  a.s. giving  $\delta = 0$ , or if  $Y = a$  is constant.
Iteratively applying this lemma gives  $\mathbb{P}(\mathrm{Type~II})\leq N\delta$  , and then we set  $N\delta = \beta$
As a first step in the above process we will use the following useful result (adapted from Kim et al., 2022) to convert from the random permutations we use in practice to the full set of permutation. This result shows that when  $B$  is taken sufficiently large (roughly  $\Omega (\alpha^{-2}\log (\beta^{-1}))$ ), the only changes to the final power results will be in constants multiplying  $\alpha$  and  $\beta$ .
Theorem 13 (From Randomised to Deterministic Permutations). Suppose  $G = (g_{1},\ldots ,g_{B + 1})$  consists of  $B$  uniformly drawn permutations from  $\mathcal{G}$ , plus the identity permutation as  $g_{B + 1}$ . Then
$$
\mathbb {P} \left(\tau (\boldsymbol {Z}) \leq \underset {1 - \alpha , G} {\text {q u a n t i l e}} \tau (g \boldsymbol {Z})\right) \leq \mathbb {P} \left(\tau (\boldsymbol {Z}) \leq \underset {1 - \alpha_ {B}, \mathcal {G}} {\text {q u a n t i l e}} \tau (g \boldsymbol {Z})\right) + \delta
$$
where  $1 - \alpha_{B} = \frac{B + 1}{B} (1 - \alpha) + \sqrt{\frac{\log(2 / \delta)}{2B}}$ .
We note that provided  $B \geq 8\alpha^{-2}\log(2/\delta)$ , then  $1 - \alpha_{B} \leq 1 - \alpha/2$ .
Proof. We note the Dvoretzky-Kiefer-Wolffowitz inequality Dvoretzky et al. (1956); Massart (1990) for empirical CDF  $F_{n}$  of  $n$  samples from original CDF  $F$ :
$$
\mathbb {P} \left(\sup  _ {x} | F _ {n} (x) - F (x) | \geq t\right) \leq 2 e ^ {- 2 n t ^ {2}} \tag {12}
$$
for every  $t > 0$
The permutation CDF for a group  $\mathcal{G}$  take the form
$$
F _ {\mathcal {G}} (x) = \frac {1}{| \mathcal {G} |} \sum_ {g \in \mathcal {G}} \mathbf {1} \{\tau (g Z) \leq x \}
$$
and given sample  $\widetilde{G} = (g_1, \ldots, g_B)$  i.i.d. uniformly from  $\mathcal{G}$  (this excludes the identity permutation added to  $G$ ), the empirical CDF is
$$
\widehat {F} _ {\widetilde {G}} (x) = \frac {1}{B} \sum_ {g \in \widetilde {G}} \mathbf {1} \{\tau (g Z) \leq x \}
$$
We can also write  $\widehat{F}_G(x)$  including the identity permutation as  $g_{B + 1}$ , and note that  $\widehat{F}_G(x) =$
Define the good event
$$
\mathcal {A} = \left\{\sup  _ {x} | F _ {\widetilde {G}} (x) - F _ {\mathcal {G}} (x) | \leq \sqrt {\frac {\log (2 / \delta)}{2 B}} \right\}
$$
which holds with probability at least  $1 - \delta$  by Equation (12). Given  $\mathcal{A}$ , we have
$$
\begin{array}{l} q _ {G} := \underset {1 - \alpha , g \in G} {\text {q u a n t i l e}} \tau (g \mathbf {Z}) = \inf  \left\{r \in \mathbb {R}: \frac {1}{B + 1} \sum_ {g \in G} \mathbf {1} \{\tau (g \mathbf {Z}) \leq r \} \geq 1 - \alpha \right\} \\ \leq \inf  \{r \in \mathbb {R}: \frac {1}{B + 1} \sum_ {g \in G} \mathbf {1} \{\tau (g Z) \leq r \} \geq 1 - \alpha \} \\ \end{array}
$$
$$
\begin{array}{l} = \inf  \left\{r \in \mathbb {R}: \widehat {F} _ {\widetilde {G}} (r) \geq \frac {B + 1}{B} (1 - \alpha) \right\} \\ \leq \inf  \left\{r \in \mathbb {R}: \widehat {F} _ {\mathcal {G}} (r) \geq \frac {B + 1}{B} (1 - \alpha) + \sqrt {\frac {\log (2 / \delta)}{2 B}} \right\} \\ = \underset {1 - \alpha_ {B}, g \in \mathcal {G}} {\text {q u a n t i l e}} \tau (g \mathbf {Z}) =: q \\ \end{array}
$$
where we have defined  $\alpha_{B}$  as above.
Overall we find  $\mathbb{P}(\tau \leq q_G) \leq \mathbb{P}(\tau \leq q_G|\mathcal{A}) + \mathbb{P}(\mathcal{A}^c) \leq \mathbb{P}(\tau \leq q) + \delta$ .
E.2 Variance of  $\widehat{\mathrm{MMD}}_k^2$
In proving our power results it is necessary to upper bound the variance of  $\widehat{\mathrm{MMD}}_k^2$ .
Theorem 14. For any kernel  $k$  upper bounded by  $\kappa$ , if  $n \leq m \leq cn$  for  $c \geq 1$ , there exists a universal constant  $C > 0$  depending only on  $c$ , such that
$$
\mathbb {V} [ \widehat {\mathrm {M M D}} _ {k} ^ {2} ] \leq C \left(\frac {4 \kappa \mathrm {M M D} _ {k} ^ {2}}{n} + \frac {\kappa^ {2}}{n ^ {2}}\right)
$$
Proof. We define
$$
\begin{array}{l} \sigma_ {1 0} ^ {2} = \mathbb {V} _ {X} \left(\mathbb {E} _ {X ^ {\prime}, Y, Y ^ {\prime}} [ h (X, X ^ {\prime}, Y, Y ^ {\prime}) ]\right) \\ \sigma_ {0 1} ^ {2} = \mathbb {V} _ {Y} \left(\mathbb {E} _ {X, X ^ {\prime}, Y ^ {\prime}} [ h (X, X ^ {\prime}, Y, Y ^ {\prime}) ]\right) \\ \sigma_ {1 1} ^ {2} = \max  \{\mathbb {E} [ k ^ {2} (X, X ^ {\prime}) ], \mathbb {E} [ k ^ {2} (X, Y) ], \mathbb {E} [ k ^ {2} (Y, Y ^ {\prime}) ] \}. \\ \end{array}
$$
From a well-known bound (based on Lee, 1990, Equation 2, p.38; see also Kim et al., 2022, Appendix F, Equation 59 or Schrab et al., 2023, Proposition 3),
$$
\begin{array}{l} \mathbb {V} [ \widehat {\mathrm {M M D}} _ {k} ^ {2} ] \leq C \left(\frac {\sigma_ {1 0} ^ {2}}{m} + \frac {\sigma_ {0 1} ^ {2}}{n} + \sigma_ {1 1} ^ {2} \left(\frac {1}{m} + \frac {1}{n}\right) ^ {2}\right) \\ \leq C \left(\frac {\sigma_ {1 0} ^ {2}}{n} + \frac {\sigma_ {0 1} ^ {2}}{n} + \frac {\sigma_ {1 1} ^ {2}}{n ^ {2}}\right) \\ \end{array}
$$
where we used that  $n \leq m \leq cn$ . and the result in red above, and the boundedness of the kernel for the final term. This gives the further bound
$$
\begin{array}{l} \sigma_ {1 0} ^ {2} = \operatorname {v a r} _ {X} \left(\mathbb {E} _ {X ^ {\prime}, Y, Y ^ {\prime}} [ h (X, X ^ {\prime}, Y, Y ^ {\prime}) ]\right) \\ = \mathbb {E} _ {X} \left[ \left(\mathbb {E} _ {X ^ {\prime}, Y, Y ^ {\prime}} [ h (X, X ^ {\prime}, Y, Y ^ {\prime}) ]\right) ^ {2} \right] \\ = \mathbb {E} _ {X} \left[ \left(\mathbb {E} _ {X ^ {\prime}, Y, Y ^ {\prime}} [ \langle \phi (X) - \phi (Y), \phi (X ^ {\prime}) - \phi (Y ^ {\prime}) \rangle ]\right) ^ {2} \right] \\ = \mathbb {E} _ {X} \left[ \left\langle \phi (X) - \mu_ {Q}, \mu_ {P} - \mu_ {Q} \right\rangle^ {2} \right] \\ \leq \left(\mathbb {E} _ {X} \left[ \| \phi (X) - \mu_ {Q} \| ^ {2} \right]\right) \| \mu_ {P} - \mu_ {Q} \| ^ {2} \\ \leq 2 \kappa \| \mu_ {P} - \mu_ {Q} \| ^ {2} \\ = 2 \kappa \mathrm {M M D} _ {k} ^ {2}, \\ \end{array}
$$
a similar result for  $\sigma_{01}$ , and the simple bound  $\sigma_{11}^2 \leq \kappa^2$ .
Proof. Define  $\alpha_{B}$  as in Theorem 13, noting that  $1 - \alpha_{B} < 1 - \alpha /2$  under the assumption  $B\geq 8\alpha^{-2}\log (4 / \beta)$ . From Theorem 13 we can consider the full permutation set as