SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kernels. As explained in Section 6 (\$Distribution on Kernels), for MMD-Fuse, we use Gaussian and Laplace kernels with bandwidths in  $\{q_{5\%}^r +i(q_{95\%}^r -q_{5\%}^r) / 9:i = 0,\ldots ,9\}$  where  $q_{5\%}^{r}$  is half the  $5 \%$  quantile of all the inter-sample distances  $\{\| z - z^{\prime}\| _r:z,z^{\prime}\in \mathbf{Z}\}$  with  $r = 1$  and  $r = 2$  for Laplace and Gaussian kernels, respectively. Similarly,  $q_{95\%}^{r}$  is twice the  $95\%$  quantile.
MMD-Split selects a Gaussian kernel on half of the data with bandwidth in  $\{q_{5\%}^r + i(q_{95\%}^r - q_{5\%}^r) / 99 : i = 0, \dots, 99\}$  by maximizing a proxy for asymptotic power which is the ratio of the estimated MMD with its estimated standard deviation under the alternative (Liu et al., 2020, Equation 3). The MMD test is then run on the other half with the selected kernel.
CTT and MMD-Median both use a Gaussian kernel with bandwidth the median of  $\{\| z - z' \|_2 : z, z' \in \mathbf{Z}\}$ , while ACTT is run with Gaussian kernels with bandwidths in  $\{q_{5\%}^2 + i(q_{95\%}^2 - q_{5\%}^2) / 9 : i = 0, \ldots, 9\}$ .
The MMDAgg and MMDAggInc tests are run with their default implementations, which similarly use collections of 20 kernels split equally between Gaussian and Laplace kernels with ten bandwidths each, but they use a different (non-uniform) discretisation of the intervals  $[q_{95\%}^r, q_{5\%}^r]$ .
Permutations. For MMD-Median, MMD-Split, and MMD-FUSE, we use 2000 permutations to estimate the quantiles. MMDAgg and MMDAggInc use  $2000 + 2000$  and  $500 + 500$  permutations, respectively, to approximate the quantiles and the multiple testing correction. The CTT and ACTT tests are run with 39 and  $299 + 200$  permutations, respectively. AutoML uses 10000 permutations
and MMD-D 100 of them. The ME and SCF tests use asymptotic quantiles. We recall that using a higher number of permutations for the quantile does not necessarily lead to a more powerful test (Rindt et al., 2021, Section 7).
A.3 Details on the experiments of Section 6
In this section, we present figures illustrating the four experimental settings described in Section 6 with results in Figure 1: Mixture of Gaussians (Figure 2), Perturbed Uniform  $d = 1$  (Figure 3), Perturbed Uniform  $d = 2$  (Figure 4), Galaxy MNIST (Figure 5), and CIFAR 10 vs 10.1 (Figure 6).
A.4 Experiment varying the number of kernels
In Figure 7, we provide an additional perturbed uniform experiment where we vary the number of kernels for MMD-FUSE and for MMDAgg. We consider the task of detecting six one-dimensional perturbations with amplitude  $a = 0.5$  for sample sizes  $m = n = 500$  while varying the number of Gaussian and Laplace kernels from 10 to 1000.
Firstly, we observe that MMD-FUSE achieves higher power than MMDAgg in this setting. Secondly, both MMD-FUSE and MMDAgg retain their power when increasing the number of bandwidths. This matches the empirical observation of Schrab et al. (2023, Section 5.7) that a discretisation of 10 bandwidths is enough to capture all the information in certain two-sample problems and that using a finer discretisation (which is computationally more expensive) does not improve the test power.
A.5 Time complexity and runtimes
The time complexity of MMD-FUSE is
$$
\mathcal {O} \left(K B \left(m ^ {2} + n ^ {2}\right)\right) \tag {5}
$$
where  $m$  and  $n$  are the sizes of the two samples,  $K$  is the number of kernels fused, and  $B$  is the number of permutations to estimate the quantile. Note that this is an improvement over the time complexity of MMDAgg (Schrab et al., 2023)
$$
\mathcal {O} \left(K \left(B + B ^ {\prime}\right) \left(m ^ {2} + n ^ {2}\right)\right) \tag {6}
$$
where the extra parameter  $B'$  corresponds to the number of permutations used to estimate the multiple testing correction, often set as  $B' = B$  in practice (Schrab et al., 2023, Section 5.2). We indeed observe in Figure 8 that MMD-FUSE runs twice as fast as MMDAgg. $^{7}$
While the runtimes of most tests should not depend on the type of data (only on the sample size and on the dimension), we note that the runtimes of tests relying on optimisation (e.g. ME & SCF) can be affected. In the experiment of Figure 8, we consider samples from multivariate Gaussians centred at zero with covariance matrices  $I_{d}$  and  $\sigma I_{d}$  with  $\sigma = 1.1$ . We vary both the sample sizes and the dimensions.
The two constants hidden in the  $\mathcal{O}$ -notations of Equations (5) and (6) are exactly the same, which is indeed verified by our empirical observations.
B Closed Form Mean Kernel
We have observed in the main text that in some cases where the prior has non-finite support,  $\widehat{\mathrm{FUSE}}_1$  still gives a straightforwardly expressed statistic. This happens because for certain kernel choices, the expectation of the kernel with respect to some parameter is still a closed form kernel. We give one example of such a statistic here.
Theorem 4. For any fixed  $\alpha >0$ , the following is an example of a  $\widehat{\mathrm{FUSE}}_1$  statistic:
$$
\widehat {\mathrm {F U S E}} _ {1} ^ {r q} (\boldsymbol {Z}) = \sup _ {R > 0} \widehat {\mathrm {M M D}} _ {k _ {r q (\alpha , \sqrt {R} \eta_ {0})}} ^ {2} (\boldsymbol {Z}) - \alpha \cdot \frac {\log R + 1 / R - 1}{\lambda}
$$
where  $k_{rq(\alpha, \eta)} = (1 + \| x - y\|^2 / 2\eta^2)^{-\alpha}$  is a rational quadratic kernel, and  $\eta_0$  is the "prior" bandwidth (which can essentially be absorbed into the data scaling).
Proof of Theorem 4. Firstly, we define a general rational quadratic kernel
$$
k _ {r q (\alpha , \eta)} (x, y) = \left(1 + \frac {\| x - y \| _ {2} ^ {2}}{2 \eta^ {2}}\right) ^ {- \alpha}.
$$
Note that  $k_{g}(r) = e^{-\tau r^{2} / 2}$  with  $r = \| x - y\| _2$  is a bounded kernel with parameter  $\tau$ . If we take the expectation of  $\tau$  with respect to a Gamma distribution,
$$
\begin{array}{l} \mathbb {E} _ {\tau \sim \Gamma (\alpha , \beta)} e ^ {- \tau r ^ {2} / 2} = \frac {\beta^ {\alpha}}{\Gamma (\alpha)} \int_ {- \infty} ^ {\infty} \tau^ {\alpha - 1} \exp (- \tau \beta) \exp (- \tau r ^ {2} / 2) d \tau \\ = \frac {\beta^ {\alpha}}{\Gamma (\alpha)} \Gamma (\alpha) \left(\beta + \frac {r ^ {2}}{2}\right) ^ {- \alpha} \\ = \left(1 + \frac {r ^ {2}}{2 \beta}\right) ^ {- \alpha} = k _ {r q (\alpha , \sqrt {\beta})} (r) \\ \end{array}
$$
where  $\Gamma$  denotes the gamma function. The KL divergence between Gamma distributions is
$$
\operatorname {K L} (\Gamma (\alpha , \beta), \Gamma (\alpha_ {0}, \beta_ {0})) = (\alpha - \alpha_ {0}) \psi (\alpha) + \log \frac {\Gamma (\alpha_ {0})}{\Gamma (\alpha)} + \alpha_ {0} \log R + \alpha \left(\frac {1}{R} - 1\right)
$$
where  $R = \beta /\beta_{0}$  and  $\psi (\alpha)\coloneqq \Gamma^{\prime}(\alpha) / \Gamma (\alpha)$  denotes the digamma function. Using the dual form of  $\widehat{\mathrm{FUSE}_1}$  we find that
$$
\widehat {\mathrm {F U S E}} _ {1} = \sup _ {\alpha > 0, \beta > 0} \widehat {\mathrm {M M D}} _ {k _ {r q (\alpha , \sqrt {\beta})}} ^ {2} (\pmb {Z}) - \frac {\mathrm {K L} (\Gamma (\alpha , \beta) , \Gamma (\alpha_ {0} , \beta_ {0}))}{\lambda}.
$$
Restricting  $\alpha = \alpha_0$ , prior  $\beta_0 = \eta_0^2$  and setting  $\beta = R\eta_0^2$  gives the result.
C Null and Permutation Concentration Results
In this section we prove the following concentration results under the null (or equivalently, under the permutation distribution). These results show that for appropriate choices of  $\lambda$ , both statistics converge to zero at rate at least  $\mathcal{O}(1/n)$  under the null or permutations. We recall that without loss of generality  $n \leq m$ .
Theorem 5. For bounded kernels  $k \leq \kappa < \infty$  and parameter  $\lambda$ , with probability at least  $1 - \delta$  over a sample  $Z$  from the null,
$$
\widehat {\mathrm {F U S E}} _ {1} (\mathbf {Z}) \leq \frac {4 \kappa^ {2} \lambda}{n (n - 1)} + \frac {\log \frac {1}{\delta}}{\lambda}
$$
provided  $0 < \lambda < \sqrt{n(n - 1)} / 8\sqrt{2}\kappa$ , and
$$
- \widehat {\mathrm {F U S E}} _ {1} (\boldsymbol {Z}) \leq \frac {1 + 3 2 \log \frac {1}{\delta}}{\sqrt {2 n (n - 1)}} \cdot \frac {\kappa}{2}.
$$
For potentially unbounded kernels and parameter  $\lambda$ , with probability at least  $1 - \delta$  over a sample  $Z$  from the null,
$$
\widehat {\mathrm {F U S E}} _ {N} (\boldsymbol {Z}) \leq \frac {1 6 \lambda}{n (n - 1)} + \frac {\log \frac {1}{\delta}}{\lambda},
$$
provided  $0 < \lambda < \sqrt{n(n - 1)} / 16\sqrt{2}$ , and
$$
- \widehat {\mathrm {F U S E}} _ {N} (\boldsymbol {Z}) \leq \frac {1 + 3 2 \log \frac {1}{\delta}}{\sqrt {2 n (n - 1)}}.
$$
The above bounds are also valid for  $\widehat{\mathrm{FUSE}}_1(\sigma Z)$ ,  $\widehat{\mathrm{FUSE}}_N(\sigma Z)$  and any fixed  $Z$  (potentially non-null) under permutation by  $\sigma \sim \mathrm{Uniform}(\mathfrak{S}_{n + m})$ .
While the upper bounds depend critically upon our choice of  $\lambda$ , the lower bounds instead hold regardless of  $\lambda$ . Choosing  $\lambda \asymp \sqrt{n(n - 1)} \asymp n$  gives the desired upper bound rates  $\mathcal{O}(1/n)$ .
C.1 Sub-Gaussian Chaos Theorem
We provided constants for Theorem 5, even though they are not strictly necessary, as we hope they could be useful when adapting FUSE statistics to other settings. In particular, the constants help understand for which values of  $\lambda$  the bounds hold, and to understand the relation between the two different right hand side terms which can be matched by tuning  $\lambda$ . In provide these constants, in this subsection we replicate the proof of a sub-result from Rudelson and Vershynin (2013), but unlike in that paper we explicitly keep track of the constants. To be clear, Rudelson and Vershynin (2013) does not give numerical constants at all (only their existence is proved), and we need then for our statement of Theorem 5. We do not claim this as a contribution, the proof is only included so that we are not stating the numerical constants without justification.
Theorem 6 (Sub-Gaussian Chaos; adapted from Rudelson and Vershynin, 2013). Let  $X_{i}$  be mean-zero 1-sub-Gaussian variables, such that  $\log \mathbb{E}_X\exp (tX)\leq \frac{1}{2} t^2$  for every  $t\in \mathbb{R}$ . Let  $A\in \mathbb{R}^{n\times n}$  be a real symmetric matrix with zeros on the diagonal and we define the sub-Gaussian chaos
$$
W := \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} A _ {i j} X _ {i} X _ {j}.
$$
For all  $|t| \leq (4\sqrt{2}\| A \|)^{-1}$ ,
$$
\mathbb {E} _ {X} \exp (t W) \leq \exp \left(1 6 t ^ {2} \| A \| _ {F} ^ {2}\right).
$$
Proof. This proof is closely adapted from Rudelson and Vershynin (2013) with some modifications to explicitly track numerical constants.
Let  $X_{i}$  be mean-zero 1-sub-Gaussian variables, such that  $\log \mathbb{E}_X\exp (tX)\leq \frac{1}{2} t^2$  for every  $t\in \mathbb{R}$ . We are considering
$$
W := \sum_ {i, j} A _ {i j} X _ {i} X _ {j}
$$
where  $A_{ij}$  has zeros on the diagonal. We introduce independent Bernoulli random variables  $\delta_i \in \{0,1\}$  with  $\mathbb{E}\delta_i = 1/2$  and define the matrix  $A^\delta$  with entries  $(A^\delta)_{ij} = \delta_i(1 - \delta_j)A_{ij}$ . Note that  $\mathbb{E}_\delta A^\delta = \frac{1}{4} A$ , since  $\mathbb{E}\delta_i(1 - \delta_j)$  equals  $1/4$  for  $i \neq j$ .
Writing the set of indices  $\Lambda_{\delta} = \{i\in [n]:\delta_{i} = 1\}$ , we introduce
$$
W _ {\delta} := \sum_ {i, j} A _ {i j} ^ {\delta} X _ {i} X _ {j} = \sum_ {i \in \Lambda_ {\delta},   j \in \Lambda_ {\delta} ^ {c}} A _ {i j} X _ {i} X _ {j} = \sum_ {j \in \Lambda_ {\delta} ^ {c}} X _ {j} \Big (\sum_ {i \in \Lambda_ {\delta}} A _ {i j} X _ {i} \Big).
$$
By Jensen's inequality,
$$
\mathbb {E} _ {X} \exp (t W) \leq \mathbb {E} _ {X, \delta} \exp (4 t W _ {\delta}).
$$
Conditioned on  $\delta$  and  $(X_{i})_{i\in \Lambda_{\delta}}$ ,  $W_{\delta}$  is a linear combination of the mean-zero sub-gaussian random variables  $X_{j}, j\in \Lambda_{\delta}^{c}$ . It follows that this conditional distribution of  $W_{\delta}$  is sub-gaussian, and so
$$
\mathbb {E} _ {(X _ {j}) _ {j \in \Lambda_ {\delta} ^ {c}}} \exp (4 t W _ {\delta}) \leq \exp \left(\frac {1}{2} (4 t) ^ {2} \sum_ {j \in \Lambda_ {\delta} ^ {c}} \left(\sum_ {i \in \Lambda_ {\delta}} A _ {i j} X _ {i}\right) ^ {2}\right).
$$
Now introduce  $g = (g_{1},\ldots ,g_{n})\sim \mathcal{N}(0,1)$  i.i.d. draws from a normal and note that the above can also arise directly as the moment generating function of these Normal variables, so we make the further equality (for fixed  $X$  and  $\delta$ ),
$$
\exp \left(\frac {1}{2} (4 t) ^ {2} \sum_ {j \in \Lambda_ {\delta} ^ {c}} \left(\sum_ {i \in \Lambda_ {\delta}} A _ {i j} X _ {i}\right) ^ {2}\right) = \mathbb {E} _ {g} \exp \left(4 t \sum_ {j \in \Lambda_ {\delta} ^ {c}} g _ {j} \left(\sum_ {i \in \Lambda_ {\delta}} A _ {i j} X _ {i}\right) ^ {2}\right).
$$
Rearranging the terms, and using that the resulting term is a linear combination of the sub-Gaussian  $X_{i}, i \in \Lambda_{\delta}$ , we find that
$$
\begin{array}{l} \mathbb {E} _ {X, g} \exp \left(4 t \sum_ {j \in \Lambda_ {\delta} ^ {c}} g _ {j} \left(\sum_ {i \in \Lambda_ {\delta}} A _ {i j} X _ {i}\right) ^ {2}\right) = \mathbb {E} _ {X, g} \exp \left(4 t \sum_ {i \in \Lambda_ {\delta}} X _ {i} \left(\sum_ {j \in \Lambda_ {\delta} ^ {c}} A _ {i j} g _ {j}\right)\right) \\ \leq \mathbb {E} _ {X, g} \exp \left(\frac {1}{2} (4 t) ^ {2} \sum_ {i \in \Lambda_ {\delta}} \left(\sum_ {j \in \Lambda_ {\delta} ^ {\mathrm {c}}} A _ {i j} g _ {j}\right) ^ {2}\right) \\ \leq \mathbb {E} _ {X, g} \exp \left(8 t ^ {2} \sum_ {i} \left(\sum_ {j} \delta_ {i} (1 - \delta_ {j}) A _ {i j} g _ {j}\right) ^ {2}\right) \\ \leq \mathbb {E} _ {X, g} \exp \left(8 t ^ {2} \| A ^ {\delta} g \| _ {2} ^ {2}\right). \\ \end{array}
$$
Now, by the rotation invariance of the distribution of  $g$ , the random variable  $\| A^{\delta}g\|_{2}^{2}$  is distributed identically with  $\sum_{i}s_{i}^{2}g_{i}^{2}$  where  $s_i$  denote the singular values of  $A^\delta$ , so by independence
$$
\begin{array}{l} \mathbb {E} _ {X, g} \exp \left(8 t ^ {2} \| A ^ {\delta} g \| _ {2} ^ {2}\right) = \mathbb {E} _ {g} \exp \left(8 t ^ {2} \sum_ {i} s _ {i} ^ {2} g _ {i} ^ {2}\right) \\ = \prod_ {i} \mathbb {E} _ {g} \exp \left(8 t ^ {2} s _ {i} ^ {2} g _ {i} ^ {2}\right) \\ \leq \prod_ {i} \exp \left(1 6 t ^ {2} s _ {i} ^ {2}\right) \\ \end{array}
$$
where the last inequality holds if  $8t^{2} \max_{i} s_{i}^{2} \leq \frac{1}{4}$  and arises since the MGF of a Chi-squared variable,  $\mathbb{E}e^{tg^2} \leq e^{2t}$  for  $0 \leq t \leq \frac{1}{4}$ .
Since  $\max_i s_i = \| A_\delta \| \leq \| A\|$  and  $\sum_{i} s_{i}^{2} = \| A_{\delta}\|_{F}^{2} \leq \| A\|_{F}$ , we combine the above steps to find that
$$
\mathbb {E} _ {X} \exp (t W) \leq \exp \left(1 6 t ^ {2} \| A \| _ {F} ^ {2}\right) \quad \text {f o r} | t | \leq \left(4 \sqrt {2} \| A \|\right) ^ {- 1}.
$$
C.2 Permutation Bounds for Two-Sample U-Statistics
The main technical result we use for the null bounds is the following. Here we assume the permutation-invariance property of the U statistic kernel that  $h(x_1, x_2; y_1, y_2) = h(x_2, x_1; y_1, y_2) = h(x_1, x_2; y_2, y_1)$ . This can always be ensured by symmetrizing the kernel, and is satisfied by the kernel used by  $\widehat{\mathrm{MMD}^2}$  which we consider in this paper.
Theorem 7. Fix combined sample  $Z$  of size  $n + m$  with  $n \leq m$  and let  $h(x_1, x_2; y_1, y_2)$  be a two-sample U-statistic kernel as described above. Define
$$
U _ {k} (\mathbf {Z}) := \frac {1}{n (n - 1) m (m - 1)} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} h \left(Z _ {i}, Z _ {i ^ {\prime}}; Z _ {n + j}, Z _ {n + j ^ {\prime}}\right).
$$
$$
\bar {U} _ {k} (\mathbf {Z}) = \frac {1}{n (n - 1) m (m - 1)} \sup  _ {\sigma \in \mathfrak {S} _ {n + m}} \sqrt {\sum_ {(i , i ^ {\prime}) \in [ n ] _ {2}} \left(\sum_ {(j , j ^ {\prime}) \in [ m ] _ {2}} h \left(Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}; Z _ {\sigma (n + j)}, Z _ {\sigma (n + j ^ {\prime})}\right)\right) ^ {2}}.
$$
If  $\sigma \in \mathrm{Uniform}(\mathfrak{S}_{n + m})$  is a random permutation of the dataset, then then for all  $|t| < (4\sqrt{2}\bar{U}_k(Z))^{-1}$ ,
$$
\mathbb {E} _ {\sigma} \exp (t U _ {k} (\sigma \boldsymbol {Z})) \leq \exp \left(t ^ {2} \bar {U} _ {k} ^ {2} (\boldsymbol {Z})\right).
$$
Theorem 8 (Normaliser Bound). If  $h(x, x'; y, y') = k(x, x') + k(y, y') - k(x, y') - k(x', y)$  then
$$
\bar {U} _ {k} (\boldsymbol {Z}) ^ {2} \leq \frac {1 6}{n (n - 1)} \widehat {N} _ {k} (\boldsymbol {Z})
$$
where  $n\leq m$  and
$$
\widehat {N} _ {k} (\boldsymbol {Z}) = \frac {1}{n (n - 1)} \sum_ {(i, j) \in [ n + m ] _ {2}} k (Z _ {i}, Z _ {j}) ^ {2}.
$$
Additionally, if  $0 \leq k(x,x') \leq \kappa$  for all  $x,x'$ , then
$$
\bar {U} _ {k} ^ {2} (\mathbf {Z}) \leq \frac {4 \kappa^ {2}}{n (n - 1)}.
$$
Proof of Theorem 7. Firstly, we make the following definitions based on the work of Kim et al. (2022): let  $L = \{l_1,\dots ,l_n\}$  be an  $n$ -tuple drawn uniformly without replacement from  $[m]$ . Then for any fixed  $Z$
$$
\mathbb {E} _ {L} \left[ \tilde {U} _ {k} ^ {L} (\boldsymbol {Z}) \right] = U _ {k} (\boldsymbol {Z}) \tag {7}
$$
$$
\tilde {U} _ {k} ^ {L} (\mathbf {Z}) := \frac {1}{n (n - 1)} \sum_ {(i, j) \in [ n ] _ {2}} h \left(Z _ {i}, Z _ {j}; Z _ {n + l _ {i}}, Z _ {n + l _ {j}}\right).
$$
Note that in the above we have used the invariance  $h(x_{1},x_{2};y_{1},y_{2}) = h(x_{1},x_{2};y_{2},y_{1})$ .
Now additionally define  $\zeta_{i}$  as i.i.d. Rademacher variables, and let  $\sigma \sim \mathrm{Uniform}(\mathfrak{S}_{n + m})$ . For any fixed  $Z$
$$
\tilde {U} _ {k} ^ {L} (\sigma \mathbf {Z}) = ^ {d} \tilde {U} _ {k} ^ {L, \zeta} (\sigma \mathbf {Z}) \tag {8}
$$
$$
\tilde {U} _ {k} ^ {L, \zeta} (\mathbf {Z}) := \frac {1}{n (n - 1)} \sum_ {(i, j) \in [ n ] _ {2}} \zeta_ {i} \zeta_ {j} h \left(Z _ {i}, Z _ {j}; Z _ {n + l _ {i}}, Z _ {n + l _ {j}}\right).
$$
This works because we can first define  $\tilde{Z}_i = Z_i$  or  $\tilde{Z}_i = Z_{n + l_i}$ , each with probability  $\frac{1}{2}$ . The distribution of  $\tilde{U}_k^L (\sigma \tilde{Z}) = {}^d\tilde{U}_k^L (\sigma Z)$ , and then using the symmetry of  $k(x,y)$  in its arguments gives the equivalence Equation (8) (c.f. eq. 28, Kim et al., 2022).
Now for fixed  $Z$ , combining Equations (7) and (8) and Jensen's inequality we gives
$$
\mathbb {E} _ {\sigma} \exp (t U _ {k} (\sigma \boldsymbol {Z})) = \mathbb {E} _ {\sigma} \exp (t \mathbb {E} _ {L} [ \tilde {U} _ {k} ^ {L} (\sigma \boldsymbol {Z}) | \sigma ])
$$
$$
\begin{array}{l} = \mathbb {E} _ {\sigma} \exp (t \mathbb {E} _ {L, \zeta} [ \tilde {U} _ {k} ^ {L, \zeta} (\sigma \pmb {Z}) | \sigma ]) \\ \leq \mathbb {E} _ {\sigma , \zeta} \exp \left(t \mathbb {E} _ {L} \left[ \frac {1}{n (n - 1)} \sum_ {(i, j) \in [ n ] _ {2}} \zeta_ {i} \zeta_ {j} h \left(Z _ {\sigma (i)}, Z _ {\sigma (j)}; Z _ {\sigma (n + l _ {i})}, Z _ {\sigma (n + l _ {j})}\right) \mid \sigma \right]\right) \\ = \mathbb {E} _ {\sigma , \zeta} \exp \left(t \frac {1}{n (n - 1)} \sum_ {(i, j) \in [ n ] _ {2}} \zeta_ {i} \zeta_ {j} \mathbb {E} _ {L} \left[ h \left(Z _ {\sigma (i)}, Z _ {\sigma (j)}; Z _ {\sigma (n + l _ {i})}, Z _ {\sigma (n + l _ {j})}\right) | \sigma \right]\right) \\ =: \mathbb {E} _ {\sigma , \zeta} \exp \left(t \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \zeta_ {i} \zeta_ {j} A _ {i j} ^ {\sigma}\right). \\ \end{array}
$$
In the final step we have defined the matrix  $A^{\sigma}$ , with entries  $A_{ij}^{\sigma} = (n(n - 1))^{-1}\mathbb{E}_L\left[h(Z_{\sigma(i)},Z_{\sigma(j)};Z_{\sigma(n + l_i)},Z_{\sigma(n + l_j)})|\sigma \right]$  for  $i \neq j$  and  $A_{ii} = 0$  for all  $i$ .
We note that  $\zeta$  is independent of  $\sigma$  and satisfies the conditions of Theorem 6, so applying this theorem we obtain that
$$
\begin{array}{l} \mathbb {E} _ {\sigma , \zeta} \exp \left(t \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \zeta_ {i} \zeta_ {j} A _ {i j} ^ {\sigma}\right) \\ \leq \mathbb {E} _ {\sigma} \exp \left(1 6 t ^ {2} \| A ^ {\sigma} \| _ {F} ^ {2}\right) \quad \text {f o r} | t | <   (4 \sqrt {2} \| A ^ {\sigma} \|) ^ {- 1} \\ \leq \mathbb {E} _ {\sigma} \exp \left(1 6 t ^ {2} \| A ^ {\sigma} \| _ {F} ^ {2}\right) \quad \text {f o r} | t | <   (4 \sqrt {2} \| A ^ {\sigma} \| _ {F}) ^ {- 1} \\ \leq \exp \left(1 6 t ^ {2} \sup  _ {\sigma \in \mathfrak {S} _ {n + m}} \| A ^ {\sigma} \| _ {F} ^ {2}\right) \quad \text {f o r} | t | <   (4 \sqrt {2} \sup  _ {\sigma \in \mathfrak {S} _ {n + m}} \| A ^ {\sigma} \| _ {F}) ^ {- 1} \\ \end{array}
$$
where in the second line we have used that  $\| M\| \leq \| M\| _F$  for any matrix  $M$
$$
\begin{array}{l} \| A ^ {\sigma} \| _ {F} ^ {2} = \sum_ {i = 1} ^ {n} \sum_ {i ^ {\prime} = 1} ^ {n} \left(A _ {i i ^ {\prime}} ^ {\sigma}\right) ^ {2} \\ = \frac {1}{n ^ {2} (n - 1) ^ {2}} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} (\mathbb {E} _ {L} \left[ h (Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}; Z _ {\sigma (n + l _ {i})}, Z _ {\sigma (n + l _ {i ^ {\prime}})}) | \sigma) \right] ^ {2} \\ = \frac {1}{n ^ {2} (n - 1) ^ {2}} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} \left(\frac {1}{m (m - 1)} \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} h \left(Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}; Z _ {\sigma (n + j)}, Z _ {\sigma (n + j ^ {\prime})}\right)\right) ^ {2}. \\ \end{array}
$$
We complete the proof by noting that  $\bar{U}_k(\mathbf{Z})^2 = \sup_{\sigma \in \mathfrak{S}_{n + m}}\| A^\sigma \| _F^2$
Proof of Theorem 8. We have
$$
\begin{array}{l} \bar {U} _ {k} ^ {2} (\mathbf {Z}) = \frac {1}{n ^ {2} (n - 1) ^ {2}} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} \left(\frac {1}{m (m - 1)} \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} h \left(Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}; Z _ {\sigma (n + j)}, Z _ {\sigma (n + j ^ {\prime})}\right)\right) ^ {2} \\ \leq \frac {1}{n ^ {2} (n - 1) ^ {2} m (m - 1)} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} h \left(Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}; Z _ {\sigma (n + j)}, Z _ {\sigma (n + j ^ {\prime})}\right) ^ {2} \\ \leq \frac {4}{n ^ {2} (n - 1) ^ {2} m (m - 1)} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} \left(k \left(Z _ {\sigma (i)}, Z _ {\sigma \left(i ^ {\prime}\right)}\right) ^ {2} + k \left(Z _ {\sigma (n + j)}, Z _ {\sigma \left(n + j ^ {\prime}\right)}\right) ^ {2} \right. \\ \left. + k (Z _ {\sigma (i)}, Z _ {\sigma (n + j ^ {\prime})}) ^ {2} + k (Z _ {\sigma (i ^ {\prime})}, Z _ {\sigma (n + j ^ {\prime})}) ^ {2}\right) \\ = \frac {4}{n ^ {2} (n - 1) ^ {2} m (m - 1)} \left(m (m - 1) \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} k \left(Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}\right) ^ {2} \right. \\ \end{array}
$$
$$
\begin{array}{l} + n (n - 1) \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} k (Z _ {\sigma (n + j)}, Z _ {\sigma (n + j ^ {\prime})}) ^ {2} \\ + (m - 1) (n - 1) \sum_ {1 \leq i \leq n} \sum_ {1 \leq j \leq m} k \left(Z _ {\sigma (i)}, Z _ {\sigma (n + j ^ {\prime})}\right) ^ {2} \\ \left. + (m - 1) (n - 1) \sum_ {1 \leq i \leq n} \sum_ {1 \leq j \leq m} k \left(Z _ {\sigma \left(i ^ {\prime}\right)}, Z _ {\sigma (n + j)}\right) ^ {2}\right) \\ \leq \frac {1 6}{n ^ {2} (n - 1) ^ {2}} \sum_ {(i, j) \in [ n + m ] _ {2}} k \left(Z _ {i}, Z _ {j}\right) ^ {2} \\ = \frac {1 6}{n (n - 1)} \widehat {N} _ {k} (\mathbf {Z}) \\ \end{array}
$$
where the first inequality holds by Jensen's inequality, the second by convexity (so that  $(a + b + c + d)^2 \leq 4(a^2 + b^2 + c^2 + d^2)$ ) and the third inequality using  $n \leq m$  to upper bound each of the four scalars inside the parentheses by  $m(m - 1)$ , and we also upper bounded each of sums over subsets of terms by the sum over all possible terms.
For the second part note that since  $h \in [-2\kappa, 2\kappa]$
$$
\begin{array}{l} \bar {U} _ {k} ^ {2} (\boldsymbol {Z}) = \frac {1}{n ^ {2} (n - 1) ^ {2}} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} \left(\frac {1}{m (m - 1)} \sum_ {(j, j ^ {\prime}) \in [ m ] _ {2}} h \left(Z _ {\sigma (i)}, Z _ {\sigma (i ^ {\prime})}; Z _ {\sigma (n + j)}, Z _ {\sigma (n + j ^ {\prime})}\right)\right) ^ {2} \\ \leq \frac {1}{n ^ {2} (n - 1) ^ {2}} \sum_ {(i, i ^ {\prime}) \in [ n ] _ {2}} (2 \kappa) ^ {2} \\ \leq \frac {4 \kappa^ {2}}{n (n - 1)}. \\ \end{array}
$$