SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note the following trick: the monomials on  $\prod_{i=1}^{d} \mathcal{B}_i$  are exactly functions of the form
$$
x \mapsto \langle F ^ {\otimes \mathbf {i}}, x ^ {\mathbf {i}} \rangle
$$
for  $F = (f_{1},\ldots ,f_{d})$ , this will be used in the proofs. We can now restate and prove the our theorem. Note that the cumulants here are defined like in Definition 4 which is a sensible definition even if the feature maps are not associated to kernels.
Theorem 4 (Generalization of Theorem 2 and Theorem 3). Let  $\mathcal{X}_1, \ldots, \mathcal{X}_d$  be Polish spaces and  $\varphi_i : \mathcal{X}_i \to \mathcal{B}_i$  be continuous, bounded and injective feature maps into separable Banach spaces  $\mathcal{B}_i$  for  $i = 1, \ldots, d$ . Let  $\gamma$  and  $\eta$  be probability measures on  $\mathcal{X}_1 \times \dots \times \mathcal{X}_d$ . Then
1.  $\gamma = \eta$  if and only if  $\kappa (\gamma) = \kappa (\eta)$
2.  $\gamma = \bigotimes_{i=1}^{d} \gamma|_{\mathcal{X}_i}$  if and only if the cross cumulants vanish, that is  $\kappa^{\mathbf{i}}(\gamma) = 0$  for all  $\mathbf{i} \in \mathbb{N}_{+}^{d}$ .
- Item 2: We want to show that the cross cumulants vanish if and only if  $\gamma = \bigotimes_{i=1}^{d} \gamma|_{\mathcal{X}_i}$ . By Lemma 5 it is enough to show that
$$
\mathbb {E} _ {\gamma} \Big [ p \big (\varphi_ {1} (X _ {1}), \ldots , \varphi_ {d} (X _ {d}) \big) \Big ] = \mathbb {E} _ {\bigotimes_ {i = 1} ^ {d} \gamma | _ {X _ {i}}} \Big [ p \big (\varphi_ {1} (X _ {1}), \ldots , \varphi_ {d} (X _ {d}) \big) \Big ]
$$
for any monomial function  $p$ . Let us take linear functionals  $F = (f_{1},\ldots ,f_{d})$  and note that
$$
\langle F ^ {\mathbf {i}}, \kappa^ {\mathbf {i}} (\gamma) \rangle = \sum_ {\pi \in P (d)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} \left[ f _ {1} \left(\varphi_ {1} \left(X _ {1}\right)\right) \dots f _ {d} \left(\varphi_ {d} \left(X _ {d}\right)\right) \right]
$$
which is the classical cumulant of the vector-valued random variable
$$
\left(\left(f _ {1} \circ \varphi_ {1}\right) \left(X _ {1}\right), \dots , \left(f _ {d} \circ \varphi_ {d}\right) \left(X _ {d}\right)\right),
$$
where  $(X_{1},\ldots ,X_{d})\sim \gamma$ . Hence by classical results (Speed, 1983), all cross cumulants of  $\bigl ((f_1\circ \varphi_1)(X_1),\ldots ,(f_d\circ \varphi_d)(X_d)\bigr)$  vanish if and only if the cross moments split, that is to say
$$
\mathbb {E} _ {\gamma} \left[ p \big ((f _ {1} \circ \varphi_ {1}) (X _ {1}), \dots , (f _ {d} \circ \varphi_ {d}) (X _ {d}) \big) \right] = \mathbb {E} _ {\bigotimes_ {i = 1} ^ {d} \gamma | x _ {i}} \left[ p \big ((f _ {1} \circ \varphi_ {1}) (X _ {1}), \dots , (f _ {d} \circ \varphi_ {d}) (X _ {d}) \big) \right]
$$
for any monomial  $p$  on  $\mathbb{R}^d$ . Since  $f_1, \ldots, f_d$  were arbitrary this holds for all monomials, which shows the assertion.
- Item 1: By assumption  $\kappa^{\mathbf{i}}(\gamma) = \kappa^{\mathbf{i}}(\eta)$  for every  $\mathbf{i} \in \mathbb{N}^d$ ; this implies that  $\mathbb{E}_{\gamma}p(\varphi_1, \ldots, \varphi_d) = \mathbb{E}_{\eta}p(\varphi_1, \ldots, \varphi_d)$  for any polynomial  $p$ , so we can apply Lemma 5.
Proposition 1 (Theorem 2 and Theorem 3). Let  $\mathcal{X}_1, \ldots, \mathcal{X}_d$  be Polish spaces and  $k_i: \mathcal{X}_i^2 \to \mathbb{R}$  be a collection of bounded, continuous, point-separating kernels. Let  $\gamma$  and  $\eta$  be probability measures on  $\mathcal{X}_1 \times \dots \times \mathcal{X}_d$ . Then
1.  $\gamma = \eta$  if and only if  $\kappa_{k_1,\dots,k_d}(\gamma) = \kappa_{k_1,\dots,k_d}(\eta)$ .
2.  $\gamma = \bigotimes_{i=1}^{d} \gamma|_{\mathcal{X}_i}$  if and only if  $\kappa_{k_1, \ldots, k_d}^{\mathbf{i}}(\gamma) = 0$  for all  $\mathbf{i} \in \mathbb{N}_+^d$ .
Proof. We reduce the proof to the checking of the conditions of Theorem 4. Let  $\varphi_{i}$  denote the canonical feature map of the kernel  $k_{i}$ , and let  $\mathcal{B}_i\coloneqq \mathcal{H}_{k_i}$  be the RKHS associated to  $k_{i}$  ( $i\in \{1,\ldots ,d\}$ ). For all  $i\in \{1,\dots ,d\}$ ,  $\varphi_{i}$  is (i) bounded by the boundedness of  $k_{i}$  since  $\| \varphi_{i}(x)\|_{\mathcal{H}_{k_{i}}}^{2} = k_{i}(x,x)\leq \sup_{x\in \mathcal{X}_{i}}|k_{i}(x,x)| < \infty$ , (ii) continuous by the continuity of  $k_{i}$  (Steinwart and Christmann, 2008, Lemma 4.29), (iii) injective by the point-separating property of  $k_{i}$ . The separability of  $\mathcal{H}_{k_i}$  follows (Steinwart and Christmann, 2008, Lemma 4.33) from the separability of  $\mathcal{X}_i$  and the continuity of  $k_{i}$  ( $i\in \{1,\dots ,d\}$ ). Note: Details on the expected kernel trick part of Theorem 2 and Theorem 3 are provided in Section E.
D Additional experiments and details
Here we give additional details on the experiments that were performed, and discuss some further experiments that did not fit into the main text.
Background on permutation testing. Permutation testing works by bootstrapping the distribution of a test statistic under the null hypothesis. This allows the user to estimate confidence intervals under the null, which is a powerful all-purpose way of doing so when analytic expressions are unavailable. As an example, assume we have two probability measures  $\gamma, \eta$  on  $\mathcal{X}$  with i.i.d. samples  $x_1, \ldots, x_N \sim \gamma, y_1, \ldots, y_N \sim \eta$ . If the null hypothesis is that  $\gamma = \eta$  then we may set
$$
\left(z _ {1}, \dots , z _ {2 N}\right) := \left(x _ {1}, \dots , x _ {N}, y _ {1}, \dots , y _ {N}\right)
$$
so that for any permutation  $\sigma$  on  $2N$  elements, we get two different sets of i.i.d. samples from  $\gamma = \eta$  by using the empirical measures
$$
\tilde {\gamma} _ {\sigma} := \left(z _ {\sigma (1)}, \dots , z _ {\sigma (N)}\right), \quad \tilde {\eta} _ {\sigma} := \left(z _ {\sigma (N + 1)}, \dots , z _ {\sigma (2 N)}\right)
$$
and for any statistic  $S:\mathcal{P}(\mathcal{X})^2\to \mathbb{R}$ , we may estimate  $S(\gamma ,\eta)$  under the null by sampling from  $S(\tilde{\gamma}_{\sigma},\tilde{\eta}_{\sigma})$ . If the null hypothesis were true, we might expect  $S(\gamma ,\eta)$  to lie in a region with high probability of the permutation estimator, and we can use this as a criteria for rejecting the null. Under fairly weak assumptions, this yields a test at the appropriate level (Chung and Romano, 2013).
Comparing a uniform and a mixture. Any uniform random variable over a symmetric interval will have 0 mean and skewness, so a symmetric mixture only needs to match the variance. If  $X$  is a  $50/50$  mixture of  $U[a, b]$  and  $U[-a, -b]$  then
$$
\operatorname {V a r} (X) = \frac {2}{3} \left(b ^ {2} + b a + a ^ {2}\right)
$$
so if  $Y$  is distributed according to  $U[-c, c]$  then we only need to solve
$$
b ^ {2} + b a + a ^ {2} = c ^ {2}
$$
which is straightforward for a given  $a$  and  $c$ .
Computational complexity of estimators. The V-statistic for  $d^{(2)}$  as written in Lemma 2 is bottlenecked by the matrix multiplications. We may note however that for two matrices  $\mathbf{A}$ ,  $\mathbf{B}$  it holds that
$$
\operatorname {T r} (\mathbf {A} ^ {\top} \mathbf {B}) = \langle \mathbf {A} \circ \mathbf {B} \rangle ,
$$
where  $\langle \cdot \rangle$  denotes the sum over elements and  $\circ$  denotes the Hadamard product. We also note that for  $\mathbf{H}_n = \frac{1}{n}\mathbf{1}_n\mathbf{1}_n^\top$  we have  $\left(\mathbf{A}\mathbf{H}_n\right)_{i,j} = \frac{1}{n}\sum_{c = 1}^{n}A_{i,c}$ . Using both of these tricks we may compute both  $d^{(2)}$  and CSIC without any matrix multiplications, which brings the computational complexity down to  $O(N^2)$  for both. For a comparison of actual computation time, see Fig. 6 and Fig. 7, where the average computational times for out methods are compared to the KME and and HSIC for  $N$  between 50 and 2000.
Type I error on the Seoul Bicycle data. The results when comparing the winter data to itself is presented in Fig. 8. As we see the performance is similar for both estimators and lies between 5 and  $10\%$ .
Figure 6: Average computational time in seconds for KME (red) and  $d^{(2)}$  (blue) for sample size  $N$  between 50 and 2000.
Figure 7: Average computational time in seconds for HSIC (red) and CSIC (blue) for sample size  $N$  between 50 and 2000.
Classical vs. kernelized cumulants. Using the same distributions as in the synthetic independence testing experiment, we now compare  $X$  with  $Y_{0.5}^{2}$  to contrast independence testing with classical cumulants with their kernelized counterpart. The results are summarized in Table 1 where they are displayed as the median value  $\pm$  half the difference between the 75th and 25th percentile. We consider every combination of classical vs. kernelized, variance vs. skewness, and two different sample sizes. One can observe that the classical variance based test performs poorly compared to a classical skewness test, the kernelized variance test is almost as powerful as the kernelized skewness test, and in all cases the kernelized tests deliver higher power.
E Kernel trick computations
Here we show how to arrive at the expressions used for the V-statistics used in the experiments.
Figure 8: Type I errors using MMD (red) and  $d^{(2)}$  (blue) on the Seoul bicycle data set.
Table 1: Comparison of classical and kernelized cumulants for independence testing with both variance and skewness.
N=20	Variance	Skewness
Classical	19% ± 3.0%	56% ± 3.5%
Rbf kernel	39% ± 4.5%	59% ± 3.0%
Classical	17% ± 0.5%	68% ± 1.0%
Rbf kernel	65% ± 3.5%	79% ± 1.5%
Given a real analytic function  $f(x, \ldots, x_d) = \sum_{\mathbf{i} \in \mathbb{N}^d} f_{\mathbf{i}} x^{\mathbf{i}}$  in  $m$  variables with nonzero radius of convergence and Hilbert spaces  $\mathcal{H}_1, \ldots, \mathcal{H}_d$  we may (formally) extend  $f$  to a function
$$
f _ {\otimes}: \prod_ {i = 1} ^ {d} \mathcal {H} _ {i} \to \mathrm {T}, \quad f _ {\otimes} (x _ {1}, \ldots , x _ {d}) = \prod_ {\mathbf {i} \in \mathbb {N} ^ {d}} f _ {\mathbf {i}} x ^ {\otimes \mathbf {i}}.
$$
Moreover, if the Hilbert spaces are RKHSs then we have the following result.
Lemma 6 (Nonlinear kernel trick). For any collection of RKHSs  $\mathcal{H}_1, \ldots, \mathcal{H}_d$  with feature maps  $\varphi_i : \mathcal{X}_i \to \mathcal{H}_i$ , assume that  $f$  and  $g$  are real analytic functions with radii of convergence  $r(f)$  and  $r(g)$  such that  $\max_{1 \leq i \leq d} \sup_{x \in \mathcal{X}_i} |\varphi_i(x)| < \min(r(f), r(g))$ . Then
$$
\langle f _ {\otimes} \big (\varphi_ {1} (x _ {1}), \dots , \varphi_ {d} (x _ {d}) \big), g _ {\otimes} \big (\varphi_ {1} (y _ {1}), \dots , \varphi_ {d} (y _ {d}) \big) \rangle_ {\mathrm {T}} = \sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} f _ {\mathbf {i}} g _ {\mathbf {i}} k _ {1} (x _ {1}, y _ {1}) ^ {i _ {1}} \dots k _ {d} (x _ {d}, y _ {d}) ^ {i _ {d}}.
$$
Proof. Since the image of the  $\varphi_{i}$  s lie inside the radius of convergence of  $f_{\otimes}$  and  $g_{\otimes}$  the power series converge absolutely and we can write
$$
\begin{array}{l} \langle f _ {\otimes} \left(\varphi^ {\otimes \mathbf {i}} (x ^ {\mathbf {i}})\right), g _ {\otimes} \left(\varphi^ {\otimes \mathbf {i}} (y ^ {\mathbf {i}})\right) \rangle_ {\mathrm {T}} = \langle \sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} f _ {\mathbf {i}} \varphi^ {\otimes \mathbf {i}} (x ^ {\mathbf {i}}), \sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} g _ {\mathbf {i}} \varphi^ {\otimes \mathbf {i}} (y ^ {\mathbf {i}}) \rangle_ {\mathrm {T}} \\ = \sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} f _ {\mathbf {i}} g _ {\mathbf {i}} \left\langle \varphi^ {\otimes \mathbf {i}} \left(x ^ {\mathbf {i}}\right), \varphi^ {\otimes \mathbf {i}} \left(y ^ {\mathbf {i}}\right) \right\rangle_ {\mathcal {H} ^ {\otimes \mathbf {i}}} = \sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} f _ {\mathbf {i}} g _ {\mathbf {i}} k _ {1} \left(x _ {1}, y _ {1}\right) ^ {i _ {1}} \dots k _ {d} \left(x _ {d}, y _ {d}\right) ^ {i _ {d}}, \\ \end{array}
$$
where  $\mathcal{H} = \mathcal{H}_1\times \dots \times \mathcal{H}_d$
Using Lemma 6, we can choose kernels  $k_{i}:\mathcal{X}_{i}^{2}\to \mathbb{R}$  with associated RKHSs  $\mathcal{H}_i$  and feature maps  $\varphi_{i}$  and some  $\mathbf{i}\in \mathbb{N}^d$  with  $\deg (\mathbf{i}) = m$ . We make the observation that with  $X = (X_{1},\ldots ,X_{d})\sim \gamma$ ,  $Y = (Y_{1},\dots ,Y_{d})\sim \eta$  and  $k^{\otimes \mathbf{i}}$  and  $\mathcal{H}^{\otimes \mathbf{i}}$  as in (4), one has
$$
\begin{array}{l} \langle \kappa^ {\mathbf {i}} (\gamma), \kappa^ {\mathbf {i}} (\eta) \rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} = \langle \sum_ {\pi \in P (m)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} \varphi^ {\otimes \mathbf {i}} (X ^ {\mathbf {i}}), \sum_ {\tau \in P (m)} c _ {\tau} \mathbb {E} _ {\eta_ {\tau} ^ {\mathbf {i}}} \varphi^ {\otimes \mathbf {i}} (Y ^ {\mathbf {i}}) \rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} \\ = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \langle \mathbb {E} _ {\gamma_ {\pi} ^ {\mathrm {i}}} \varphi^ {\otimes \mathrm {i}} (X ^ {\mathrm {i}}), \mathbb {E} _ {\eta_ {\tau} ^ {\mathrm {i}}} \varphi^ {\otimes \mathrm {i}} (Y ^ {\mathrm {i}}) \rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} \\ = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathrm {i}} \otimes \eta_ {\tau} ^ {\mathrm {i}}} \left\langle \varphi^ {\otimes \mathrm {i}} \left(X ^ {\mathrm {i}}\right), \varphi^ {\otimes \mathrm {i}} \left(Y ^ {\mathrm {i}}\right) \right\rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} \\ = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathrm {i}} \otimes \eta_ {\tau} ^ {\mathrm {i}}} k ^ {\otimes \mathrm {i}} ((X _ {1}, \dots , X _ {m}), (Y _ {1}, \dots , Y _ {m})), \\ \end{array}
$$
$$
\begin{array}{l} \left\| \kappa^ {\mathbf {i}} (\gamma) \right\| _ {\mathcal {H} ^ {\otimes \mathrm {i}}} ^ {2} = \left\langle \kappa^ {\mathbf {i}} (\gamma), \kappa^ {\mathbf {i}} (\gamma) \right\rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} \\ \left\| \kappa^ {\mathbf {i}} (\gamma) - \kappa^ {\mathbf {i}} (\eta) \right\| _ {\mathcal {H} ^ {\otimes \mathrm {i}}} ^ {2} = \left\langle \kappa^ {\mathbf {i}} (\gamma), \kappa^ {\mathbf {i}} (\gamma) \right\rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} + \left\langle \kappa^ {\mathbf {i}} (\eta), \kappa^ {\mathbf {i}} (\eta) \right\rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} - 2 \left\langle \kappa^ {\mathbf {i}} (\gamma), \kappa^ {\mathbf {i}} (\eta) \right\rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} \\ \end{array}
$$
one gets the expected kernel trick statements of Theorem 2 and Theorem 3.
We are now interested in explicitly computing the expression  $\| \kappa_{k,\ell}^{(1,2)}(\gamma)\|_{\mathcal{H}_k^{\otimes 1}\otimes \mathcal{H}_\ell^{\otimes 2}}^2$  and  $\| \kappa_k^{(2)}(\gamma) - \kappa_k^{(2)}(\eta)\|_{\mathcal{H}^{(1,1)}}^2$  and  $\| \kappa_k^{(3)}(\gamma) - \kappa_k^{(3)}(\eta)\|_{\mathcal{H}_k^{\otimes 3}}^2$ , and their corresponding V-statistics. Recall that for
a (w.l.o.g.) symmetric, measurable function  $h(z_{1},\ldots ,z_{m})$ , the V-statistic of  $h$  with  $N$  samples  $Z_{1},\ldots ,Z_{N}$  is defined as
$$
\mathrm {V} (h; Z _ {1}, \dots , Z _ {N}) := N ^ {- m} \sum_ {i _ {1}, \dots , i _ {m} = 1} ^ {N} h \left(Z _ {i _ {1}}, \dots , Z _ {i _ {m}}\right).
$$
Under fairly general conditions, the V-statistic converges in distribution to  $\mathbb{E}[h(Z_1,\ldots,Z_m)]$  and a well-developed theory describes this convergence (Van der Waart, 2000; Serfling, 1980; Arcones and Giné, 1992).
Example E.1 (Estimating  $\| \kappa_k^{(2)}(\gamma) - \kappa_k^{(2)}(\eta)\|_{\mathcal{H}^{(1,1)}}^2$ ). Let  $X,X^{\prime},X^{\prime \prime},X^{\prime \prime \prime}$  denote independent copies of  $\gamma$  and  $Y,Y^{\prime},Y^{\prime \prime},Y^{\prime \prime \prime}$  denote independent copies of  $\eta$ . The full expression for  $\| \kappa_k^{(2)}(\gamma) - \kappa_k^{(2)}(\eta)\|_{\mathcal{H}^{(1,1)}}^2$  is
$$
\begin{array}{l} \left\| \kappa_ {k} ^ {(2)} (\gamma) - \kappa_ {k} ^ {(2)} (\eta) \right\| _ {\mathcal {H} ^ {(1, 1)}} ^ {2} = \mathbb {E} k (X, X ^ {\prime}) k \left(X ^ {\prime \prime}, X ^ {\prime \prime \prime}\right) + \mathbb {E} k \left(Y, Y ^ {\prime}\right) k \left(Y ^ {\prime \prime}, Y ^ {\prime \prime \prime}\right) \tag {12} \\ + \mathbb {E} k (X, X ^ {\prime}) ^ {2} + \mathbb {E} k (Y, Y ^ {\prime}) ^ {2} \\ + 2 \mathbb {E} k (X, Y) k \left(X ^ {\prime}, Y\right) + 2 \mathbb {E} k (X, Y) k \left(X, Y ^ {\prime}\right) \\ - 2 \mathbb {E} k (X, Y) k \left(X ^ {\prime}, Y ^ {\prime}\right) - 2 \mathbb {E} k (X, Y) ^ {2} \\ - 2 \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime \prime}) - 2 \mathbb {E} k (Y, Y ^ {\prime}) k (Y, Y ^ {\prime \prime}). \\ \end{array}
$$
Given samples  $(x_{i})_{i = 1}^{N}$ ,  $(y_{i})_{i = 1}^{M}$  from  $\gamma$  and  $\eta$  respectively the corresponding  $V$  statistic is
$$
\begin{array}{l} \frac {1}{N ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {\kappa}, x _ {l}\right) + \frac {1}{M ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {M} k \left(y _ {i}, y _ {j}\right) k \left(y _ {\kappa}, y _ {l}\right) \tag {13} \\ + \frac {1}{N ^ {2}} \sum_ {i, j = 1} ^ {N} k (x _ {i}, x _ {j}) ^ {2} + \frac {1}{M ^ {2}} \sum_ {i, j = 1} ^ {M} k (y _ {i}, y _ {j}) ^ {2} \\ + \frac {2}{N ^ {2} M} \sum_ {i, \kappa = 1} ^ {N} \sum_ {j = 1} ^ {M} k (x _ {i}, y _ {j}) k (x _ {\kappa}, y _ {j}) + \frac {2}{N M ^ {2}} \sum_ {i = 1} ^ {N} \sum_ {j, \kappa = 1} ^ {M} k (x _ {i}, y _ {j}) k (x _ {i}, y _ {\kappa}) \\ - \frac {2}{N ^ {2} M ^ {2}} \sum_ {i, l = 1} ^ {N} \sum_ {j, \kappa = 1} ^ {M} k (x _ {i}, y _ {j}) k (x _ {\kappa}, y _ {l}) - \frac {2}{N M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} k (x _ {i}, y _ {j}) ^ {2} \\ - \frac {2}{N ^ {3}} \sum_ {i, j, \kappa = 1} ^ {N} k (x _ {i}, x _ {j}) k (x _ {i}, x _ {\kappa}) - \frac {2}{M ^ {3}} \sum_ {i, j, \kappa = 1} ^ {M} k (y _ {i}, y _ {j}) k (y _ {i}, y _ {\kappa}). \\ \end{array}
$$
Let us define the Gram matrices  $\mathbf{K}_x = [k(x_i, x_j)]_{i,j=1}^N \in \mathbb{R}^{N \times N}$ ,  $\mathbf{K}_y = [k(y_i, y_j)]_{i,j=1}^M \in \mathbb{R}^{M \times M}$ ,  $\mathbf{K}_{x,y} = [k(x_i, y_j)]_{i,j=1}^{N,M}$  and let  $\mathbf{H}_N = \frac{1}{N}\mathbf{1}_N\mathbf{1}_N^\top \in \mathbb{R}^{N \times N}$ ,  $\mathbf{H}_M = \frac{1}{M}\mathbf{1}_M\mathbf{1}_M^\top \in \mathbb{R}^{M \times M}$  be the centering, then (13) can be rewritten as
$$
\begin{array}{l} \frac {1}{N ^ {2}} \mathrm {T r} (\mathbf {H} _ {N} \mathbf {K} _ {x} \mathbf {H} _ {N} \mathbf {K} _ {x}) + \frac {1}{M ^ {2}} \mathrm {T r} (\mathbf {H} _ {M} \mathbf {K} _ {y} \mathbf {H} _ {M} \mathbf {K} _ {y}) + \frac {1}{N ^ {2}} \mathrm {T r} (\mathbf {K} _ {x} ^ {2}) + \frac {1}{M ^ {2}} \mathrm {T r} (\mathbf {K} _ {y} ^ {2}) \\ + \frac {2}{N M} \mathrm {T r} (\mathbf {K} _ {x y} \mathbf {H} _ {N} \mathbf {K} _ {x y}) + \frac {2}{N M} \mathrm {T r} (\mathbf {K} _ {x y} \mathbf {H} _ {M} \mathbf {K} _ {x y} ^ {\top}) - \frac {2}{N M} \mathrm {T r} (\mathbf {H} _ {M} \mathbf {K} _ {x y} ^ {\top} \mathbf {H} _ {N} \mathbf {K} _ {x y}) - \frac {2}{N M} \mathrm {T r} (\mathbf {K} _ {x y} ^ {2}) \\ - \frac {2}{N ^ {2}} \operatorname {T r} \left(\mathbf {K} _ {x} \mathbf {H} _ {N} \mathbf {K} _ {x}\right) - \frac {2}{M ^ {2}} \operatorname {T r} \left(\mathbf {K} _ {y} \mathbf {H} _ {M} \mathbf {K} _ {y}\right) \\ \end{array}
$$
$$
\frac {1}{N ^ {2}} \mathrm {T r} \big [ (\mathbf {K} _ {x} (\mathbf {I} - \mathbf {H} _ {N})) ^ {2} \big ] + \frac {1}{M ^ {2}} \mathrm {T r} \big [ (\mathbf {K} _ {y} (\mathbf {I} - \mathbf {H} _ {M})) ^ {2} \big ] - \frac {2}{N M} \mathrm {T r} \big [ \mathbf {K} _ {x y} (\mathbf {I} - \mathbf {H} _ {M}) \mathbf {K} _ {x y} ^ {\top} (\mathbf {I} - \mathbf {H} _ {N}) \big ].
$$
This estimator can be computed in quadratic time.
Example E.2 (Estimating  $\| \kappa_{k,\ell}^{(1,2)}(\gamma)\|_{\mathcal{H}_k^{\otimes 1}\otimes \mathcal{H}_{\ell}^{\otimes 2}}^2$ ). Let  $k$  denote the kernel on  $\mathcal{X}_1$  and  $\ell$  denote the kernel on  $\mathcal{X}_2$ . Let  $(X,Y),(X',Y'),(X'',Y'')$ ,  $(X^{(3)},Y^{(3)}),(X^{(4)},Y^{(4)})$ ,  $(X^{(5)},Y^{(5)})$  denote in-
dependent copies of  $\gamma \in \mathcal{P}(\mathcal{X}_1\times \mathcal{X}_2)$ . The full expression for  $\| \kappa_{k,\ell}^{(1,2)}(\gamma)\|_{\mathcal{H}_k^{\otimes 1}\otimes \mathcal{H}_\ell^{\otimes 2}}^2$  is
$$
\begin{array}{l} \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime}) \ell (Y, Y ^ {\prime}) - 4 \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime \prime}) \ell (Y, Y ^ {\prime}) \\ - 2 \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime}) \ell (Y, Y ^ {\prime \prime}) + 4 \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime \prime}) \ell (Y, Y ^ {(3)}) \\ + 2 \mathbb {E} k (X, X ^ {\prime}) k \left(X ^ {\prime \prime}, X ^ {(3)}\right) \ell \left(Y, Y ^ {\prime}\right) + 2 \mathbb {E} k (X, X ^ {\prime}) k \left(X ^ {\prime \prime}, X ^ {(3)}\right) \ell \left(Y, Y ^ {(3)}\right) \\ + 4 \mathbb {E} k (X, X ^ {\prime}) k \left(X ^ {\prime \prime}, X ^ {\prime}\right) \ell \left(Y, Y ^ {(3)}\right) + \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime}) \ell \left(Y ^ {\prime \prime}, Y ^ {(3)}\right) \\ - 8 \mathbb {E} k (X, X ^ {\prime}) k (X ^ {\prime \prime}, X ^ {(3)}) \ell (Y ^ {(4)}, Y ^ {\prime}) - 4 \mathbb {E} k (X, X ^ {\prime}) k (X ^ {\prime \prime}, X ^ {\prime}) \ell (Y ^ {(4)}, Y ^ {(3)}) \\ + 4 \mathbb {E} k (X, X ^ {\prime}) k \left(X ^ {\prime \prime}, X ^ {(3)}\right) \ell \left(Y ^ {(4)}, Y ^ {(5)}\right). \\ \end{array}
$$
Given samples  $(x_{i},y_{i})_{i = 1}^{N}$  from  $\gamma$  the corresponding V-statistic for this expression is
$$
\begin{array}{l} \frac {1}{N ^ {2}} \sum_ {i, j = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {i}, x _ {j}\right) \ell \left(y _ {i}, y _ {j}\right) - \frac {4}{N ^ {3}} \sum_ {i, j, \kappa = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {i}, x _ {\kappa}\right) \ell \left(y _ {i}, y _ {j}\right) \\ - \frac {2}{N ^ {3}} \sum_ {i, j, \kappa = 1} ^ {N} k (x _ {i}, x _ {j}) k (x _ {i}, x _ {j}) \ell (y _ {i}, y _ {\kappa}) + \frac {4}{N ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {N} k (x _ {i}, x _ {j}) k (x _ {i}, x _ {\kappa}) \ell (y _ {i}, y _ {l}) \\ + \frac {2}{N ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {N} k (x _ {i}, x _ {j}) k (x _ {\kappa}, x _ {l}) \ell (y _ {i}, y _ {j}) + \frac {2}{N ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {N} k (x _ {i}, x _ {j}) k (x _ {\kappa}, x _ {l}) \ell (y _ {i}, y _ {l}) \\ + \frac {4}{N ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {\kappa}, x _ {j}\right) \ell \left(y _ {i}, y _ {l}\right) + \frac {1}{N ^ {4}} \sum_ {i, j, \kappa , l = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {i}, x _ {j}\right) \ell \left(y _ {\kappa}, y _ {l}\right) \\ - \frac {8}{N ^ {5}} \sum_ {i, j, \kappa , l, m = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {\kappa}, x _ {l}\right) \ell \left(y _ {m}, y _ {j}\right) - \frac {4}{N ^ {5}} \sum_ {i, j, \kappa , l, m = 1} ^ {N} k \left(x _ {i}, x _ {j}\right) k \left(x _ {\kappa}, x _ {j}\right) \ell \left(y _ {m}, y _ {l}\right) \\ + \frac {4}{N ^ {6}} \sum_ {i, j, \kappa , l, m, n = 1} ^ {N} k (x _ {i}, x _ {j}) k (x _ {\kappa}, x _ {l}) \ell (y _ {m}, y _ {n}). \\ \end{array}
$$
Using the shorthand notation  $\mathbf{K} = \mathbf{K}_x, \mathbf{L} = \mathbf{L}_y$  and  $\mathbf{H} = \mathbf{H}_N$  and denoting by  $\circ$  the Hadamard product  $[\mathbf{A} \circ \mathbf{B}]_{i,j} = A_{i,j} B_{i,j}$  and  $\langle \cdot \rangle$  the sum over all elements of a matrix  $\langle \mathbf{A} \rangle = \sum_{i,j=1}^{N} A_{i,j}$ , the  $V$ -statistic above can be written in the simpler form