SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, the expected kernel trick applies and for  $\mathbf{i} \in \mathbb{N}^d$  with  $\deg(\mathbf{i}) = m$ , and  $k^{\otimes \mathbf{i}}$  and  $\mathcal{H}^{\otimes \mathbf{i}}$  as in (4)
$$
\begin{array}{l} d ^ {\mathbf {i}} (\gamma , \eta) := \| \kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma) - \kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\eta) \| _ {\mathcal {H} ^ {\otimes \mathrm {i}}} ^ {2} \tag {6} \\ = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \left[ \mathbb {E} _ {\gamma_ {\pi} ^ {\mathrm {i}} \otimes \gamma_ {\tau} ^ {\mathrm {i}}} k ^ {\otimes \mathrm {i}} \left(\left(X _ {1}, \dots , X _ {m}\right), \left(Y _ {1}, \dots , Y _ {m}\right)\right) \right. \\ + \mathbb {E} _ {\eta_ {\pi} ^ {\mathbf {i}} \otimes \eta_ {\tau} ^ {\mathbf {i}}} k ^ {\otimes \mathbf {i}} \left(\left(X _ {1}, \dots , X _ {m}\right), \left(Y _ {1}, \dots , Y _ {m}\right)\right) \\ \left. - 2 \mathbb {E} _ {\gamma \overset {\mathbf {i}} {\pi} \otimes \eta \overset {\mathbf {i}} {\tau}} k ^ {\otimes \mathbf {i}} ((X _ {1}, \ldots , X _ {m}), (Y _ {1}, \ldots , Y _ {m})) \right]. \\ \end{array}
$$
We recall Example 3.1 and now give examples of distances between such expressions
Example 3.3 ( $m = 1$ ). Applied with  $m = 1$  and  $d = 1$ , (6) becomes  $\mathrm{MMD}_k^2(\gamma, \eta)$
$$
\left\| \kappa_ {k} ^ {(1)} (\gamma) - \kappa_ {k} ^ {(1)} (\eta) \right\| _ {\mathcal {H} _ {k}} ^ {2} = \mathbb {E} k (X, X ^ {\prime}) + \mathbb {E} k (Y, Y ^ {\prime}) - 2 \mathbb {E} k (X, Y),
$$
where  $X, X'$  denotes independent copies of  $\gamma$  and  $Y, Y'$  denotes independent copies of  $\eta$ .
Example 3.4 ( $m = 2$ ). For  $m = 2$  and  $d = 1$ , (6) reduces to
$$
\begin{array}{l} \left\| \kappa_ {k} ^ {(2)} (\gamma) - \kappa_ {k} ^ {(2)} (\eta) \right\| _ {\mathcal {H} ^ {(1, 1)}} ^ {2} = \mathbb {E} k (X, X ^ {\prime}) k (X ^ {\prime \prime}, X ^ {\prime \prime \prime}) + \mathbb {E} k (Y, Y ^ {\prime}) k (Y ^ {\prime \prime}, Y ^ {\prime \prime \prime}) + \mathbb {E} k (X, X ^ {\prime}) ^ {2} \\ + \mathbb {E} k (Y, Y ^ {\prime}) ^ {2} + 2 \mathbb {E} k (X, Y) k (X ^ {\prime}, Y) + 2 \mathbb {E} k (X, Y) k (X, Y ^ {\prime}) - 2 \mathbb {E} k (X, Y) k (X ^ {\prime}, Y ^ {\prime}) \\ - 2 \mathbb {E} k (X, Y) ^ {2} - 2 \mathbb {E} k (X, X ^ {\prime}) k (X, X ^ {\prime \prime}) - 2 \mathbb {E} k (Y, Y ^ {\prime}) k (Y, Y ^ {\prime \prime}), \\ \end{array}
$$
where  $X, X', X'', X'''$  denotes independent copies of  $\gamma$  and  $Y, Y', Y'', Y'''$  denotes independent copies of  $\eta$ . This expression compares the variances in the RKHS instead of the means. This is an example of the kernel variance embedding defined in the next subsection.
The price for the weak assumption of a point-separating kernel is that without any stronger assumptions one does not get a metric in general, and the all-purpose way to achieve a metric is to take an infinite sum over all  $d^i$ 's. If we only use the degree  $m = 1$  term  $d^i$  reduces to the well-known MMD formula which requires characteristicness to become a metric (see Example 3.3). There are two reasons why working under weaker assumptions is useful: firstly, if the underlying kernel is not characteristic this sum gives a structured way to incorporate finer information that discriminates the two distributions; an extreme case is the linear kernel  $k(x,y) = \langle x,y\rangle$  which is point-separating, and in this case the sum reduces to the differences of classical cumulants. Secondly, under the stronger assumption of characteristicness one already has a metric after truncation at degree  $m = 1$  (the classical MMD). However, in the finite-sample case adding higher degree terms can lead to increased power. Indeed, our experiments (Section 4) show that even just going one degree further (i.e. taking  $m = 2$ ), can lead to more powerful tests.
3.2 A characterization of independence
Here we characterize independence in terms of kernelized cumulants.
Theorem 3 (Characterization of independence with cumulants). Let  $\gamma$  be a probability measure on  $\mathcal{X}_1\times \dots \times \mathcal{X}_d$ , and  $(\mathcal{H}_1,k_1),\ldots ,(\mathcal{H}_d,k_d)$  RKHSs on Polish spaces  $\mathcal{X}_1,\ldots ,\mathcal{X}_d$  such that for every  $1\leq j\leq d$ $k_{j}$  is a bounded, continuous, point-separating kernel. Then
$$
\gamma = \gamma | _ {\mathcal {X} _ {1}} \otimes \dots \otimes \gamma | _ {\mathcal {X} _ {d}} i f a n d o n l y i f \kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathrm {i}} (\gamma) = 0
$$
for every  $\mathbf{i} \in \mathbb{N}_{+}^{d}$ . Moreover, the expected kernel trick applies in the sense that for  $\mathbf{i} \in \mathbb{N}_{+}^{d}$
$$
\| \kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma) \| _ {\mathcal {H} ^ {\otimes \mathrm {i}}} ^ {2} = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}} \otimes \gamma_ {\tau} ^ {\mathbf {i}}} k ^ {\otimes \mathbf {i}} \left(\left(X _ {1}, \dots , X _ {m}\right), \left(Y _ {1}, \dots , Y _ {m}\right)\right), \tag {7}
$$
where  $m\coloneqq \deg (\mathbf{i})$  , and  $k^{\otimes \mathbf{i}}$  and  $\mathcal{H}^{\otimes \mathbf{i}}$  are defined as in (4).
Applied to  $\mathbf{i} = (1,1)$ , the expression (7) reduces to the classical HSIC for two components, see Example 3.5 below. But for general  $\mathbf{i}$  this construction leads to genuine new statistics in RKHSs.
Example 3.5 (Specific case: HSIC, kernel Lancaster interaction, kernel Streitberg interaction). If  $d = 2$  there is only one order 2 index in  $\mathbb{N}_{+}^{d}$ , namely  $\mathbf{i} = (1,1)$ ; in this case (7) reduces to the classical HSIC equation
$$
\| \kappa_ {k _ {1}, k _ {2}} ^ {(1, 1)} (\gamma) \| _ {\mathcal {H} ^ {(1, 1)}} ^ {2} = \mathbb {E} k _ {1} (X, Y) k _ {2} (X, Y) + \mathbb {E} k _ {1} (X, Y) k _ {2} (X ^ {\prime}, Y ^ {\prime}) - 2 \mathbb {E} k _ {1} (X, Y) k _ {2} (X ^ {\prime}, Y),
$$
where  $(X,Y)$  and  $(X^{\prime},Y^{\prime})$  are independent copies of the same random variable following  $\gamma$ . More generally, with  $\mathbf{i} = \mathbf{1}_d$  one gets the kernel Streitberg interaction (Streitberg, 1990; Sejdinovic et al., 2013a; Liu et al., 2023), and specifically the kernel Lancaster interaction (Sejdinovic et al., 2013a) for  $d\in \{2,3\}$ ; the latter reduces to HSIC for two random variables ( $d = 2$ ).
3.3 Finite-sample statistics
To apply Theorem 2 and Theorem 3 in practice, one needs to estimate expressions such as  $\mathbb{E}k^{\otimes \mathbf{i}}((X_1,\ldots ,X_m),(Y_1,\ldots ,Y_m))$ . One could use classical estimators such as  $U$ -statistic (Van der Waart, 2000) which lead to unbiased estimators. However, we follow Gretton et al. (2008) and use a  $V$ -statistic which is biased but conceptually simpler, easier, and efficient to compute. We note that the estimators presented here all have quadratic complexity like MMD and HSIC, see Appendix E.
A two-sample test for non-characteristic feature maps. If  $k$  is characteristic then  $\mathrm{MMD}_k(\gamma, \eta) = 0$  exactly when  $\gamma = \eta$ , but we can still increase testing power by considering the distance between the kernel variance and skewness embeddings, which leads us to use our semi-metrics  $d^{(2)}(\gamma, \eta)$  and  $d^{(3)}(\gamma, \eta)$  as defined in (6). An efficient estimator for  $d^{(3)}$  is given in detail in Appendix E; we provide the full expression for  $d^{(2)}$  here.
Lemma 2 ( $d^{(2)}$  estimation, see (6)). The  $V$ -statistic for  $d^{(2)}(\gamma, \eta) = \|\kappa_k^{(2)}(\gamma) - \kappa_k^{(2)}(\eta)\|_{\mathcal{H}^{(1,1)}}^2$  is
$$
\frac {1}{N ^ {2}} \mathrm {T r} \big [ (\mathbf {K} _ {x} \mathbf {J} _ {N}) ^ {2} \big ] + \frac {1}{M ^ {2}} \mathrm {T r} \big [ (\mathbf {K} _ {y} \mathbf {J} _ {M}) ^ {2} \big ] - \frac {2}{N M} \mathrm {T r} \big [ \mathbf {K} _ {x y} \mathbf {J} _ {M} \mathbf {K} _ {x y} ^ {\top} \mathbf {J} _ {N} \big ],
$$
where  $\mathrm{Tr}$  denotes trace,  $(x_{n})_{n = 1}^{N}\stackrel {\mathrm{i.i.d.}}{\sim}\gamma$ ,  $(y_{m})_{m = 1}^{M}\stackrel {\mathrm{i.i.d.}}{\sim}\eta$ ,  $\mathbf{K}_x = [k(x_i,x_j)]_{i,j = 1}^N\in \mathbb{R}^{N\times N}$ ,  $\mathbf{K}_y = [k(y_i,y_j)]_{i,j = 1}^M\in \mathbb{R}^{M\times M}$ ,  $\mathbf{K}_{x,y} = [k(x_i,y_j)]_{i,j = 1}^{N,M}\in \mathbb{R}^{N\times M}$ ,  $\mathbf{J}_n = \mathbf{I}_n - \frac{1}{n}\mathbf{1}_n\mathbf{1}_n^\top \in \mathbb{R}^{n\times n}$ , with  $\mathbf{1}_n = (1,\dots ,1)\in \mathbb{R}^n$ .
A kernel independence test. By Theorem 3, if  $\gamma = \gamma |_{\mathcal{X}_1} \otimes \gamma |_{\mathcal{X}_2}$ , then  $\kappa^{(2,1)}(\gamma) = 0$  and  $\kappa^{(1,2)}(\gamma) = 0$ . We may compute the magnitude of either  $\kappa^{(2,1)}(\gamma)$  or  $\kappa^{(1,2)}(\gamma)$  - we will refer to these quantities as cross skewness independence criterion (CSIC). Note that these criteria are asymmetric. When  $d = 2$  we have a probability measure  $\gamma$  on  $\mathcal{X}_1 \times \mathcal{X}_2$  and two kernels  $k: \mathcal{X}_1^2 \to \mathbb{R}, \ell: \mathcal{X}_2^2 \to \mathbb{R}$ . Assume that we have samples  $(x_i, y_i)_{i=1}^N$  and use the shorthand notation  $\mathbf{K} = \mathbf{K}_x$ ,  $\mathbf{L} = \mathbf{L}_y$  (similarly to Lemma 2) and  $\mathbf{H} = \mathbf{H}_N = \frac{1}{N}\mathbf{1}_N\mathbf{1}_N^\top \in \mathbb{R}^{N \times N}$ . Denote by  $\circ$  the Hadamard product and  $\langle \cdot \rangle$  the sum over all elements of a matrix. Then one can derive the following CSIC estimator.(Note that matrix multiplication takes precedence over the Hadamard product.)
Lemma 3 (CSIC estimation). The  $V$ -statistic for  $\| \kappa_{k,\ell}^{(1,2)}(\gamma)\|_{\mathcal{H}_k^{\otimes 1}\otimes \mathcal{H}_\ell^{\otimes 2}}^2$  is
$$
\begin{array}{l} \frac {1}{N ^ {2}} \left\langle \mathbf {K} \circ \mathbf {K} \circ \mathbf {L} - 4 \mathbf {K} \circ \mathbf {K H} \circ \mathbf {L} - 2 \mathbf {K} \circ \mathbf {K} \circ \mathbf {L H} + 4 \mathbf {K H} \circ \mathbf {K} \circ \mathbf {L H} \right. \\ + 2 \mathbf {K} \circ \mathbf {L} \left\langle \frac {\mathbf {K}}{N ^ {2}} \right\rangle + 2 \mathbf {K H} \circ \mathbf {H K} \circ \mathbf {L} + 4 \mathbf {K} \circ \mathbf {H K} \circ \mathbf {L H} + \mathbf {K} \circ \mathbf {K} \left\langle \frac {\mathbf {L}}{N ^ {2}} \right\rangle \\ - 8 \mathbf {K} \circ \mathbf {L H} \left\langle \frac {\mathbf {K}}{N ^ {2}} \right\rangle - 4 \mathbf {K} \circ \mathbf {H K} \left\langle \frac {\mathbf {L}}{N ^ {2}} \right\rangle + 4 \left\langle \frac {\mathbf {K}}{N ^ {2}} \right\rangle^ {2} \mathbf {L} \Bigg >. \\ \end{array}
$$
Remark (computational complexity w.r.t. degree  $m$ ). We saw that the computational complexity of the cumulant based measures is quadratic w.r.t. the sample size. Let  $B_{m} = |P(m)|$  be the  $m$ -th
Bell number, in other words the number of elements in  $P(m)$ . The Bell numbers follow a recursion:  $B_{m + 1} = |P(m + 1)| = \sum_{k = 0}^{m}\binom{m}{k}B_{k}$ , with the first elements of the sequence being  $B_0 = B_1 = 1$ ,  $B_{2} = 2$ ,  $B_{3} = 5$ ,  $B_{4} = 15$ ,  $B_{5} = 52$ . By (6)-(7), in the worst case the number of operations to compute  $d^{\mathbf{i}}(\gamma ,\eta)$  or  $\| \kappa_{k_1,\ldots ,k_d}^{\mathbf{i}}(\gamma)\|_{\mathcal{H}^{\otimes i}}^2 (m = \deg (\mathbf{i}))$  is proportional to  $B_m^2$  (it equals to  $3B_m^2$  and to  $B_{m}^{2}$ , respectively). Though asymptotically  $B_{m}$  grows quickly (de Bruijn, 1981; Lovász, 1993), for reasonably small degrees the computation is still manageable. In addition, merging various terms in the estimator can often be carried out, which leads to computational saving. For instance, the estimator of  $d^{(2)}$  (see Lemma 2, Example E.1), CSIC (Lemma 3, Example E.2) and  $d^{(3)}$  (Example E.3) consists of only 3, 11 and  $10 + 2\times 7 = 24$  terms compared to the predicted worst-case setting of  $3B_2^2 = 12$ ,  $B_{3}^{2} = 25$ , and  $3B_{3}^{2} = 75$  terms, respectively. On a practical side, we found that using  $m\in \{2,3\}$  is a good compromise between gain in sample efficiency and ease of implementation.
In this section, we demonstrate the efficiency of the proposed kernel cumulants in two-sample and independence testing.
- Two-sample test: Given  $N - N$  samples from two probability measures  $\gamma$  and  $\eta$  on a space  $\mathcal{X}$ , the goal was to test the null hypothesis  $H_0: \gamma = \eta$  against the alternative  $H_1: \gamma \neq \eta$ . The compared test statistics  $(S)$  were MMD,  $d^{(2)}$ , and  $d^{(3)}$ .
- Independence test: Given  $N$  paired samples from a probability measure  $\gamma$  on a product space  $\mathcal{X}_1 \times \mathcal{X}_2$ , the aim was to test the null hypothesis  $H_0: \gamma = \gamma_1 \otimes \gamma_2$  against the alternative  $H_1: \gamma \neq \gamma_1 \otimes \gamma_2$ . The compared test statistics  $(S)$  were HSIC and CSIC.
In our experiments  $H_{1}$  held, and the estimated power of the tests is reported. Permutation test was applied to approximate the null distribution and its 0.95-quantile (which corresponds to the level choice  $\alpha = 0.05$ ): We first computed our test statistic  $S$  using the given samples  $(S_0 = S)$ , and then permuted the samples 100 times. If  $S_0$  was in a high percentile ( $\geq 95\%$  in our case) of the resulting distribution of  $S$  under the permutations, we rejected the null. We repeated these experiments 100 times to estimate the power of the test. This procedure was in turn repeated 5 times and the 5 samples are plotted as a box plot along with a line plot showing the mean against the number of samples  $(N)$  used. All experiments were performed using the rbf-kernel  $\mathrm{rbf}_{\sigma}(\mathbf{x},\mathbf{y}) = e^{-\frac{\|\mathbf{x} - \mathbf{y}\|_2^2}{2\sigma^2}}$ , where the parameter  $\sigma$  is called the bandwidth. We performed all experiments for every bandwidth of the form  $\sigma = a10^b$  where  $a = 1,2.5,5,7.5$  and  $b = -5, -4, -3, -2, -1,0$  and the optimal value across the bandwidths was chosen for each method and sample size. The experiments were carried out on a laptop with an i7 CPU and 16GBs of RAM.
4.1 Synthetic data
For synthetic data we designed two experiments.
- 2-sample test: We compared a uniform distribution with a mixture of two uniforms.
- Independence test: We considered the joint measure of a uniform and a correlated  $\chi^2$  random variable. We also use this same benchmark to compare the efficiency of classical and kernelized cumulants in Appendix D.
Comparing a uniform with a mixture of uniforms. Even for simpler distributions like mixtures of uniform distributions it can be hard to pick up higher-order features, and  $d^{(2)}$  can outperform MMD even when provided with a moderate number of samples. Here we compared one uniform distribution  $U[-1, 1]$  with an equal mixture of  $U[0.35, 0.778]$  and  $U[-0.35, -0.778]$ . The endpoints in the mixture were chosen to match the first three moments of  $U[-1, 1]$ . The number of samples used ranged from 5 to 50, and the results are summarized in Fig. 1. One can see that with  $d^{(2)}$  the power approaches  $100\%$  much faster than with using MMD.
4All the code replicating our experiments is available at https://github.com/PatricBonnier/ Kernelized-Cumulants.
Figure 2: Test power as a function of the sample size  $(N)$  of independence testing using HSIC (red) and CSIC statistics (blue), with the independence testing between  $Y_{0.5}^{2}$  and  $X$ .
Figure 3: Two-sample testing using MMD (red) and  $d^{(2)}$  (blue) on the Seoul bicycle data set.
Independence between a uniform and a  $\chi^2$ . Let  $X \sim U[0,1]$  and  $Z \sim N(0,1)$  be independent of  $X$ . Denote by  $\Phi$  the c.d.f. of a standard normal distribution and define  $Y_p$  to be a mixture with weight  $p$  at  $\Phi^{-1}(X)$  and weight  $1 - p$  at  $Z$  so that  $Y_0 = Z$  and  $Y_1 = \Phi^{-1}(X)$ . We test for independence for  $p = 0.5$  between  $Y_p^2$ —which will be  $\chi^2$  distributed with 1 degree of freedom and  $X$ . As the statistical dependence of  $Y_p^2$  and  $X$  is more complicated than a simple correlation we expect that higher-order features of the data will help in the independence testing. The number of samples used ranged from 6 to 60, with the results summarized in Fig. 2. One can see that CSIC supersedes HSIC for every sample size, and the difference is more pronounced for smaller ones.
Figure 1: Test power as a function of the sample size  $(N)$  of two-sample test using the MMD (red) and  $d^{(2)}$  statistics (blue), with  $U[-1,1]$  and an equal mixture of  $U[0.35,0.778]$  and  $U[-0.35,-0.778]$  compared.
4.2 Real-world data
We demonstrate the efficiency of the kernelized cumulants on real-world data. We designed two experiments.
- 2-sample test: Here the goal was to test if environmental data in two different seasons, and traffic data at different speeds can be distinguished.
- independence test: The aim was to test if two distributions describing traffic flow and other traffic factors are independent.
To improve performance of both test statistics, all features are standardized to lie between 0 and 1.
Seoul bicycle data. The Seoul bicycle data set (E et al., 2020) consists of environmental data along with the number of bicycle rentals. The environmental data consists of 9 numerical values, 1 categorical value (season), and two binary values. We compare the distribution of the environmental data in the winter and the fall, as we expect these distributions to be different. Concretely, we do 2-sample testing on two measures  $\gamma, \eta$  on  $\mathbb{R}^{11}$ , and assume that  $\gamma \neq \eta$  where  $\gamma$  is the distribution of the environmental data in winter, and  $\eta$  is that of the data in the fall. Permutation testing was performed for  $N$  between 4 and 44, with results summarized in Fig. 3. As it can be observed that  $d^{(2)}$  outperforms MMD in terms of test power. For the Type I error, i.e. the probability of falsely rejecting the null hypothesis (comparing winter data with itself), it hovers between  $5 - 10\%$  for both statistics,
Figure 4: Independence testing using HSIC (red) and CSIC (blue) on the Sao Paulo traffic dataset.
Figure 5: Two-sample testing using MMD (red) and  $d^{(3)}$  (blue) on the Sao Paulo traffic dataset.
which is admittedly slightly higher than the desired  $5\%$  due to the small sample size, but very similar for both statistics; for further details, the reader is referred to Fig. 8 in Appendix D.
Brazilian traffic data. We used the Sao Paulo traffic benchmark (Ferreira, 2016) to perform independence testing. The dataset consists of 16 different integer-valued statistics about the hourly traffic in Sao Paulo such as blockages, fires and other reasons that might hold up traffic. This is combined with a number that describes the slowness of traffic at the given hour; so  $\mathcal{X}_1 = \mathbb{R}^{16}$ ,  $\mathcal{X}_2 = \mathbb{R}$ . One expects a strong dependence between the two sets—or equivalently, for the null hypothesis to be false—and for the statistics are heavily skewed towards 0 as it is naturally sparse. For independence testing we performed permutation testing for  $N$  between 4 and 40. The resulting test powers are summarized in Fig. 4. As it can be seen, HSIC and CSIC performs similarly for very low sample sizes, but for anything else CSIC is the favorable statistic in terms of test power. For two-sample testing, we sampled  $N$  between 5 and 50 and compared the distribution of slow moving traffic with the fast moving traffic. The results are summarized in Fig. 5. It is clear that  $d^{(3)}$  performs similarly to MMD in terms of test power for very small sample sizes, but significantly better for larger ones.
We defined cumulants for random variables in RKHSs by extending the algebraic characterization of cumulants on  $\mathbb{R}^d$ . This construction results in a structured description of the law of random variables that goes beyond the classic kernelized mean and covariance. A kernel trick allows us to compute the resulting kernelized cumulants. We applied our theoretical results to two-sample and independence testing; although kernelized mean and covariance are sufficient for this task, the higher-order kernelized cumulants have the potential to increase the test power and to relax the assumptions on the kernel. Our experiments on real and synthetic data show that kernelized cumulants can indeed lead to significant improvement of the test power. A disadvantage of these higher-order statistics is that their theoretical analysis requires more mathematical machinery although we emphasize that the resulting estimators are simple V-statistics.
Arcones, M. A. and Gine, E. (1992). On the bootstrap of  $U$  and  $V$  statistics. Annals of Statistics, 20:655-674.
Aronszajn, N. (1950). Theory of reproducing kernels. Transactions of the American Mathematical Society, 68:337-404.
Baringhaus, L. and Franz, C. (2004). On a new multivariate two-sample test. Journal of Multivariate Analysis, 88:190-206.
Berlinnet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer.
Tolstikhin, I., Sriperumbudur, B., and Scholkopf, B. (2016). Minimax estimation of maximal mean discrepancy with radial kernels. In Advances in Neural Information Processing Systems (NIPS), pages 1930-1938.
Van der Waart, A. A. (2000). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics.
Willard, S. (1970). General Topology. Addison-Wesley.
Zinger, A., Kakosyan, A., and Klebanov, L. (1992). A characterization of distributions by mean values of statistics and certain probabilistic metrics. Journal of Soviet Mathematics.
Zolotarev, V. (1983). Probability metrics. Theory of Probability and its Applications, 28:278-302.
These appendices provide additional background and elaborate on some of the finer points in the main text. In Appendix A we illustrate that cumulants have typically lower variance estimators compared to moments. Technical background on tensor products and tensor sums of Hilbert spaces, and on tensor algebras is provided in Appendix B. We present our proofs in Appendix C. In Appendix D additional details on our numerical experiments are provided. Our V-statistic based estimators are detailed in Appendix E.
A Moments and cumulants
Already for real-valued random variable  $X$ , moments have well-known drawbacks that make cumulants often preferable as statistics. For a detailed introduction to the use of cumulants in statistics we refer to McCullagh (2018). Here we just mention that
1. the moment generating function  $f(t) = \mathbb{E}[e^{tX}] = \sum_{m} \mu_{m} t^{m} / m!$  describes the law of  $X$  with sequence  $(\mu_{m})$  of moments  $\mu_{m} = \mathbb{E}[X^{m}] \in \mathbb{R}$ . However, since the function  $t \mapsto f(t)$  is the expectation of an exponential, one would often expect that  $f$  is also "exponential in  $t$ ", hence  $g(t) = \log f(t) = \sum_{m} \kappa_{m} \frac{t^{m}}{m!}$  should be simpler to describe as a power series. For example, for a Gaussian  $f(t) = e^{t\mathbb{E}(X) + \frac{t^2}{2} Var(X)}$  and while  $\mu_{m}$  can be in this case explicitly calculated and uneven moments vanish, the  $m$ -moments are fairly complicated compared to the power series expansion of  $g(t) = \kappa_{1} t + \kappa_{2} \frac{t^2}{2}$  which just consists of  $\kappa_{1}$  (mean) and  $\kappa_{2}$  (variance).
2. In the moment sequence  $\mu_{m}$ , lower moments can dominate higher moments. Hence, a natural idea to compensate for these "different scales" is to systematically subtract lower moments from higher moments. As mentioned in the introduction, this is in particular troublesome if finite samples are available. Even in dimension  $d = 1$  the second moment is dominated by the squared mean, that is for a real-valued random variable  $X\sim \gamma$
$$
\mu^ {2} (\gamma) = (\mu^ {1} (\gamma)) ^ {2} + \operatorname {V a r} (X),
$$