SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Patric Bonnier $^{1*}$  Harald Oberhauser $^{1}$  Zoltán Szabó $^{2}$
$^{1}$ Mathematical Institute, University of Oxford  $^{2}$ Department of Statistics, London School of Economics
bonnier, oberhauser@maths.ox.ac.uk
z.szabo@1se.ac.uk
In  $\mathbb{R}^d$ , it is well-known that cumulants provide an alternative to moments that can achieve the same goals with numerous benefits such as lower variance estimators. In this paper we extend cumulants to reproducing kernel Hilbert spaces (RKHS) using tools from tensor algebras and show that they are computationally tractable by a kernel trick. These kernelized cumulants provide a new set of all-purpose statistics; the classical maximum mean discrepancy and Hilbert-Schmidt independence criterion arise as the degree one objects in our general construction. We argue both theoretically and empirically (on synthetic, environmental, and traffic data analysis) that going beyond degree one has several advantages and can be achieved with the same computational complexity and minimal overhead in our experiments.
Keywords: kernel, cumulant, mean embedding, Hilbert-Schmidt independence criterion, kernel Lancaster interaction, kernel Streitberg interaction, maximum mean discrepancy, maximum variance discrepancy
The moments of a random variable are arguably the most popular all-purpose statistic. However, cumulants are often more favorable statistics than moments. For example, if  $\mu_{m} \coloneqq \mathbb{E}[X^{m}]$  denotes the moments of a real-valued random variable  $X$ , then  $\mu_{2} = \mu_{1}^{2} + \operatorname{Var}(X)$  and hence the variance that directly measures the fluctuation around the mean is a much better statistic for scale than the second moment  $\mu_{2}$ , see Appendix A. Cumulants provide a systematic way to only record the parts of the moment sequence that are not already captured by lower-order moments. While the moment and cumulant sequences  $(\mu_{m})_{m}$  and  $(\kappa_{m})_{m}$  carry the same information, cumulants have several desirable properties that generalize to  $\mathbb{R}^d$ -valued random variables (McCullagh, 2018). Among these properties of cumulants, the ones that are important for our paper is that they can characterize distributions and statistical (in)dependence.
Kernel embeddings. Mean and covariance arise naturally in the context of kernel-enriched domains. Kernel techniques (Scholkopf and Smola, 2002; Steinwart and Christmann, 2008; Saitoh and Sawano, 2016) provide a principled and powerful approach for lifting data points to a so-called reproducing kernel Hilbert space (RKHS; Aronszajn 1950). Considering the mean of this feature – referred to as kernel mean embedding (KME; Berlinet and Thomas-Agnan 2004; Smola et al. 2007) – also enables one to represent probability measures, and to induce a semi-metric referred to as maximum mean discrepancy (MMD; Smola et al. 2007; Gretton et al. 2012) and an independence measure called the
Hilbert-Schmidt independence criterion (HSIC<sup>1</sup>; Gretton et al. 2005). It is known that MMD is a metric when the underlying kernel is characteristic (Fukumizu et al., 2008; Sriperumbudur et al., 2010). HSIC captures independence for  $d = 2$  components with characteristic kernels (Lyons, 2013, Theorem 3.11) and for  $d > 2$  components (Quadrianto et al., 2009; Sejdinovic et al., 2013a; Pfister et al., 2018) with universal ones (Szabó and Sriperumbudur, 2018). MMD belongs to the family of integral probability metrics (IPM; Zolotarev 1983; Müller 1997) when in the IPM the underlying function class is chosen to be the unit ball of the RKHS. MMD and HSIC (with  $d = 2$ ) are known to be equivalent (Sejdinovic et al., 2013b) to the notions of energy distance (Baringhaus and Franz, 2004; Székely and Rizzo, 2004, 2005)—also called N-distance (Zinger et al., 1992; Klebanov, 2005)—and distance covariance (Székely et al., 2007; Székely and Rizzo, 2009; Lyons, 2013) of the statistics literature. Both MMD and HSIC can be expressed in terms of expectations of kernel values which can be leveraged to design efficient estimators for them; we will refer to this trick as the expected kernel trick. A recent survey on mean embedding and their applications is given by Muandet et al. (2017). The closest previous work to ours is by Makigusa (2020) who considers the variance in the RKHS – which in our setting can be identified as the first kernelized cumulant after the kernel mean embedding – for two-sample testing like we do in parts of this paper. Unfortunately, Makigusa (2020) does not provide conditions on the validity of the resulting maximum variance discrepancy, and it is not formulated in the context of cumulant embeddings.
Contribution. The main contribution of our paper is to introduce cumulants of random variables in RKHSs and to show that under mild conditions the proposed kernelized cumulants characterize distributions (Theorem 2) and independence (Theorem 3). Thanks to the RKHS formulation, kernelized cumulants have computable estimators (Lemma 2 and Lemma 3) and they show strong performance in two-sample and independence testing on various benchmarks (Section 4). Although cumulants are a classic tool in multi-variate statistics, they have not received attention in the kernel literature. The primary technical challenge to circumvent in the derivation of fundamental properties of cumulants is the rich combinatorial structure which already arises in  $\mathbb{R}^d$  from their definition via moment-generating function which is closely linked to the partition lattice (Speed, 1983, 1984). In an RKHS, even the definition of cumulants is non-straightforward. The key insight for our extension is that the combinatorial expressions for cumulants in  $\mathbb{R}^d$  can be generalized by using tools from tensor algebras. This in turn allows us to derive the main properties of the RKHS cumulants that underpin their statistical properties.
Broader impact & limitations. We do not see any direct negative societal impact arising from the proposed new set of all-purpose kernel-based divergence and dependence measure. Choosing the underlying kernels in an optimal fashion—even for MMD in two-sample testing (Hagrass et al., 2022) or goodness-of-fit testing (Hagrass et al., 2023)—and showing optimal rates—even for MMD with radial kernels on  $\mathbb{R}^d$  (Tolstikhin et al., 2016)—are quite challenging problems requiring dedicated analysis, and are not addressed here.
Outline. The paper is structured as follows: In Section 2 we formulate the notion of cumulants of random variables in Hilbert spaces. In Section 3 we prove a kernelized version of the classical result of  $\mathbb{R}^d$ -valued random variables on the characterization of distributions and independence using cumulants. We show that one can leverage the expected kernel trick to derive efficient estimators for our novel statistics, with MMD and HSIC arising specifically as the "degree 1" objects. In Section 4 we demonstrate numerically that going beyond degree 1 is advantageous. We provide a technical background (on cumulants, tensor products and tensor algebras), proofs, further details on numerical experiments, and our V-statistic based estimators in the Appendices.
2 Moments and cumulants
We briefly revisit classical cumulants and define cumulants of random variables in Hilbert spaces.
HSIC is MMD with the tensor product kernel evaluated on the joint distribution and the product of the marginals, or equivalently HSIC equals to the Hilbert-Schmidt norm of the cross-covariance operator.
Moments. Let  $\gamma$  be a probability measure on  $\mathbb{R}^d$  and  $(X_1, \ldots, X_d) \sim \gamma$ . The moments  $\mu(\gamma) = (\mu^{\mathbf{i}}(\gamma))_{\mathbf{i} \in \mathbb{N}^d}$  of  $\gamma$  are defined as
$$
\mu^ {\mathbf {i}} (\gamma) := \mathbb {E} \left[ X _ {1} ^ {i _ {1}} \dots X _ {d} ^ {i _ {d}} \right] \in \mathbb {R}, \tag {1}
$$
where  $\mathbf{i} = (i_1, \dots, i_d) \in \mathbb{N}^d$  denotes a  $d$ -tuple of non-negative integers  $(i_1, \dots, i_d \geq 0)$ . The degree of an element  $\mathbf{i} \in \mathbb{N}^d$  is defined as  $\deg(\mathbf{i}) := i_1 + \dots + i_d$ . For  $m \in \mathbb{N}$ , let  $\mu^m(\gamma) := (\mu^{\mathbf{i}}(\gamma))_{\deg(\mathbf{i})=m}$  which we refer to as the  $m$ -th moments of  $\gamma$  with the convention that  $\mu^0(\gamma) = 1$ .
Cumulants. Cumulants  $\kappa (\gamma) = (\kappa^{\mathbf{i}}(\gamma))_{\mathbf{i}\in \mathbb{N}^d}$  can be defined by the moment generating function as
$$
\sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} \kappa^ {\mathbf {i}} (\gamma) \frac {\theta^ {\mathbf {i}}}{\mathbf {i} !} = \log \sum_ {\mathbf {i} \in \mathbb {N} ^ {d}} \mu^ {\mathbf {i}} (\gamma) \frac {\theta^ {\mathbf {i}}}{\mathbf {i} !}, \quad \theta = \left(\theta_ {1}, \dots , \theta_ {d}\right) \in \mathbb {R} ^ {d}, \tag {2}
$$
where we denote  $\mathbf{i}! = i_1!\cdots i_d!$  and  $\theta^{\mathbf{i}} = \theta_{1}^{i_{1}}\cdots\theta_{d}^{i_{d}}$ ; an equivalent definition of cumulants is via a combinatorial expression of partitions (elaborated in Appendix C.1). Cumulants have several attractive properties, the following forms our main motivation.
Theorem 1 (Characterization of distributions with cumulants on  $\mathbb{R}^d$ , from Proposition 1 in Jammalamada et al. 2006). Let  $\gamma$  be a probability measure on a bounded subset of  $\mathbb{R}^d$  with cumulants  $\kappa(\gamma)$  and let  $(X_1, \ldots, X_d) \sim \gamma$ . Then
1.  $\gamma \mapsto \kappa (\gamma)$  is injective.
2.  $X_{1},\ldots ,X_{d}$  are jointly independent if and only if  $\kappa^{\mathbf{i}}(\gamma) = 0$  for all  $d$ -tuples of positive integers  $\mathbf{i}\in \mathbb{N}_{+}^{d}$ .
2.1 Moments in Hilbert spaces
Instead of directly considering the law of a tuple of random variables  $(X_{1},\ldots ,X_{d})$  in a product space  $\mathcal{X}_1\times \dots \times \mathcal{X}_d$ , it can be advantageous to use feature maps  $\Phi_i:\mathcal{X}_i\to \mathcal{H}_i$  and instead study the distribution of the  $\mathcal{H}_1\times \dots \times \mathcal{H}_d$ -valued random variable  $(\Phi_1(X_1),\ldots ,\Phi_d(X_d))$ . Motivated by this lifting, we study here moments of Hilbert-space valued random variables and assume in this subsection (with a slight abuse of notations) that one has already applied the lifting and  $X_{i}\in \mathcal{H}_{i}$  where  $i = 1,\ldots ,d$ . In Section 3 we specialize the construction to RKHSs, and use these moments (Def. 1) to define kernelized cumulants.
Moments. In the finite-dimensional case (1) we defined the moment sequence by taking expectations of products of the coordinates of the underlying random variable. For the infinite-dimensional case, it is convenient to develop a coordinate-free definition which can be accomplished by using tensors. To do so we make use of the following results about Hilbert spaces: for real Hilbert spaces  $\mathcal{H}_1$  and  $\mathcal{H}_2$  the tensor product  $\mathcal{H}_1 \otimes \mathcal{H}_2$  is the Hilbert space given by completion of the tensor product of  $\mathcal{H}_1$  and  $\mathcal{H}_2$  as vector space; we also write  $\mathcal{H}_1^{\otimes m} := \underbrace{\mathcal{H}_1 \otimes \cdots \otimes \mathcal{H}_1}_{m \text{ times}}$ . Similarly, the direct sum
$\mathcal{H}_1 \oplus \mathcal{H}_2$  is a Hilbert space. It is natural to consider  $\mathbb{E}\left[X_1^{\otimes m}\right] \in \mathcal{H}_1^{\otimes m}$  as the  $m$ -th moment of a  $\mathcal{H}_1$ -valued random variable  $X_1$  where the integral in the expectation is meant in Bochner sense. Consequently the natural state space for all moments of a  $\mathcal{H}_1$ -valued random variable is the tensor algebra  $\mathrm{T}_1 \coloneqq \prod_{m \geq 0} \mathcal{H}_1^{\otimes m}$  where by convention  $\mathcal{H}_1^{\otimes 0} \coloneqq \mathbb{R}$ . See Appendix B for more details on tensor products of Hilbert spaces and tensor algebras.
Example 2.1 ( $\mathcal{H}_1 = \mathbb{R}^d$ ,  $m = 2$ ). If  $X_1 = (X_1^1, \ldots, X_1^d)$  is  $\mathcal{H}_1 = \mathbb{R}^d$ -valued then  $\mathbb{E}\left[X_1^{\otimes 2}\right] \in (\mathbb{R}^d)^{\otimes 2}$  can be identified with a  $(d \times d)$ -sized matrix whose  $(i,j)$ -th entry is  $\mathbb{E}\left[X_1^i X_1^j\right]$ .
Since we are interested in the general case of a  $\mathcal{H}_1 \times \dots \times \mathcal{H}_d$ -valued random variable  $X = (X_1, \ldots, X_d)$  we arrive at the definition below.
Definition 1 (Moments in Hilbert spaces). Let  $\gamma$  be a probability measure on  $\mathcal{H} \coloneqq \mathcal{H}_1 \times \dots \times \mathcal{H}_d$  and let  $(X_1, \ldots, X_d) \sim \gamma$ . We define
$$
\mu^ {\mathbf {i}} (\gamma) := \mathbb {E} \left[ X _ {1} ^ {\otimes i _ {1}} \otimes \dots \otimes X _ {d} ^ {\otimes i _ {d}} \right] \in \mathcal {H} ^ {\otimes \mathbf {i}}, \quad \mathcal {H} ^ {\otimes \mathbf {i}} := \mathcal {H} _ {1} ^ {\otimes i _ {1}} \otimes \dots \otimes \mathcal {H} _ {d} ^ {\otimes i _ {d}} \tag {3}
$$
for every  $\mathbf{i} \in \mathbb{N}^d$  whenever the above expectation exists. The moment sequence is defined as the element
$$
\mu (\gamma) = (\mu^ {\mathbf {i}} (\gamma)) _ {\mathbf {i} \in \mathbb {N} ^ {d}} \in \mathrm {T} := \mathrm {T} _ {1} \otimes \dots \otimes \mathrm {T} _ {d}, w i t h \mathrm {T} _ {j} := \prod_ {m \geq 0} \mathcal {H} _ {j} ^ {\otimes m},
$$
and for  $m \in \mathbb{N}$  we refer to  $\mu^m(\gamma) = \bigoplus_{\mathbf{i} \in \mathbb{N}^d: \deg(\mathbf{i}) = m} \mu^{\mathbf{i}}(\gamma)$  as the  $m$ -moments of  $\gamma$ .
In case of  $\mathcal{H}_i = \mathbb{R}$ , both definitions (1) and (3) apply for  $\mu^{\mathrm{i}}(\gamma)$ . Henceforth, we always refer to (3) when we write  $\mu^{\mathrm{i}}(\gamma)$ . Even in the finite-dimensional case, Def. 1 is useful, for instance when  $X_{1} \in \mathcal{H}_{1}$  and  $X_{2} \in \mathcal{H}_{2}$  have different state space ( $\mathcal{H}_1 \neq \mathcal{H}_2$ ).
3 Kernelized cumulants
We lift a random variable  $X = (X_{1},\ldots ,X_{d})\in \mathcal{X} = \mathcal{X}_{1}\times \dots \times \mathcal{X}_{d}$  via a feature map  $\Phi :\mathcal{X}\to \mathcal{H}$  into a Hilbert space valued random variable  $\Phi (X)$ . For the rest of the paper (i)  $\mathcal{X}_1,\ldots ,\mathcal{X}_d$  will denote a collection of Polish spaces, but the reader is invited to think of them as finite-dimensional Euclidean spaces, (ii)  $\mathcal{H}$  is an RKHS with kernel  $k$  and canonical feature map  $\Phi (x) = k(x,\cdot),^2$  and (iii) all kernels are assumed to be bounded. Our main results (Theorem 2 and Theorem 3) are that in this case the expected kernel trick applies to both items in the kernelized version of Theorem 1. The key to these results is an expression for inner products of cumulants in RKHSs (Lemma 1).
A combinatorial expression of cumulants. Classical cumulants can be defined via the moment generating function or via combinatorial sums over partitions (Appendix C.1). To generalize cumulants to RKHSs the combinatorial definition is the most efficient way. A partition  $\pi$  of  $m$  elements is a family of non-empty, disjoint subsets  $\pi_1, \ldots, \pi_b$  of  $\{1, \ldots, m\}$  whose union is the whole set; formally  $\bigcup_{j=1}^{b} \pi_j = \{1, \ldots, m\}$  and  $\pi_i \cap \pi_j = \emptyset$  for  $i \neq j$ . We call  $b$  the number of blocks of the partition  $\pi$  and use the shorthand  $|\pi|$  to denote it. The set of all partitions of  $m$  is denoted with  $P(m)$ . To formulate our main results, it is convenient to associate with a measure  $\gamma$  and a partition  $\pi$  the so-called partition measure  $\gamma_\pi$  that is given by permuting the marginals of  $\gamma$ .
Definition 2 (Partition measure). Let  $\gamma$  be a probability measure on  $\mathcal{X}_1\times \dots \times \mathcal{X}_d$  and  $\pi \in P(d)$  Define
$$
\gamma_ {\pi} := \gamma | _ {\mathcal {X} _ {\pi_ {1}}} \otimes \dots \otimes \gamma | _ {\mathcal {X} _ {\pi_ {b}}},
$$
where  $\mathcal{X}_{\pi_i}$  denotes the product space  $\prod_{j\in \pi_i}\mathcal{X}_j$  and  $\gamma |_{\mathcal{X}_{\pi_i}}$  is the corresponding marginal distribution of  $\gamma$ . We call  $\gamma_{\pi}$  the partition measure induced by  $\pi$ .
We also associate with  $\gamma$  and a multi-index  $\mathbf{i}$  the so-called diagonal measure  $\gamma^{\mathbf{i}}$  that is given by repeating marginals according to  $\mathbf{i}$ .
Definition 3 (Diagonal measure). Let  $\gamma$  be a probability measure on  $\mathcal{X}_1 \times \dots \times \mathcal{X}_d$  and  $\mathbf{i} = (i_1, \ldots, i_d) \in \mathbb{N}^d$ . Define
$$
\gamma^ {\mathbf {i}} := \mathrm {L a w} (\underbrace {X _ {1} , \ldots , X _ {1}} _ {i _ {1} t i m e s}, \underbrace {X _ {2} , \ldots , X _ {2}} _ {i _ {2} t i m e s}, \ldots , \underbrace {X _ {d} , \ldots , X _ {d}} _ {i _ {d} t i m e s}),
$$
where  $(X_{1},\ldots ,X_{d})\sim \gamma$  . We call  $\gamma^{\mathrm{i}}$  the diagonal measure induced by i.
In general, the partition measure  $\gamma_{\pi}$  and the diagonal measure are not probability measures on  $\mathcal{X}_1\times \dots \times \mathcal{X}_d$  but on spaces that are constructed by permuting or repeating  $\mathcal{X}_1,\ldots ,\mathcal{X}_d$ . Formally,  $\gamma_{\pi}$  is a probability measure on  $\mathcal{X}_{\pi_1}\times \dots \times \mathcal{X}_{\pi_b}$  and  $\gamma^{\mathrm{i}}$  is a probability measure on  $\mathcal{X}_1^{i_1}\times \dots \times \mathcal{X}_d^{i_d}$ ; thus,  $\gamma_{\pi}$  has  $d$  coordinates and  $\gamma^{\mathrm{i}}$  has  $\deg (\mathbf{i})$  coordinates. These two constructions can be combined, writing  $\gamma_{\pi}^{\mathrm{i}}$  for the measure  $(\gamma^{\mathrm{i}})_{\pi}$  which makes sense whenever  $\pi \in P(\deg (\mathbf{i}))$ . We can now write down our generalization of cumulants.
${}^{2}k\left( {x,\cdot }\right)$  denotes the function  ${x}^{\prime } \mapsto  k\left( {x,{x}^{\prime }}\right)$  with  $x \in  \mathcal{X}$  fixed.
$^3$  A kernel  $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$  is called bounded if there exists  $B\in \mathbb{R}$  such that  $\sup_{x,x^{\prime}\in \mathcal{X}}k(x,x^{\prime})\leq B$
Definition 4 (Kernelized cumulants). Let  $\gamma$  be a probability measure on  $\mathcal{X}_1 \times \dots \times \mathcal{X}_d$  and let  $(\mathcal{H}_1, k_1), \ldots, (\mathcal{H}_d, k_d)$  be RKHSs on  $\mathcal{X}_1, \ldots, \mathcal{X}_d$  respectively. We define the kernelized cumulants
$$
\kappa_ {k _ {1}, \dots , k _ {d}} (\gamma) := \left(\kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma)\right) _ {\mathbf {i} \in \mathbb {N} ^ {d}} \in \mathrm {T}
$$
as follows
$$
\kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma) := \sum_ {\pi \in P (m)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} k ^ {\otimes \mathbf {i}} ((X _ {1}, \dots , X _ {m}), \cdot),
$$
where  $m = \deg (\mathbf{i})$ ,  $c_{\pi} \coloneqq (-1)^{|\pi| - 1}(|\pi| - 1)!$ ,  $\gamma_{\pi}^{\mathbf{i}} = (\gamma^{\mathbf{i}})_{\pi}$  and
$$
\begin{array}{l} k ^ {\otimes \mathbf {i}} ((x _ {1}, \dots , x _ {m}), (y _ {1}, \dots , y _ {m})) := k _ {1} (x _ {1}, y _ {1}) \dots k _ {1} (x _ {i _ {1}}, y _ {i _ {1}}) \tag {4} \\ \dots k _ {d} \left(x _ {m - i _ {d} + 1}, y _ {m - i _ {d} + 1}\right) \dots k _ {d} \left(x _ {m}, y _ {m}\right) \\ \end{array}
$$
is the reproducing kernel of  $\mathcal{H}^{\otimes \mathbf{i}}$  where  $\mathcal{H} = \mathcal{H}_1\times \dots \times \mathcal{H}_d$ .
Def. 4 is the natural generalization of the combinatorial definition of cumulants in  $\mathbb{R}^d$  and Appendix C.2 gives an equivalent definition via a generating function analogous to (2). However, our posthoc justification that these are the "right" definitions for cumulants in an RKHS are Theorems 2 and 3 that show that these kernelized cumulants have the same powerful properties as classic cumulants in  $\mathbb{R}^d$  (Theorem 1).
Example 3.1 (Kernelized cumulants). Let  $\gamma$  be a probability measure on  $\mathcal{X}_1 \times \mathcal{X}_2$ , with the RKHSs  $(\mathcal{H}_1, k_1)$ ,  $(\mathcal{H}_2, k_2)$  given. Denote the random variables  $K_1 = k_1(X_1, \cdot)$ ,  $K_2 = k_2(X_2, \cdot)$  where  $(X_1, X_2) \sim \gamma$ . Then the degree two kernelized cumulants are given as  $\kappa_{k_1, k_2}^{(2,0)}(\gamma) = \mathbb{E}\left[K_1^{\otimes 2}\right] - \mathbb{E}\left[K_1^{\otimes 2}, \kappa_{k_1, k_2}^{(1,1)}(\gamma) = \mathbb{E}\left[K_1 \otimes K_2\right] - \mathbb{E}\left[K_1\right] \otimes \mathbb{E}\left[K_2\right], \kappa_{k_1, k_2}^{(0,2)}(\gamma) = \mathbb{E}\left[K_2^{\otimes 2}\right] - \mathbb{E}\left[K_2\right]^{\otimes 2}$ .
Inner products of cumulants. Computing inner products of moments is straightforward thanks to a nonlinear kernel trick, see Lemma 6 in the Appendix. For example, given two probability measures  $\gamma_{1},\gamma_{2}$  with corresponding random variables  $(X_{1},\ldots ,X_{d})\sim \gamma_{1},(Y_{1},\ldots ,Y_{d})\sim \gamma_{2}$  on  $\mathcal{X}_1\times \dots \times \mathcal{X}_d$  and RKHSs  $(\mathcal{H}_1,k_1),\ldots ,(\mathcal{H}_d,k_d)$  on  $\mathcal{X}_1,\ldots ,\mathcal{X}_d$  with bounded kernels, and  $\mathcal{H} = \mathcal{H}_1\times \dots \times \mathcal{H}_d$  we can express:
$$
\langle \mu_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma_ {1}), \mu_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma_ {2}) \rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} = \mathbb {E} _ {\gamma_ {1} \otimes \gamma_ {2}} k _ {1} (X _ {1}, Y _ {1}) ^ {i _ {1}} \dots k _ {d} (X _ {d}, Y _ {d}) ^ {i _ {d}}, \tag {5}
$$
where  $\mu_{k_1,\dots ,k_d}^i$  is defined in Def. 1, and the expectation is taken over the product measure  $\gamma_{1}\otimes \gamma_{2}$
Example 3.2. In the particular case of  $d = 1$ , (5) reduces to the well-known formula for the inner product of mean embeddings  $\langle \mu_k^{(1)}(\gamma_1), \mu_k^{(1)}(\gamma_2) \rangle_{\mathcal{H}_k} = \mathbb{E}_{\gamma_1 \otimes \gamma_2} k(X, Y)$ .
Lemma 1 (Inner product of cumulants). Let  $(\mathcal{H}_1, k_1), \ldots, (\mathcal{H}_d, k_d)$  be RKHSs with bounded kernels on  $\mathcal{X}_1, \ldots, \mathcal{X}_d$  respectively, and let  $\gamma$  and  $\eta$  two probability measures on  $\mathcal{X}_1 \times \dots \times \mathcal{X}_d$ ,  $\mathbf{i} = (i_1, \ldots, i_d) \in \mathbb{N}^d$  such that  $\deg(\mathbf{i}) = m$ . Then
$$
\langle \kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\gamma), \kappa_ {k _ {1}, \dots , k _ {d}} ^ {\mathbf {i}} (\eta) \rangle_ {\mathcal {H} ^ {\otimes \mathrm {i}}} = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}} \otimes \eta_ {\tau} ^ {\mathbf {i}}} k ^ {\otimes \mathbf {i}} ((X _ {1}, \dots , X _ {m}), (Y _ {1}, \dots , Y _ {m})).
$$
Point separating kernels. In the classic MMD setting the injectivity of the mean embedding  $\gamma \mapsto \mathbb{E}_{X\sim \gamma}[k(X,\cdot)]$  on probability measures (known as the characteristic property of the kernel  $k$ ) is equivalent to the MMD being a metric; this property is central in applications. We formulate our theoretical results in the next section using the much weaker property of what we term "point-separating" which is satisfied for essentially all popular kernels.
Definition 5 (Point-separating kernel). We call a kernel  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  point-separating if the canonical feature map  $\Phi: x \mapsto k(x, \cdot)$  is injective.
3.1 (Semi-)metrics for probability measures
In this section we use cumulants to characterize probability measures and show how to compute the distance between kernelized cumulants with the expected kernel trick.
Theorem 2 (Characterization of distributions with cumulants). Let  $\gamma$  and  $\eta$  be two probability measures on  $\mathcal{X}_1 \times \dots \times \mathcal{X}_d$ ,  $(\mathcal{H}_1, k_1), \ldots, (\mathcal{H}_d, k_d)$  RKHSs on the Polish spaces  $\mathcal{X}_1, \ldots, \mathcal{X}_d$  such that for every  $1 \leq j \leq d$ $k_j$  is a bounded, continuous, point-separating kernel. Then
$$
\gamma = \eta \text {i f a n d o n l y i f} \kappa_ {k _ {1}, \dots , k _ {d}} (\gamma) = \kappa_ {k _ {1}, \dots , k _ {d}} (\eta).
$$