SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\operatorname{Var}(X) \coloneqq \mathbb{E}[(X - \mu^1(\gamma))^2]$ . It is well known that the minimum variance unbiased estimators for the variance are more efficient than that for the second moment: denoting them by  $\widehat{\mu^2}$  and  $\widehat{\kappa}$  respectively, one can show (Bonnier and Oberhauser, 2020) that given  $N$  samples from  $X$ , the following holds
$$
\operatorname {V a r} \left(\widehat {\mu^ {2}}\right) = \operatorname {V a r} (\widehat {\kappa}) + \frac {2}{N} \left[ (\mathbb {E} X) ^ {4} - (\mathbb {E} X) ^ {2} \operatorname {V a r} (X) - 2 \frac {\operatorname {V a r} (X) ^ {2}}{N - 1} \right].
$$
This means that when  $X$  has a large mean, it is more efficient to estimate its variance than its second moment since the last term in the above expression dominates. Hence, the variance  $\operatorname{Var}(X)$  is typically a much more sensible second-order statistic than  $\mu^2(\gamma)$ . However, we emphasize that there are many other reasons why cumulants can have better properties as estimators
3. Cumulants characterize laws and the independence of two random variables manifests itself simply as vanishing of cross-cumulants. In view of the above item 2, this means for example that testing independence can be preferable in terms of vanishing cumulants rather than testing if moments factor  $\mathbb{E}[X^m Y^n] = \mathbb{E}[X^m ]\mathbb{E}[X^n ]$ , and similarly for testing if distributions are the same.
The caveat to the above points is that it is not true that cumulants are always preferable. For example, there are distributions for which (a) the moment generating function is not naturally exponential in  $t$ , (b) lower moments do not dominate higher moments, (c) consequently independence or two-sample testing become worse with cumulants. While one can write down conditions under which for example, the variance of the kernelized cumulants is lower, the use of cumulants among statisticians is to simply regard cumulants as arising from natural motivations which leads to another estimator in their toolbox.
The main idea of our paper is simply that for the same reasons that cumulants can turn out to be powerful for real or vector-valued random variables, cumulants of RKHS-valued random variables are a natural choice of statistics. The situation is more complicated since it requires formalizing moment- and cumulant-generating functions in RKHS but ultimately a kernel trick allows for circumventing the computational bottleneck of working in infinite dimensions and leads to computable estimators for independence and two-sample testing.
Further, we note that although cumulants are classic for vector-valued data, there seems to be not much work done about extending their properties to general structured data. Our kernelized cumulants apply to any set  $\mathcal{X}$  where a kernel is given. This includes many practically relevant examples such as strings (Lodhi et al., 2002), graphs (Kriege et al., 2020), or general sequentially ordered data (Kiraly and Oberhauser, 2019; Chevyrev and Oberhauser, 2022); a survey of kernels for structured data is provided by Gartner (2003).
B Technical background
In Section B.1 the tensor products  $\left(\bigotimes_{j=1}^{d}\mathcal{H}_{j}\right)$  and direct sums of Hilbert spaces  $\left(\bigoplus_{i\in I}\mathcal{H}_{i}\right)$  are recalled. Section B.2 is about tensor algebras over Hilbert spaces  $\left(\prod_{m\geq 0}\mathcal{H}^{\otimes m}\right)$ .
B.1 Tensor products and direct sums of Banach and Hilbert spaces
Tensor products of Hilbert spaces. For Hilbert spaces  $\mathcal{H},\ldots ,\mathcal{H}_d$  and  $(h_1,\dots ,h_d)\in \mathcal{H}_1\times \dots \times$ $\mathcal{H}_d$ , the multi-linear operator  $h_1\otimes \dots \otimes h_d\in \mathcal{H}_1\otimes \dots \otimes \mathcal{H}_d$  is defined as
$$
(h _ {1} \otimes \dots \otimes h _ {d}) (f _ {1}, \dots , f _ {d}) = \prod_ {j = 1} ^ {d} \langle h _ {j}, f _ {j} \rangle_ {\mathcal {H} _ {j}}
$$
for all  $(f_{1},\ldots ,f_{d})\in \mathcal{H}_{1}\times \dots \times \mathcal{H}_{d}$ . By extending the inner product
$$
\langle a _ {1} \otimes \dots \otimes a _ {d}, b _ {1} \otimes \dots \otimes b _ {d} \rangle_ {\mathcal {H} _ {1} \otimes \dots \otimes \mathcal {H} _ {d}} := \prod_ {j = 1} ^ {d} \langle a _ {j}, b _ {j} \rangle_ {\mathcal {H} _ {j}}
$$
to finite linear combinations of  $a_1 \otimes \dots \otimes a_d$ -s
$$
\left\{\sum_ {i = 1} ^ {n} c _ {i} \otimes_ {j = 1} ^ {d} a _ {i, j}: c _ {i} \in \mathbb {R}, a _ {i, j} \in \mathcal {H} _ {j}, n \geq 1 \right\}
$$
by linearity, and taking the topological completion one arrives at  $\mathcal{H}_1\otimes \dots \otimes \mathcal{H}_d$ . Specifically, if  $(\mathcal{H}_1,k_1),\ldots ,(\mathcal{H}_d,k_d)$  are RKHSs, then so is  $\mathcal{H}_1\otimes \dots \otimes \mathcal{H}_d = \mathcal{H}_{\otimes_{j = 1}^d k_j}$  (Berlinet and Thomas-Agnan, 2004, Theorem 13) with the tensor product kernel
$$
\left(\otimes_ {j = 1} ^ {d} k _ {j}\right) \left(\left(x _ {1}, \dots , x _ {d}\right), \left(x _ {1} ^ {\prime}, \dots , x _ {d} ^ {\prime}\right)\right) := \prod_ {j = 1} ^ {d} k _ {j} \left(x _ {j}, x _ {j} ^ {\prime}\right),
$$
where  $(x_{1},\ldots ,x_{d}),(x_{1}^{\prime},\ldots ,x_{d}^{\prime})\in \mathcal{X}_{1}\times \dots \times \mathcal{X}_{d}$
Tensor products of Banach spaces. For Banach spaces  $\mathcal{B}_1, \ldots, \mathcal{B}_d$ , the construction of  $\mathcal{B}_1 \otimes \dots \otimes \mathcal{B}_d$  is a little more involved (Lang, 2002) as one cannot rely on an inner product.
Direct sums of Hilbert and Banach spaces. Let  $(\mathcal{H}_i)_{i\in I}$  be Hilbert or Banach spaces where  $I$  is some index set. The direct sum of  $\mathcal{H}_i$ -s—written as  $\bigoplus_{i\in I}\mathcal{H}_i$ —consists of ordered tuples  $h = (h_i)_{i\in I}$  such that  $h_i\in \mathcal{H}_i$  for all  $i\in I$  and  $h_i = 0$  for all but a finite number of  $i\in I$ . Operations (addition, scalar multiplication) are performed coordinate-wise, and the inner product of  $a,b\in \bigoplus_{i\in I}\mathcal{H}_i$  is defined as  $\langle a,b\rangle_{\bigoplus_{i\in I}\mathcal{H}_i} = \sum_{i\in I}a_ib_i$ .
B.2 Tensor algebras
The tensor algebra  $\mathrm{T}_j$  over a Hilbert space  $\mathcal{H}_j$  is defined as the topological completion of the space
$$
\bigoplus_ {m \geq 0} \mathcal {H} _ {j} ^ {\otimes m}.
$$
Note that it can equivalently be defined as the subset of  $(h_0, h_1, h_2, \ldots) \in \prod_{m \geq 0} \mathcal{H}_j^{\otimes m}$  such that  $\sum_{m \geq 0} \|h_m\|_{\mathcal{H}_j^{\otimes m}}^2 < \infty$ , and as such it is a Hilbert space with norm
$$
\| (h _ {0}, h _ {1}, h _ {2}, \dots) \| _ {\prod_ {m \geq 0} \mathcal {H} _ {j} ^ {\otimes m}} ^ {2} = \sum_ {m \geq 0} \| h _ {m} \| _ {\mathcal {H} _ {j} ^ {\otimes m}} ^ {2}.
$$
$\mathrm{T}_j$  is also an algebra, endowed with the tensor product over  $\mathcal{H}_j$  as its product. For  $a = (a_0, a_1, a_2, a_3, \ldots)$ ,  $b = (b_0, b_1, b_2, b_3, \ldots) \in \mathrm{T}_j$ , their product can be written down in coordinates as
$$
a \cdot b = \left(\sum_ {i = 0} ^ {m} a _ {i} \otimes b _ {m - i}\right) _ {m \geq 0}.
$$
For a sequence  $\mathcal{H}_1, \ldots, \mathcal{H}_d$  of Hilbert spaces, we define
$$
\mathrm {T} := \mathrm {T} _ {1} \otimes \dots \otimes \mathrm {T} _ {d},
$$
where  $\mathrm{T}_j = \prod_{m\geq 0}\mathcal{H}_j^{\otimes m}(j = 1,\ldots ,d)$ . Let  $\mathcal{H} = \mathcal{H}_1\times \dots \times \mathcal{H}_d$ , and recall that given a tuple of integers  $\mathbf{i} = (i_1,\dots ,i_d)\in \mathbb{N}^d$  we define  $\mathcal{H}^{\otimes \mathbf{i}}\coloneqq \mathcal{H}_1^{\otimes i_1}\otimes \dots \otimes \mathcal{H}_d^{\otimes i_d}$ . This allows us to write down a multi-grading for  $\mathrm{T}$  as
$$
\mathrm {T} = \prod_ {\mathbf {i} \in \mathbb {N} ^ {d}} \mathcal {H} ^ {\otimes \mathbf {i}}. \tag {8}
$$
Note that this gives credence to us using multi-indices  $\mathbf{i} \in \mathbb{N}^d$  to describe elements of the tensor algebra, as the multi-indices form its multi-grading.
Furthermore,  $\mathrm{T}$  is a multi-graded algebra when endowed with the (linear extension of the) following multiplication defined on the components of  $\mathrm{T}$
$$
\star : \mathcal {H} ^ {\otimes \mathbf {i} ^ {1}} \times \mathcal {H} ^ {\otimes \mathbf {i} ^ {2}} \rightarrow \mathcal {H} ^ {\otimes (\mathbf {i} ^ {1} + \mathbf {i} ^ {2})}, \tag {9}
$$
$$
\left(x _ {1} \otimes \dots \otimes x _ {d}\right) \star \left(y _ {1} \otimes \dots \otimes y _ {d}\right) = \left(x _ {1} \cdot y _ {1}\right) \otimes \dots \otimes \left(x _ {d} \cdot y _ {d}\right),
$$
so that for  $a = \left(a^{\mathbf{i}}\right)_{\mathbf{i}\in \mathbb{N}^{d}},b = \left(b^{\mathbf{i}}\right)_{\mathbf{i}\in \mathbb{N}^{d}}\in \mathrm{T},$  their product can be written down as
$$
(a \star b) ^ {\mathbf {i}} = \sum_ {\mathbf {i} ^ {1} + \mathbf {i} ^ {2} = \mathbf {i}} a ^ {\mathbf {i} ^ {1}} \star b ^ {\mathbf {i} ^ {2}} \tag {10}
$$
where addition of tuples  $\mathbf{i}^1, \mathbf{i}^2 \in \mathbb{N}^d$  is defined as  $\mathbf{i}^1 + \mathbf{i}^2 = (i_1^1 + i_1^2, \ldots, i_d^1 + i_d^2)$ . With the degree of a tuple defined as  $\deg(\mathbf{i}) = i_1 + \cdots + i_d$ ,  $\mathrm{T}$  is also a graded algebra, with the grading written down as
$$
\mathrm {T} = \prod_ {m \geq 0} \bigoplus_ {\{\mathbf {i} \in \mathbb {N} ^ {d}: \deg (\mathbf {i}) = m \}} \mathcal {H} ^ {\otimes \mathbf {i}},
$$
so that if one multiplies two elements together, the degree of their product is the sum of their degree.
Finally we note that  $\mathrm{T}$  is a unital algebra and the unit has the explicit form
$$
(1, 0, 0, \ldots),
$$
i.e. the element consisting of only a 1 at degree 0.
This section is dedicated to proofs. The equivalence between the combinatorial expressions of cumulants and the definition via a moment generating function is proved in Section C.2. The derivation of our main results (Theorem 2 and Theorem 3) are detailed in Section C.3.
C.1 Equivalent definitions of cumulants in  $\mathbb{R}^d$
Here we introduce a classical definition of cumulants via a moment generating function and its equivalence to the combinatorial expressions. If  $X = (X_{1},\ldots ,X_{d})$  is an  $\mathbb{R}^d$ -valued random variable distributed according to  $X\sim \gamma$ , then recall that
$$
\mu^ {\mathbf {i}} (\gamma) = \mathbb {E} \left[ X _ {1} ^ {i _ {1}} \dots X _ {d} ^ {i _ {d}} \right] \in \mathbb {R}
$$
for  $\mathbf{i} = (i_1,\dots ,i_d)\in \mathbb{N}^d$
The following definition of the cumulants  $\kappa^{\mathrm{i}}(\gamma)$  of  $\gamma$  are equivalent
1.  $\sum_{\mathbf{i}\in \mathbb{N}^d}\kappa^{\mathbf{i}}(\gamma)\frac{\pmb{\theta}^{\mathbf{i}}}{\mathbf{i}!} = \log \sum_{\mathbf{i}\in \mathbb{N}^d}\mu^{\mathbf{i}}(\gamma)\frac{\pmb{\theta}^{\mathbf{i}}}{\mathbf{i}!},$
2.  $\kappa^{\mathbf{i}}(\gamma) = \sum_{\pi \in P(d)}c_{\pi}\mu^{\mathbf{i}}(\gamma_{\pi}^{\mathbf{i}}),$
where  $\theta = (\theta_{1},\dots ,\theta_{d})\in \mathbb{R}^{d}$ ,  $c_{\pi} = (-1)^{|\pi |}(|\pi | - 1)!$ . The equivalence between these two definitions of cumulants via a generating function and via their combinatorial definition, is classical (McCullagh, 2018) even if our notation here is non-standard in the classical case. This equivalence is also at the heart of many proofs about properties of cumulants since some properties are easier to prove via one or the other definition.
C.2 Equivalent definitions of cumulants in RKHS
In the main text, we defined cumulants in RKHS by mimicking the combinatorial definition of cumulants in  $\mathbb{R}^d$ . It is natural and useful to also have the analogous definition via a "generating function" for RKHS-valued random variables. However, to generalize the definition via the logarithm of the moment generating function to random variables in RKHS, requires to define a logarithm for tensor series of moments. In this part, we show that this can be done and that indeed the two definitions are equivalent.
We use the shorthand  $\kappa(\gamma) \coloneqq \kappa_{k_1, \ldots, k_d}(\gamma)$ ,  $\mu(\gamma) \coloneqq \mu_{k_1, \ldots, k_d}(\gamma)$ , and we overload the notation  $(X_1, \ldots, X_d)$  with  $(k_1(\cdot, X_1), \ldots, k_d(\cdot, X_d))$ . With this notation, we show that given coordinates  $\mathbf{i} \in \mathbb{N}^d$ , one may express the generalized cumulant  $\kappa^{\mathrm{i}}(\gamma)$  as either a combinatorial sum over moments indexed by partitions, or by using the cumulant generating function.
More specifically, we show that the generalized cumulant of a probability measure  $\gamma$  on  $\mathcal{H}_1 \times \dots \times \mathcal{H}_d$  defined as
$$
\kappa^ {\mathbf {i}} (\gamma) = \sum_ {\pi \in P (m)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} (X ^ {\otimes \mathbf {i}}),
$$
where  $c_{\pi} = (-1)^{|\pi| - 1} (|\pi| - 1)!$  can also be expressed as coordinates in the tensorized logarithm of the moment series. Motivated by the Taylor series expansion of the classic logarithm, we define
$$
\log : \mathrm {T} \rightarrow \mathrm {T}, \quad x \mapsto \sum_ {n \geq 1} \frac {(- 1) ^ {n - 1}}{n} (x - 1) ^ {\star n},
$$
where  $\star$  denotes the product as defined in (9) and for  $t\in \mathrm{T}$ ,  $t^{\star n}$  is defined as
$$
t ^ {\star n} = \underbrace {t \star \cdots \star t} _ {\text {n - t i m e s}},
$$
or coordinate-wise  $(t^{\star n})^{\mathbf{i}} = \sum_{\mathbf{i}^{1} + \dots +\mathbf{i}^{n} = \mathbf{i}}t^{\mathbf{i}^{1}}\star \dots \star t^{\mathbf{i}^{n}}$  for  $\mathbf{i}\in \mathbb{N}^d$ . Note that unlike the classical logarithm  $\log :\mathbb{R}_{+}\to \mathbb{R}$ , the tensorized logarithm is defined on the whole space as a formal expression. This can be summarized in the following lemma:
$$
\kappa^ {\mathbf {i}} (\gamma) = \sum_ {\pi \in P (m)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} \left(X ^ {\otimes \mathbf {i}}\right) = \left(\log \mu \left(\gamma^ {\mathbf {i}}\right)\right) ^ {\mathbf {1} _ {m}}, \tag {11}
$$
where  $\mathbf{1}_m = (1,\dots ,1)\in \mathbb{N}^m$
By iterating (10) we can express (11) as
$$
\sum_ {j = 1} ^ {m} \frac {(- 1) ^ {j - 1}}{j} \sum_ {\mathbf {i} ^ {1} + \dots + \mathbf {i} ^ {j} = \mathbf {1} _ {m}} \mu^ {\mathbf {i} ^ {1}} (\gamma^ {\mathbf {i}}) \star \dots \star \mu^ {\mathbf {i} ^ {j}} (\gamma^ {\mathbf {i}}),
$$
and our goal is to express this as a sum over partitions. We will use the notation  $[n] = \{1, \dots, n\}$ . We can achieve our goal in two parts:
1. Show that for a fixed  $\mathbf{i} \in \mathbb{N}^d$  with  $\deg(\mathbf{i}) = m$  we can express (11) as a sum over all surjective functions from  $[m]$  to  $[j]$ .
2. Show that this sum over functions reduces to a sum over partitions.
Part 1. Note that given  $\mathbf{i}^1 +\dots +\mathbf{i}^j = \mathbf{1}_m$  we may define  $h:[m]\to [j]$  by the relation  $(\mathbf{i}^{h(n)})_n = 1$  that is, we take  $h(n)$  to be the index  $c$  for which the multi-index  $\mathbf{i}^c$  is 1 at  $n$ . Note that this function is necessarily surjective since the sum is taken over non-zero multi-indices. Equivalently, for any surjective function  $h:[m]\rightarrow [j]$  we may define multi-indices by setting
$$
(\mathbf {i} ^ {c}) _ {n} = \left\{ \begin{array}{l l} 1 & \text {i f} n \in h ^ {- 1} (c) \\ 0 & \text {o t h e r w i s e} \end{array} \right..
$$
Note that any such multi-index will be non-zero since the function is assumed to be surjective. With this identification we can write
$$
\left(\log \mu (\gamma^ {\mathbf {i}})\right) ^ {\mathbf {1} _ {m}} = \sum_ {j = 1} ^ {m} \frac {(- 1) ^ {j - 1}}{j} \sum_ {h: [ m ] \rightarrow [ j ]} \mu^ {\mathbf {i} ^ {h - 1} (1)} (\gamma^ {\mathbf {i}}) \star \dots \star \mu^ {\mathbf {i} ^ {h - 1} (j)} (\gamma^ {\mathbf {i}}).
$$
Part 2. Recall that given a function  $h: [m] \to [j]$  we can associate it to its corresponding partition  $\pi_h \in \mathcal{P}(m)$  by considering the set  $\{h^{-1}(1), \ldots, h^{-1}(j)\}$ , and there are exactly  $j!$  different functions corresponding to a given partition, which are given by re-ordering the values  $1, \ldots, j$ . This reordering of the blocks does not change the summands since the marginals of the partition measure are always copies of each other and hence self-commute, hence a product of moments like  $\mu^{\mathrm{i}^{h^{-1}(1)}}(\gamma^{\mathrm{i}}) \star \dots \star \mu^{\mathrm{i}^{h^{-1}(j)}}(\gamma^{\mathrm{i}})$  can always be written as  $\mu^{\mathrm{i}}(\gamma_{\pi_h}^{\mathrm{i}})$ , the  $\mathrm{i}$ -th coordinate of the moment sequence of the partition measure  $\gamma_{\pi_h}^{\mathrm{i}}$ . With this in mind we can write
$$
\begin{array}{l} \left(\log \mu (\gamma^ {\mathbf {i}})\right) ^ {\mathbf {1} _ {m}} = \sum_ {j = 1} ^ {m} \frac {(- 1) ^ {j - 1}}{j} \sum_ {h: [ m ] \rightarrow [ j ]} \mu^ {\mathbf {i}} (\gamma_ {\pi_ {h}} ^ {\mathbf {i}}) = \sum_ {\pi \in P (m)} \frac {(- 1) ^ {| \pi | - 1}}{| \pi |} | \pi |! \mu^ {\mathbf {i}} (\gamma_ {\pi} ^ {\mathbf {i}}) \\ = \sum_ {\pi \in P (m)} c _ {\pi} \mu^ {\mathbf {i}} (\gamma_ {\pi} ^ {\mathbf {i}}) = \sum_ {\pi \in P (m)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} (X ^ {\otimes \mathbf {i}}). \\ \end{array}
$$
From this it immediately follows that for two probability measures  $\gamma, \eta$  we can write
$$
\begin{array}{l} \langle \kappa^ {\mathbf {i}} (\gamma), \kappa^ {\mathbf {i}} (\eta) \rangle_ {\mathcal {H} ^ {\otimes \mathbf {i}}} = \langle \sum_ {\pi \in P (m)} c _ {\pi} \mathbb {E} _ {\gamma_ {\pi} ^ {\mathbf {i}}} (X ^ {\otimes \mathbf {i}}), \sum_ {\tau \in P (m)} c _ {\tau} \mathbb {E} _ {\eta_ {\tau} ^ {\mathbf {i}}} (Y ^ {\otimes \mathbf {i}}) \rangle_ {\mathcal {H} ^ {\otimes \mathbf {i}}} \\ = \sum_ {\pi , \tau \in P (m)} c _ {\pi} c _ {\tau} \mathbb {E} _ {(X, Y) \sim \gamma_ {\pi} ^ {\mathbf {i}} \otimes \eta_ {\tau} ^ {\mathbf {i}}} \langle X ^ {\otimes \mathbf {i}}, Y ^ {\otimes \mathbf {i}} \rangle_ {\mathcal {H} ^ {\otimes \mathbf {i}}}. \\ \end{array}
$$
Lemma 1 then follows from the definition of the tensor products.
C.3 Proof of Theorem 2 and Theorem 3
In this section we present the proofs of Theorem 2 and Theorem 3. We do this in a slightly more abstract setting where the feature maps take values in Banach spaces for clarity, until the end when we again restrict our attention to RKHSs. We start out by showing that polynomial functions of the feature maps characterize measures (Lemma 5). From this result we show that cumulants have the same property (Theorem 4), and lastly that this also holds when working directly with the kernels (Proposition 1).
A monomial on separable Banach spaces  $\mathcal{B}_1, \ldots, \mathcal{B}_d$  is any expression of the form
$$
M (x _ {1}, \ldots , x _ {d}) = \prod_ {j = 1} ^ {i _ {1}} \langle f _ {j} ^ {1}, x _ {1} \rangle \dots \prod_ {j = 1} ^ {i _ {d}} \langle f _ {j} ^ {d}, x _ {d} \rangle
$$
for some  $(i_1, \ldots, i_d) \in \mathbb{N}^d$ , where  $f_j^i \in \mathcal{B}_i^\star$  are elements of the dual space  $\mathcal{B}_i^\star$  and  $x_i \in \mathcal{B}_i$ . Finite linear combinations of monomials are called the polynomials. Recall that a set of functions  $F$  on a set  $S$  is said to separate the points of  $S$  if for every  $x \neq y \in S$  there exists  $f \in F$  such that  $f(x) \neq f(y)$ .
Lemma 5 (Polynomial functions of feature maps characterize probability measures). Let  $\mathcal{X}_1, \ldots, \mathcal{X}_d$  be Polish spaces,  $\mathcal{B}_1, \ldots, \mathcal{B}_d$  separable Banach spaces and  $\varphi_i : \mathcal{X}_i \to \mathcal{B}_i$  be continuous, bounded, and injective functions. Then the set of functions on the Borel probability measures  $\mathcal{P}\left(\prod_{i=1}^d \mathcal{X}_i\right)$  of  $\prod_{i=1}^d \mathcal{X}_i$
$$
\mathcal {P} \left(\prod_ {i = 1} ^ {d} \mathcal {X} _ {i}\right)\rightarrow \mathbb {R}, \quad \gamma \mapsto \int_ {\prod_ {i = 1} ^ {d} \mathcal {X} _ {i}} p \left(\varphi_ {1} \left(x _ {1}\right), \dots , \varphi_ {d} \left(x _ {d}\right)\right) \mathrm {d} \gamma \left(x _ {1}, \dots , x _ {d}\right),
$$
where  $p$  ranges over all polynomials, separates the points of  $\mathcal{P}\left(\prod_{i=1}^{d} \mathcal{X}_i\right)$ .
Proof. We first show that the pushforward map
$$
\prod_ {i = 1} ^ {d} \varphi_ {i}: \mathcal {P} \left(\prod_ {i = 1} ^ {d} \mathcal {X} _ {i}\right)\rightarrow \mathcal {P} \left(\prod_ {i = 1} ^ {d} \mathcal {B} _ {i}\right)
$$
is injective. This is done in two parts, first we show that every Borel measure on  $\prod_{i=1}^{d}\mathcal{X}_{i}$  is a Radon measure, then we show that the pushforward map is injective on Radon measures. To see the first part, note that since  $\mathcal{X}_1,\ldots,\mathcal{X}_d$  are Polish spaces, so is their product space  $\prod_{i=1}^{d}\mathcal{X}_{i}$  (Dudley 2004, Theorem 2.5.7; Willard 1970, Theorem 16.4c), and since Borel measures on Polish spaces are Radon measures (Bogachev, 2007, Theorem 7.1.7), any  $\gamma \in \mathcal{P}(\prod_{i=1}^{d}\mathcal{X}_{i})$  must be a Radon measure.
For the second part, note that
$$
\prod_ {i = 1} ^ {d} \varphi_ {i}: \prod_ {i = 1} ^ {d} \mathcal {X} _ {i} \rightarrow \prod_ {i = 1} ^ {d} \mathcal {B} _ {i}, \quad \left(\prod_ {i = 1} ^ {d} \varphi_ {i}\right) (x _ {1}, \dots , x _ {d}) \mapsto \prod_ {i = 1} ^ {d} \varphi_ {i} (x _ {i})
$$
is a norm bounded, continuous injection. Since  $\prod_{i=1}^{d}\mathcal{B}_{i}$  is a Hausdorff space,  $\prod_{i=1}^{d}\varphi_{i}$  is a homeomorphism on compacts since continuous injections into Hausdorff spaces are homeomorphisms on compacts (Rudin, 1953, Theorem 4.17). Let  $\mu, \nu \in \mathcal{P}\left(\prod_{i=1}^{d}\mathcal{X}_{i}\right)$  be two Radon measures such that their pushforwards are the same  $\prod_{i=1}^{d}\varphi_{i}(\mu) = \prod_{i=1}^{d}\varphi_{i}(\nu)$ , then for any compact  $C \subseteq \prod_{i=1}^{d}\mathcal{X}_{i}$  we have  $\mu(C) = \nu(C)$  as  $\prod_{i=1}^{d}\varphi_{i}: C \to \prod_{i=1}^{d}\varphi_{i}(C)$  is a homeomorphism. Since Radon measures are characterized by their values on compacts, this implies that  $\mu = \nu$ . Hence the pushforward map is injective.
Denote by  $K$  the image of  $\prod_{i=1}^{d}\mathcal{X}_{i}$  under the mapping  $\prod_{i=1}^{d}\varphi_{i}$  in  $\prod_{i=1}^{d}\mathcal{B}_{i}$ . Note that  $K$  is a bounded Polish space. It is enough to show that the polynomials separate the points of  $\mathcal{P}(K)$ . To see this, note that the polynomials form an algebra of continuous functions that separate the points of  $\prod_{i=1}^{d}\mathcal{B}_{i}$ , and when restricted to  $K$  they are bounded, since  $K$  is norm bounded. Since  $K$  is Polish, any Borel measure is Radon, and we can apply the Stone-Weierstrass theorem for Radon measures (Bogachev, 2007, Exercise 7.14.79) to get the assertion.
In what follows we will use the following index notation for linear functionals. Fix some tuple  $\mathbf{i} = (i_1, \ldots, i_d) \in \mathbb{N}^d$  with  $\deg(\mathbf{i}) = m$ . Given separable Banach spaces  $\mathcal{B}_1, \ldots, \mathcal{B}_d$  we use the notation
$$
\mathcal {B} ^ {\otimes \mathbf {i}} := \mathcal {B} _ {1} ^ {\otimes i _ {1}} \otimes \dots \otimes \mathcal {B} _ {d} ^ {\otimes i _ {d}}
$$
These monomials naturally extend the classical ones.
and given an element  $x = (x_{1},\ldots ,x_{d})\in \prod_{i = 1}^{d}\mathcal{B}_{i}$  we write  $x^{\mathbf{i}}\coloneqq x_1^{\otimes i_1}\otimes \dots \otimes x_d^{\otimes i_d}$  so that  $x^{\mathbf{i}}\in \mathcal{B}^{\otimes \mathbf{i}}$ . If we have functions  $(\varphi_i)^d_{i = 1}$  such that  $\varphi_{i}:\mathcal{X}_{i}\to \mathcal{B}_{i}$  on some Polish spaces  $\mathcal{X}_1,\ldots ,\mathcal{X}_d$ , then we write
$$
\varphi^ {\otimes \mathbf {i}} := \varphi_ {1} ^ {\otimes i _ {1}} \otimes \dots \otimes \varphi_ {d} ^ {\otimes i _ {d}}, \quad \varphi^ {\otimes \mathbf {i}}: \prod_ {i = 1} ^ {d} \mathcal {X} _ {i} \to \mathcal {B} ^ {\otimes \mathbf {i}}.
$$
Given a collection of linear functionals  $F \in \prod_{j=1}^{d} \left( \mathcal{B}_j^\star \right)^{i_j}$  such that  $F = (f_1, \ldots, f_d)$  we write
$$
F ^ {\otimes \mathbf {i}} := f _ {1} \otimes \dots \otimes f _ {d}, \quad F ^ {\otimes \mathbf {i}} \in \left(\mathcal {B} ^ {\otimes \mathbf {i}}\right) ^ {\star}.
$$