Title: Kernelized Cumulants: Beyond Kernel Mean Embeddings
Venue: NeurIPS 2023
Accepted: yes
Keywords: kernel

cumulant

mean embedding

Hilbert-Schmidt independence criterion

maximum mean discrepancy
Abstract: In $\mathbb{R}^d$, it is well-known that cumulants provide an alternative to moments that can achieve the same goals with numerous benefits such as lower variance estimators. In this paper we extend cumulants to reproducing kernel Hilbert spaces (RKHS) using tools from tensor algebras and show that they are computationally tractable by a kernel trick. These kernelized cumulants provide a new set of all-purpose statistics; the classical maximum mean discrepancy and Hilbert-Schmidt independence criterion arise as the degree one objects in our general construction. We argue both theoretically and empirically (on synthetic, environmental, and traffic data analysis) that going beyond degree one has several advantages and can be achieved with the same computational complexity and minimal overhead in our experiments.

Kernelized Cumulants: Beyond Kernel Mean Embeddings

Abstract

In R d , it is well-known that cumulants provide an alternative to moments that can achieve the same goals with numerous benefits such as lower variance estimators. In this paper we extend cumulants to reproducing kernel Hilbert spaces (RKHS) using tools from tensor algebras and show that they are computationally tractable by a kernel trick. These kernelized cumulants provide a new set of all-purpose statistics; the classical maximum mean discrepancy and Hilbert-Schmidt independence criterion arise as the degree one objects in our general construction. We argue both theoretically and empirically (on synthetic, environmental, and traffic data analysis) that going beyond degree one has several advantages and can be achieved with the same computational complexity and minimal overhead in our experiments.

Keywords: kernel, cumulant, mean embedding, Hilbert-Schmidt independence criterion, kernel Lancaster interaction, kernel Streitberg interaction, maximum mean discrepancy, maximum variance discrepancy

1 Introduction

The moments of a random variable are arguably the most popular all-purpose statistic. However, cumulants are often more favorable statistics than moments. For example, if µ m := E [ X m ] denotes the moments of a real-valued random variable X , then µ 2 = µ 2 1 +Var( X ) and hence the variance that directly measures the fluctuation around the mean is a much better statistic for scale than the second moment µ 2 , see Appendix A. Cumulants provide a systematic way to only record the parts of the moment sequence that are not already captured by lower-order moments. While the moment and cumulant sequences ( µ m ) m and ( κ m ) m carry the same information, cumulants have several desirable properties that generalize to R d -valued random variables (McCullagh, 2018). Among these properties of cumulants, the ones that are important for our paper is that they can characterize distributions and statistical (in)dependence.

Kernel embeddings. Mean and covariance arise naturally in the context of kernel-enriched domains. Kernel techniques (Schölkopf and Smola, 2002; Steinwart and Christmann, 2008; Saitoh and Sawano, 2016) provide a principled and powerful approach for lifting data points to a so-called reproducing kernel Hilbert space (RKHS; Aronszajn 1950). Considering the mean of this feature - referred to as kernel mean embedding (KME; Berlinet and Thomas-Agnan 2004; Smola et al. 2007) - also enables one to represent probability measures, and to induce a semi-metric referred to as maximum mean discrepancy (MMD; Smola et al. 2007; Gretton et al. 2012) and an independence measure called the

Hilbert-Schmidt independence criterion (HSIC 1 ; Gretton et al. 2005). It is known that MMD is a metric when the underlying kernel is characteristic (Fukumizu et al., 2008; Sriperumbudur et al., 2010). HSIC captures independence for d = 2 components with characteristic kernels (Lyons, 2013, Theorem 3.11) and for d > 2 components (Quadrianto et al., 2009; Sejdinovic et al., 2013a; Pfister et al., 2018) with universal ones (Szabó and Sriperumbudur, 2018). MMD belongs to the family of integral probability metrics (IPM; Zolotarev 1983; Müller 1997) when in the IPM the underlying function class is chosen to be the unit ball of the RKHS. MMD and HSIC (with d = 2 ) are known to be equivalent (Sejdinovic et al., 2013b) to the notions of energy distance (Baringhaus and Franz, 2004; Székely and Rizzo, 2004, 2005)-also called N-distance (Zinger et al., 1992; Klebanov, 2005)and distance covariance (Székely et al., 2007; Székely and Rizzo, 2009; Lyons, 2013) of the statistics literature. Both MMD and HSIC can be expressed in terms of expectations of kernel values which can be leveraged to design efficient estimators for them; we will refer to this trick as the expected kernel trick. A recent survey on mean embedding and their applications is given by Muandet et al. (2017). The closest previous work to ours is by Makigusa (2020) who considers the variance in the RKHS - which in our setting can be identified as the first kernelized cumulant after the kernel mean embedding - for two-sample testing like we do in parts of this paper. Unfortunately, Makigusa (2020) does not provide conditions on the validity of the resulting maximum variance discrepancy, and it is not formulated in the context of cumulant embeddings.

Contribution. The main contribution of our paper is to introduce cumulants of random variables in RKHSs and to show that under mild conditions the proposed kernelized cumulants characterize distributions (Theorem 2) and independence (Theorem 3). Thanks to the RKHS formulation, kernelized cumulants have computable estimators (Lemma 2 and Lemma 3) and they show strong performance in two-sample and independence testing on various benchmarks (Section 4). Although cumulants are a classic tool in multi-variate statistics, they have not received attention in the kernel literature. The primary technical challenge to circumvent in the derivation of fundamental properties of cumulants is the rich combinatorial structure which already arises in R d from their definition via moment-generating function which is closely linked to the partition lattice (Speed, 1983, 1984). In an RKHS, even the definition of cumulants is non-straightforward. The key insight for our extension is that the combinatorial expressions for cumulants in R d can be generalized by using tools from tensor algebras. This in turn allows us to derive the main properties of the RKHS cumulants that underpin their statistical properties.

Broader impact & limitations. We do not see any direct negative societal impact arising from the proposed new set of all-purpose kernel-based divergence and dependence measure. Choosing the underlying kernels in an optimal fashion-even for MMD in two-sample testing (Hagrass et al., 2022) or goodness-of-fit testing (Hagrass et al., 2023)-and showing optimal rates-even for MMD with radial kernels on R d (Tolstikhin et al., 2016)- are quite challenging problems requiring dedicated analysis, and are not addressed here.

Outline. The paper is structured as follows: In Section 2 we formulate the notion of cumulants of random variables in Hilbert spaces. In Section 3 we prove a kernelized version of the classical result of R d -valued random variables on the characterization of distributions and independence using cumulants. We show that one can leverage the expected kernel trick to derive efficient estimators for our novel statistics, with MMD and HSIC arising specifically as the 'degree 1 ' objects. In Section 4 we demonstrate numerically that going beyond degree 1 is advantageous. We provide a technical background (on cumulants, tensor products and tensor algebras), proofs, further details on numerical experiments, and our V-statistic based estimators in the Appendices.

2 Moments and cumulants

We briefly revisit classical cumulants and define cumulants of random variables in Hilbert spaces.

Moments. Let γ be a probability measure on R d and ( X 1 , . . . , X d ) ∼ γ . The moments µ ( γ ) = ( µ i ( γ )) i ∈ N d of γ are defined as

where i = ( i 1 , . . . , i d ) ∈ N d denotes a d -tuple of non-negative integers ( i 1 , . . . , i d ≥ 0 ). The degree of an element i ∈ N d is defined as deg( i ):= i 1 + · · · + i d . For m ∈ N , let µ m ( γ ) := ( µ i ( γ )) deg( i )= m which we refer to as the m -th moments of γ with the convention that µ 0 ( γ ) = 1 .

Cumulants. Cumulants κ ( γ ) = ( κ i ( γ )) i ∈ N d can be defined by the moment generating function as

where we denote i ! = i 1 ! · · · i d ! and θ i = θ i 1 1 · · · θ i d d ; an equivalent definition of cumulants is via a combinatorial expression of partitions (elaborated in Appendix C.1). Cumulants have several attractive properties, the following forms our main motivation.

Theorem 1 (Characterization of distributions with cumulants on R d , from Proposition 1 in Jammalamadaka et al. 2006) . Let γ be a probability measure on a bounded subset of R d with cumulants κ ( γ ) and let ( X 1 , . . . , X d ) ∼ γ . Then

γ ↦→ κ ( γ ) is injective.

X 1 , . . . , X d are jointly independent if and only if κ i ( γ ) = 0 for all d-tuples of positive integers i ∈ N d + .

2.1 Moments in Hilbert spaces

Instead of directly considering the law of a tuple of random variables ( X 1 , . . . , X d ) in a product space X 1 ×··· × X d , it can be advantageous to use feature maps Φ i : X i →H i and instead study the distribution of the H 1 ×··· × H d -valued random variable ( Φ 1 ( X 1 ) , . . . , Φ d ( X d ) ) . Motivated by this lifting, we study here moments of Hilbert-space valued random variables and assume in this subsection (with a slight abuse of notations) that one has already applied the lifting and X i ∈ H i where i = 1 , . . . , d . In Section 3 we specialize the construction to RKHSs, and use these moments (Def. 1) to define kernelized cumulants.

Moments. In the finite-dimensional case (1) we defined the moment sequence by taking expectations of products of the coordinates of the underlying random variable. For the infinite-dimensional case, it is convenient to develop a coordinate-free definition which can be accomplished by using tensors. To do so we make use of the following results about Hilbert spaces: for real Hilbert spaces H 1 and H 2 the tensor product H 1 ⊗H 2 is the Hilbert space given by completion of the tensor product of H 1 and H 2 as vector space; we also write H ⊗ m 1 : = H 1 ⊗··· ⊗ H 1 ︸ ︷︷ ︸ . Similarly, the direct sum m-times

H 1 ⊕ H 2 is a Hilbert space. It is natural to consider E [ X ⊗ m 1 ] ∈ H ⊗ m 1 as the m -th moment of a H 1 -valued random variable X 1 where the integral in the expectation is meant in Bochner sense. Consequently the natural state space for all moments of a H 1 -valued random variable is the tensor algebra T 1 := ∏ m ≥ 0 H ⊗ m 1 where by convention H ⊗ 0 1 := R . See Appendix B for more details on tensor products of Hilbert spaces and tensor algebras.

Since we are interested in the general case of a H 1 × · · · × H d -valued random variable X = ( X 1 , . . . , X d ) we arrive at the definition below.

Definition 1 (Moments in Hilbert spaces) . Let γ be a probability measure on H : = H 1 ×··· × H d and let ( X 1 , . . . , X d ) ∼ γ . We define

for every i ∈ N d whenever the above expectation exists. The moment sequence is defined as the element

and for m ∈ N we refer to µ m ( γ ) = ⊕ i ∈ N d :deg( i )= m µ i ( γ ) as the m -moments of γ .

̸

In case of H i = R , both definitions (1) and (3) apply for µ i ( γ ) . Henceforth, we always refer to (3) when we write µ i ( γ ) . Even in the finite-dimensional case, Def. 1 is useful, for instance when X 1 ∈ H 1 and X 2 ∈ H 2 have different state space ( H 1 = H 2 ).

3 Kernelized cumulants

We lift a random variable X = ( X 1 , . . . , X d ) ∈ X = X 1 ×··· × X d via a feature map Φ : X → H into a Hilbert space valued random variable Φ( X ) . For the rest of the paper (i) X 1 , . . . , X d will denote a collection of Polish spaces, but the reader is invited to think of them as finite-dimensional Euclidean spaces, (ii) H is an RKHS with kernel k and canonical feature map Φ( x ) = k ( x, · ) , 2 and (iii) all kernels are assumed to be bounded. 3 Our main results (Theorem 2 and Theorem 3) are that in this case the expected kernel trick applies to both items in the kernelized version of Theorem 1. The key to these results is an expression for inner products of cumulants in RKHSs (Lemma 1).

̸

A combinatorial expression of cumulants. Classical cumulants can be defined via the moment generating function or via combinatorial sums over partitions (Appendix C.1). To generalize cumulants to RKHSs the combinatorial definition is the most efficient way. A partition π of m elements is a family of non-empty, disjoint subsets π 1 , . . . , π b of { 1 , . . . , m } whose union is the whole set; formally ⋃ b j =1 π j = { 1 , . . . , m } and π i ∩ π j = ∅ for i = j . We call b the number of blocks of the partition π and use the shorthand | π | to denote it. The set of all partitions of m is denoted with P ( m ) . To formulate our main results, it is convenient to associate with a measure γ and a partition π the so-called partition measure γ π that is given by permuting the marginals of γ .

Definition 2 (Partition measure) . Let γ be a probability measure on X 1 ×··· × X d and π ∈ P ( d ) . Define

where X π i denotes the product space ∏ j ∈ π i X j and γ | X π i is the corresponding marginal distribution of γ . We call γ π the partition measure induced by π .

We also associate with γ and a multi-index i the so-called diagonal measure γ i that is given by repeating marginals according to i .

Definition 3 (Diagonal measure) . Let γ be a probability measure on X 1 × · · · × X d and i = ( i 1 , . . . , i d ) ∈ N d . Define

where ( X 1 , . . . , X d ) ∼ γ . We call γ i the diagonal measure induced by i .

In general, the partition measure γ π and the diagonal measure are not probability measures on X 1 ×··· × X d but on spaces that are constructed by permuting or repeating X 1 , . . . , X d . Formally, γ π is a probability measure on X π 1 ×··· × X π b and γ i is a probability measure on X i 1 1 ×··· × X i d d ; thus, γ π has d coordinates and γ i has deg( i ) coordinates. These two constructions can be combined, writing γ i π for the measure ( γ i ) π which makes sense whenever π ∈ P (deg( i )) . We can now write down our generalization of cumulants.

Definition 4 (Kernelized cumulants) . Let γ be a probability measure on X 1 × · · · × X d and let ( H 1 , k 1 ) , . . . , ( H d , k d ) be RKHSs on X 1 , . . . , X d respectively. We define the kernelized cumulants

where m = deg( i ) , c π := ( -1) | π |-1 ( | π | -1)! , γ i π = ( γ i ) π and

is the reproducing kernel of H ⊗ i where H = H 1 ×··· × H d .

Def. 4 is the natural generalization of the combinatorial definition of cumulants in R d and Appendix C.2 gives an equivalent definition via a generating function analogous to (2). However, our posthoc justification that these are the "right" definitions for cumulants in an RKHS are Theorems 2 and 3 that show that these kernelized cumulants have the same powerful properties as classic cumulants in R d (Theorem 1).

Example 3.1 (Kernelized cumulants) . Let γ be a probability measure on X 1 ×X 2 , with the RKHSs ( H 1 , k 1 ) , ( H 2 , k 2 ) given. Denote the random variables K 1 = k 1 ( X 1 , · ) , K 2 = k 2 ( X 2 , · ) where ( X 1 , X 2 ) ∼ γ . Then the degree two kernelized cumulants are given as κ (2 , 0) k 1 ,k 2 ( γ ) = E [ K ⊗ 2 1 ] -E [ K 1 ] ⊗ 2 , κ (1 , 1) k 1 ,k 2 ( γ ) = E [ K 1 ⊗ K 2 ] -E [ K 1 ] ⊗ E [ K 2 ] , κ (0 , 2) k 1 ,k 2 ( γ ) = E [ K ⊗ 2 2 ] -E [ K 2 ] ⊗ 2 .

Inner products of cumulants. Computing inner products of moments is straightforward thanks to a nonlinear kernel trick, see Lemma 6 in the Appendix. For example, given two probability measures γ 1 , γ 2 with corresponding random variables ( X 1 , . . . , X d ) ∼ γ 1 , ( Y 1 , . . . , Y d ) ∼ γ 2 on X 1 ×···×X d and RKHSs ( H 1 , k 1 ) , . . . , ( H d , k d ) on X 1 , . . . , X d with bounded kernels, and H = H 1 ×··· × H d , we can express:

where µ i k 1 , ··· ,k d is defined in Def. 1, and the expectation is taken over the product measure γ 1 ⊗ γ 2 .

Example 3.2. In the particular case of d = 1 , (5) reduces to the well-known formula for the inner product of mean embeddings ⟨ µ (1) k ( γ 1 ) , µ (1) k ( γ 2 ) ⟩ H k = E γ 1 ⊗ γ 2 k ( X,Y ) .

Lemma 1 (Inner product of cumulants) . Let ( H 1 , k 1 ) , . . . , ( H d , k d ) be RKHSs with bounded kernels on X 1 , . . . , X d respectively, and let γ and η two probability measures on X 1 × · · · × X d , i = ( i 1 , . . . , i d ) ∈ N d such that deg( i ) = m . Then

Point separating kernels. In the classic MMD setting the injectivity of the mean embedding γ ↦→ E X ∼ γ [ k ( X, · )] on probability measures (known as the characteristic property of the kernel k ) is equivalent to the MMD being a metric; this property is central in applications. We formulate our theoretical results in the next section using the much weaker property of what we term 'pointseparating' which is satisfied for essentially all popular kernels.

Definition 5 (Point-separating kernel) . We call a kernel k : X × X → R point-separating if the canonical feature map Φ : x ↦→ k ( x, · ) is injective.

3.1 (Semi-)metrics for probability measures

In this section we use cumulants to characterize probability measures and show how to compute the distance between kernelized cumulants with the expected kernel trick.

as follows

Theorem 2 (Characterization of distributions with cumulants) . Let γ and η be two probability measures on X 1 ×··· × X d , ( H 1 , k 1 ) , . . . , ( H d , k d ) RKHSs on the Polish spaces X 1 , . . . , X d such that for every 1 ≤ j ≤ d k j is a bounded, continuous, point-separating kernel. Then

Moreover, the expected kernel trick applies and for i ∈ N d with deg( i ) = m , and k ⊗ i and H ⊗ i as in (4)

We recall Example 3.1 and now give examples of distances between such expressions

Example 3.3 ( m = 1 ) . Applied with m = 1 and d = 1 , (6) becomes MMD 2 k ( γ, η )

′ ′

where X,X denotes independent copies of γ and Y, Y denotes independent copies of η . Example 3.4 ( m = 2 ) . For m = 2 and d = 1 , (6) reduces to

where X,X ′ , X ′′ , X ′′′ denotes independent copies of γ and Y, Y ′ , Y ′′ , Y ′′′ denotes independent copies of η . This expression compares the variances in the RKHS instead of the means. This is an example of the kernel variance embedding defined in the next subsection.

The price for the weak assumption of a point-separating kernel is that without any stronger assumptions one does not get a metric in general, and the all-purpose way to achieve a metric is to take an infinite sum over all d i 's. If we only use the degree m = 1 term d i reduces to the well-known MMDformula which requires characteristicness to become a metric (see Example 3.3). There are two reasons why working under weaker assumptions is useful: firstly, if the underlying kernel is not characteristic this sum gives a structured way to incorporate finer information that discriminates the two distributions; an extreme case is the linear kernel k ( x, y ) = ⟨ x, y ⟩ which is point-separating, and in this case the sum reduces to the differences of classical cumulants. Secondly, under the stronger assumption of characteristicness one already has a metric after truncation at degree m = 1 (the classical MMD). However, in the finite-sample case adding higher degree terms can lead to increased power. Indeed, our experiments (Section 4) show that even just going one degree further (i.e. taking m = 2 ), can lead to more powerful tests.

3.2 A characterization of independence

Here we characterize independence in terms of kernelized cumulants.

Theorem 3 (Characterization of independence with cumulants) . Let γ be a probability measure on X 1 ×··· × X d , and ( H 1 , k 1 ) , . . . , ( H d , k d ) RKHSs on Polish spaces X 1 , . . . , X d such that for every 1 ≤ j ≤ d k j is a bounded, continuous, point-separating kernel. Then

for every i ∈ N d + . Moreover, the expected kernel trick applies in the sense that for i ∈ N d +

where m := deg( i ) , and k ⊗ i and H ⊗ i are defined as in (4) .

Applied to i = (1 , 1) , the expression (7) reduces to the classical HSIC for two components, see Example 3.5 below. But for general i this construction leads to genuine new statistics in RKHSs.

Example 3.5 (Specific case: HSIC, kernel Lancaster interaction, kernel Streitberg interaction) . If d = 2 there is only one order 2 index in N d + , namely i = (1 , 1) ; in this case (7) reduces to the classical HSIC equation

where ( X,Y ) and ( X ′ , Y ′ ) are independent copies of the same random variable following γ . More generally, with i = 1 d one gets the kernel Streitberg interaction (Streitberg, 1990; Sejdinovic et al., 2013a; Liu et al., 2023), and specifically the kernel Lancaster interaction (Sejdinovic et al., 2013a) for d ∈ { 2 , 3 } ; the latter reduces to HSIC for two random variables ( d = 2 ).

3.3 Finite-sample statistics

To apply Theorem 2 and Theorem 3 in practice, one needs to estimate expressions such as E k ⊗ i (( X 1 , . . . , X m ) , ( Y 1 , . . . , Y m )) . One could use classical estimators such as U-statistic (Van der Waart, 2000) which lead to unbiased estimators. However, we follow Gretton et al. (2008) and use a V-statistic which is biased but conceptually simpler, easier, and efficient to compute. We note that the estimators presented here all have quadratic complexity like MMD and HSIC, see Appendix E.

Atwo-sample test for non-characteristic feature maps. If k is characteristic then MMD k ( γ, η ) = 0 exactly when γ = η , but we can still increase testing power by considering the distance between the kernel variance and skewness embeddings, which leads us to use our semi-metrics d (2) ( γ, η ) and d (3) ( γ, η ) as defined in (6). An efficient estimator for d (3) is given in detail in Appendix E; we provide the full expression for d (2) here.

Lemma 2 ( d (2) estimation, see (6)) . The V-statistic for d (2) ( γ, η ) = ∥ κ (2) k ( γ ) -κ (2) k ( η ) ∥ 2 H (1 , 1) is

where Tr denotes trace, ( x n ) N n =1 i . i . d . ∼ γ , ( y m ) M m =1 i . i . d . ∼ η , K x = [ k ( x i , x j )] N i,j =1 ∈ R N × N , K y = [ k ( y i , y j )] M i,j =1 ∈ R M × M , K x,y = [ k ( x i , y j )] N,M i,j =1 ∈ R N × M , J n = I n -1 n 1 n 1 ⊤ n ∈ R n × n , with 1 n = (1 , . . . , 1) ∈ R n .

A kernel independence test. By Theorem 3, if γ = γ | X 1 ⊗ γ | X 2 , then κ (2 , 1) ( γ ) = 0 and κ (1 , 2) ( γ ) = 0 . Wemay compute the magnitude of either κ (2 , 1) ( γ ) or κ (1 , 2) ( γ ) - we will refer to these quantities as cross skewness independence criterion (CSIC). Note that these criteria are asymmetric. When d = 2 we have a probability measure γ on X 1 ×X 2 and two kernels k : X 2 1 → R , ℓ : X 2 2 → R . Assume that we have samples ( x i , y i ) N i =1 and use the shorthand notation K = K x , L = L y (similarly to Lemma 2) and H = H N = 1 N 1 N 1 ⊤ N ∈ R N × N . Denote by ◦ the Hadamard product and ⟨·⟩ the sum over all elements of a matrix. Then one can derive the following CSIC estimator.(Note that matrix multiplication takes precedence over the Hadamard product.)

Lemma 3 (CSIC estimation) . The V-statistic for ∥ κ (1 , 2) k,ℓ ( γ ) ∥ 2 H ⊗ 1 k ⊗H ⊗ 2 ℓ is

Remark (computational complexity w.r.t. degree m ). We saw that the computational complexity of the cumulant based measures is quadratic w.r.t. the sample size. Let B m = | P ( m ) | be the m -th

Bell number, in other words the number of elements in P ( m ) . The Bell numbers follow a recursion: B m +1 = | P ( m +1) | = ∑ m k =0 ( m k ) B k , with the first elements of the sequence being B 0 = B 1 = 1 , B 2 = 2 , B 3 = 5 , B 4 = 15 , B 5 = 52 . By (6)-(7), in the worst case the number of operations to compute d i ( γ, η ) or ∥ κ i k 1 ,...,k d ( γ ) ∥ 2 H ⊗ i ( m = deg ( i ) ) is proportional to B 2 m (it equals to 3 B 2 m and to B 2 m , respectively). Though asymptotically B m grows quickly (de Bruijn, 1981; Lovász, 1993), for reasonably small degrees the computation is still manageable. In addition, merging various terms in the estimator can often be carried out, which leads to computational saving. For instance, the estimator of d (2) (see Lemma 2, Example E.1), CSIC (Lemma 3, Example E.2) and d (3) (Example E.3) consists of only 3 , 11 and 10 + 2 × 7 = 24 terms compared to the predicted worst-case setting of 3 B 2 2 = 12 , B 2 3 = 25 , and 3 B 2 3 = 75 terms, respectively. On a practical side, we found that using m ∈ { 2 , 3 } is a good compromise between gain in sample efficiency and ease of implementation.

4 Experiments

In this section, we demonstrate the efficiency of the proposed kernel cumulants in two-sample and independence testing. 4

̸

Two-sample test: Given N -N samples from two probability measures γ and η on a space X , the goal was to test the null hypothesis H 0 : γ = η against the alternative H 1 : γ = η . The compared test statistics ( S ) were MMD, d (2) , and d (3) .

̸

Independence test: Given N paired samples from a probability measure γ on a product space X 1 × X 2 , the aim was to the test the null hypothesis H 0 : γ = γ 1 ⊗ γ 2 against the alternative H 1 : γ = γ 1 ⊗ γ 2 . The compared test statistics ( S ) were HSIC and CSIC.

In our experiments H 1 held, and the estimated power of the tests is reported. Permutation test was applied to approximate the null distribution and its 0 . 95 -quantile (which corresponds to the level choice α = 0 . 05 ): We first computed our test statistic S using the given samples ( S 0 = S ), and then permuted the samples 100 times. If S 0 was in a high percentile ( ≥ 95% in our case) of the resulting distribution of S under the permutations, we rejected the null. We repeated these experiments 100 times to estimate the power of the test. This procedure was in turn repeated 5 times and the 5 samples are plotted as a box plot along with a line plot showing the mean against the number of samples

∥

x

-

y

∥

2

( N ) used. All experiments were performed using the rbf-kernel rbf σ ( x , y ) = e -2 2 σ 2 , where the parameter σ is called the bandwidth . We performed all experiments for every bandwidth of the form σ = a 10 b where a = 1 , 2 . 5 , 5 , 7 . 5 and b = -5 , -4 , -3 , -2 , -1 , 0 and the optimal value across the bandwidths was chosen for each method and sample size. The experiments were carried out on a laptop with an i7 CPU and 16GBs of RAM.

4.1 Synthetic data

For synthetic data we designed two experiments.

2-sample test: We compared a uniform distribution with a mixture of two uniforms.

Independence test: We considered the joint measure of a uniform and a correlated χ 2 random variable. We also use this same benchmark to compare the efficiency of classical and kernelized cumulants in Appendix D.

Comparing a uniform with a mixture of uniforms. Even for simpler distributions like mixtures of uniform distributions it can be hard to pick up higher-order features, and d (2) can outperform MMDeven when provided with a moderate number of samples. Here we compared one uniform distribution U [ -1 , 1] with an equal mixture of U [0 . 35 , 0 . 778] and U [ -0 . 35 , -0 . 778] . The endpoints in the mixture were chosen to match the first three moments of U [ -1 , 1] . The number of samples used ranged from 5 to 50 , and the results are summarized in Fig. 1. One can see that with d (2) the power approaches 100% much faster than with using MMD.

6

12

18

24

30

36

42

48

54

60

N

0

20

40

60

80

100

Test power (%)

Independence between a uniform and a χ 2 . Let X ∼ U [0 , 1] and Z ∼ N (0 , 1) be independent of X . Denote by Φ the c.d.f. of a standard normal distribution and define Y p to be a mixture with weight p at Φ -1 ( X ) and weight 1 -p at Z so that Y 0 = Z and Y 1 = Φ -1 ( X ) . We test for independence for p = 0 . 5 between Y 2 p -which will be χ 2 distributed with 1 degree of freedomand X . As the statistical dependence of Y 2 p and X is more complicated than a simple correlation we expect that higher-order features of the data will help in the independence testing. The number of samples used ranged from 6 to 60 , with the results summarized in Fig. 2. One can see that CSIC supersedes HSIC for every sample size, and the difference is more pronounced for smaller ones.

4.2 Real-world data

We demonstrate the efficiency of the kernelized cumulants on real-world data. We designed two experiments.

2-sample test: Here the goal was to test if environmental data in two different seasons, and traffic data at different speeds can be distinguished.

independence test: The aim was to test if two distributions describing traffic flow and other traffic factors are independent.

To improve performance of both test statistics, all features are standardized to lie between 0 and 1 .

̸

Seoul bicycle data. The Seoul bicycle data set (E et al., 2020) consists of environmental data along with the number of bicycle rentals. The environmental data consists of 9 numerical values, 1 categorical value (season), and two binary values. We compare the distribution of the environmental data in the winter and the fall, as we expect these distributions to be different. Concretely, we do 2-sample testing on two measures γ, η on R 11 , and assume that γ = η where γ is the distribution of the environmental data in winter, and η is that of the data in the fall. Permutation testing was performed for N between 4 and 44 , with results summarized in Fig. 3. As it can be observed that d (2) outperforms MMD in terms of test power. For the Type I error, i.e. the probability of falsely rejecting the null hypothesis (comparing winter data with itself), it hovers between 5 -10% for both statistics,

4

8

12

16

20

24

28

32

36

40

44

N

0

20

40

60

80

100

Test power (%)

5

10

15

20

25

30

35

40

45

50

N

0

20

40

60

80

100

Test power (%)

4

8

12

16

20

24

28

32

36

40

N

0

20

40

60

80

100

Test power (%)

N

which is admittedly slightly higher than the desired 5% due to the small sample size, but very similar for both statistics; for further details, the reader is referred to Fig. 8 in Appendix D.

Brazilian traffic data. We used the Sao Paulo traffic benchmark (Ferreira, 2016) to perform independence testing. The dataset consists of 16 different integer-valued statistics about the hourly traffic in Sao Paulo such as blockages, fires and other reasons that might hold up traffic. This is combined with a number that describes the slowness of traffic at the given hour; so X 1 = R 16 , X 2 = R . One expects a strong dependence between the two sets-or equivalently, for the null hypothesis to be false-and for the statistics are heavily skewed towards 0 as it is naturally sparse. For independence testing we performed permutation testing for N between 4 and 40 . The resulting test powers are summarized in Fig. 4. As it can be seen, HSIC and CSIC performs similarly for very low sample sizes, but for anything else CSIC is the favorable statistic in terms of test power. For two-sample testing, we sampled N between 5 and 50 and compared the distribution of slow moving traffic with the fast moving traffic. The results are summarized in Fig. 5. It is clear that d (3) performs similarly to MMD in terms of test power for very small sample sizes, but significantly better for larger ones.

5 Conclusion

We defined cumulants for random variables in RKHSs by extending the algebraic characterization of cumulants on R d . This construction results in a structured description of the law of random variables that goes beyond the classic kernelized mean and covariance. A kernel trick allows us to compute the resulting kernelized cumulants. We applied our theoretical results to two-sample and independence testing; although kernelized mean and covariance are sufficient for this task, the higherorder kernelized cumulants have the potential to increase the test power and to relax the assumptions on the kernel. Our experiments on real and synthetic data show that kernelized cumulants can indeed lead to significant improvement of the test power. A disadvantage of these higher-order statistics is that their theoretical analysis requires more mathematical machinery although we emphasize that the resulting estimators are simple V-statistics.

References

Arcones, M. A. and Giné, E. (1992). On the bootstrap of U and V statistics. Annals of Statistics , 20:655-674.

Aronszajn, N. (1950). Theory of reproducing kernels. Transactions of the American Mathematical Society , 68:337-404.

Baringhaus, L. and Franz, C. (2004). On a new multivariate two-sample test. Journal of Multivariate Analysis , 88:190-206.

Berlinet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics . Kluwer.

Test power (%)

100

80

60

40

20

0

5

10

15

20

25

30

35

40

45

50

Bogachev, V. I. (2007). Measure theory. Vol. I, II . Springer.

Bonnier, P. and Oberhauser, H. (2020). Signature cumulants, ordered partitions, and independence of stochastic processes. Bernoulli , 26(4):2727-2757.

Chevyrev, I. and Oberhauser, H. (2022). Signature moments to characterize laws of stochastic processes. Journal of Machine Learning Research , 23(176):1-42.

Chung, E. and Romano, J. P. (2013). Exact and asymptotically robust permutation tests. Annals of Statistics , 41(2):484-507.

de Bruijn, N. G. (1981). Asymptotic Methods in Analysis . Dover.

Dudley, R. (2004). Real Analysis and Probability . Cambridge University Press.

E, S. V., Park, J., and Cho, Y. (2020). Using data mining techniques for bike sharing demand prediction in metropolitan city. Computer Communications , 153:353-366.

Ferreira, R. P. (2016). Combination of artificial intelligence techniques for prediction the behavior of urban vehicular traffic in the city of São Paulo. In Anais do 10. Congresso Brasileiro de Inteligência Computacional , pages 1-5.

Fukumizu, K., Gretton, A., Sun, X., and Schölkopf, B. (2008). Kernel measures of conditional dependence. In Advances in Neural Information Processing Systems (NIPS) , pages 498-496.

Gärtner, T. (2003). A survey of kernels for structured data. ACM SIGKDD Explorations Newsletter , 5(1):49-58.

Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., and Smola, A. (2012). A kernel two-sample test. Journal of Machine Learning Research , 13(25):723-773.

Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. (2005). Measuring statistical dependence with Hilbert-Schmidt norms. In Algorithmic Learning Theory (ALT) , pages 63-78.

Gretton, A., Fukumizu, K., Teo, C. H., Song, L., Schölkopf, B., and Smola, A. J. (2008). A kernel statistical test of independence. In Advances in Neural Information Processing Systems (NIPS) , pages 585-592.

Hagrass, O., Sriperumbudur, B. K., and Li, B. (2022). Spectral regularized kernel two-sample tests. Technical report. ( https://arxiv.org/abs/2212.09201 ).

Hagrass, O., Sriperumbudur, B. K., and Li, B. (2023). Spectral regularized kernel goodness-of-fit tests. Technical report. ( https://arxiv.org/abs/2308.04561 ).

Jammalamadaka, S. R., Rao, T. S., and Terdik, G. (2006). Higher order cumulants of random vectors and applications to statistical inference and time series. Indian Journal of Statistics , 68(2):326-356.

Király, F. J. and Oberhauser, H. (2019). Kernels for sequentially ordered data. Journal of Machine Learning Research , 20.

Klebanov, L. (2005). N-Distances and Their Applications . Charles University, Prague.

Kriege, N. M., Johansson, F. D., and Morris, C. (2020). A survey on graph kernels. Applied Network Science , 5(1).

Lang, S. (2002). Algebra . Springer.

Liu, Z., Peach, R. L., Mediano, P. A., and Barahona, M. (2023). Interaction measures, partition lattices and kernel tests for high-order interactions. Technical report. ( https://arxiv.org/ abs/2306.00904 ).

Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., and Watkins, C. (2002). Text classification using string kernels. Journal of machine learning research , 2:419-444.

Lovász, L. (1993). Combinatorial Problems and Exercise . 2nd ed. Amsterdam, Netherlands: North-Holland.

Lyons, R. (2013). Distance covariance in metric spaces. The Annals of Probability , 41:3284-3305.

Makigusa, N. (2020). Two-sample test based on maximum variance discrepancy. Technical report. ( https://arxiv.org/abs/2012.00980 ).

McCullagh, P. (2018). Tensor Methods in Statistics . Courier Dover Publications.

Muandet, K., Fukumizu, K., Sriperumbudur, B., and Schölkopf, B. (2017). Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning , 10(1-2):1-141.

Müller, A. (1997). Integral probability metrics and their generating classes of functions. Advances in Applied Probability , 29:429-443.

Pfister, N., Bühlmann, P., Schölkopf, B., and Peters, J. (2018). Kernel-based tests for joint independence. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(1):5-31.

Quadrianto, N., Song, L., and Smola, A. (2009). Kernelized sorting. In Advances in Neural Information Processing Systems (NIPS) , pages 1289-1296.

Rudin, W. (1953). Principles of mathematical analysis . McGraw-Hill Book Company, Inc., New York-Toronto-London.

Saitoh, S. and Sawano, Y. (2016). Theory of Reproducing Kernels and Applications . Springer Singapore.

Schölkopf, B. and Smola, A. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond . MIT Press.

Sejdinovic, D., Gretton, A., and Bergsma, W. (2013a). A kernel test for three-variable interactions. In Advances in Neural Information Processing Systems (NIPS) , pages 1124-1132.

Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. (2013b). Equivalence of distancebased and RKHS-based statistics in hypothesis testing. Annals of Statistics , 41:2263-2291.

Serfling, R. (1980). Approximation Theorems of Mathematical Statistics . Wiley Series in Probability and Statistics.

Smola, A., Gretton, A., Song, L., and Schölkopf, B. (2007). A Hilbert space embedding for distributions. In Algorithmic Learning Theory (ALT) , pages 13-31.

Speed, T. P. (1983). Cumulants and partition lattices. Australian Journal of Statistics , 25(2):378-388.

Speed, T. P. (1984). Cumulants and partition lattices II. Australian Journal of Statistics , 4(1):34-53.

Sriperumbudur, B., Gretton, A., Fukumizu, K., Schölkopf, B., and Lanckriet, G. (2010). Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research , 11:1517-1561.

Steinwart, I. and Christmann, A. (2008). Support Vector Machines . Springer.

Streitberg, B. (1990). Lancaster interactions revisited. Annals of Statistics , 18(4):1878-1885.

Szabó, Z. and Sriperumbudur, B. K. (2018). Characteristic and universal tensor product kernels. Journal of Machine Learning Research , 18(233):1-29.

Székely, G. and Rizzo, M. (2004). Testing for equal distributions in high dimension. InterStat , 5:1249-1272.

Székely, G. and Rizzo, M. (2005). A new test for multivariate normality. Journal of Multivariate Analysis , 93:58-80.

Székely, G. J. and Rizzo, M. L. (2009). Brownian distance covariance. The Annals of Applied Statistics , 3:1236-1265.

Székely, G. J., Rizzo, M. L., and Bakirov, N. K. (2007). Measuring and testing dependence by correlation of distances. The Annals of Statistics , 35:2769-2794.

Tolstikhin, I., Sriperumbudur, B., and Schölkopf, B. (2016). Minimax estimation of maximal mean discrepancy with radial kernels. In Advances in Neural Information Processing Systems (NIPS) , pages 1930-1938.

Van der Waart, A. A. (2000). Asymptotic Statistics . Cambridge Series in Statistical and Probabilistic Mathematics.

Willard, S. (1970). General Topology . Addison-Wesley.

Zinger, A., Kakosyan, A., and Klebanov, L. (1992). A characterization of distributions by mean values of statistics and certain probabilistic metrics. Journal of Soviet Mathematics .

Zolotarev, V. (1983). Probability metrics. Theory of Probability and its Applications , 28:278-302.

These appendices provide additional background and elaborate on some of the finer points in the main text. In Appendix A we illustrate that cumulants have typically lower variance estimators compared to moments. Technical background on tensor products and tensor sums of Hilbert spaces, and on tensor algebras is provided in Appendix B. We present our proofs in Appendix C. In Appendix D additional details on our numerical experiments are provided. Our V-statistic based estimators are detailed in Appendix E.

A Moments and cumulants

Already for real-valued random variable X , moments have well-known drawbacks that make cumulants often preferable as statistics. For a detailed introduction to the use of cumulants in statistics we refer to McCullagh (2018). Here we just mention that

the moment generating function f ( t ) = E [ e tX ] = ∑ m µ m t m /m ! describes the law of X with sequence ( µ m ) of moments µ m = E [ X m ] ∈ R . However, since the function t ↦→ f ( t ) is the expectation of an exponential, one would often expect that f is also "exponential in t ", hence g ( t ) = log f ( t ) = ∑ m κ m t m m ! should be simpler to describe as a power series. For example, for a Gaussian f ( t ) = e t E ( X )+ t 2 2 V ar ( X ) and while µ m can be in this case explicitly calculated and uneven moments vanish, the m -moments are fairly complicated compared to the power series expansion of g ( t ) = κ 1 t + κ 2 t 2 2 which just consists of κ 1 (mean) and κ 2 (variance).

In the moment sequence µ m , lower moments can dominate higher moments. Hence, a natural idea to compensate for these "different scales" is to systematically subtract lower moments from higher moments. As mentioned in the introduction, this is in particular troublesome if finite samples are available. Even in dimension d = 1 the second moment is dominated by the squared mean, that is for a real-valued random variable X ∼ γ

where Var( X ) := E [( X -µ 1 ( γ )) 2 ] . It is well known that the minimum variance unbiased estimators for the variance are more efficient than that for the second moment: denoting them by ̂ µ 2 and ̂ κ respectively, one can show (Bonnier and Oberhauser, 2020) that given N samples from X , the following holds

This means that when X has a large mean, it is more efficient to estimate its variance than its second moment since the last term in the above expression dominates. Hence, the variance Var( X ) is typically a much more sensible second-order statistic than µ 2 ( γ ) . However, we emphasize that there are many other reasons why cumulants can have better properties as estimators

Cumulants characterize laws and the independence of two random variables manifests itself simply as vanishing of cross-cumulants. In view of the above item 2, this means for example that testing independence can be preferable in terms of vanishing cumulants rather than testing if moments factor E [ X m Y n ] = E [ X m ] E [ X n ] , and similarly for testing if distributions are the same.

The caveat to the above points is that it is not true that cumulants are always preferable. For example, there are distributions for which (a) the moment generating function is not naturally exponential in t , (b) lower moments do not dominate higher moments, (c) consequently independence or two-sample testing become worse with cumulants. While one can write down conditions under which for example, the variance of the kernelized cumulants is lower, the use of cumulants among statisticians is to simply regard cumulants as arising from natural motivations which leads to another estimator in their toolbox.

The main idea of our paper is simply that for the same reasons that cumulants can turn out to be powerful for real or vector-valued random variables, cumulants of RKHS-valued random variables are a natural choice of statistics. The situation is more complicated since it requires formalizing momentand cumulant-generating functions in RKHS but ultimately a kernel trick allows for circumventing the computational bottleneck of working in infinite dimensions and leads to computable estimators for independence and two-sample testing.

Further, we note that although cumulants are classic for vector-valued data, there seems to be not much work done about extending their properties to general structured data. Our kernelized cumulants apply to any set X where a kernel is given. This includes many practically relevant examples such as strings (Lodhi et al., 2002), graphs (Kriege et al., 2020), or general sequentially ordered data (Király and Oberhauser, 2019; Chevyrev and Oberhauser, 2022); a survey of kernels for structured data is provided by Gärtner (2003).

B Technical background

In Section B.1 the tensor products ( ⊗ d j =1 H j ) and direct sums of Hilbert spaces ( ⊕ i ∈ I H i ) are recalled. Section B.2 is about tensor algebras over Hilbert spaces ( ∏ m ≥ 0 H ⊗ m ).

B.1 Tensor products and direct sums of Banach and Hilbert spaces

Tensor products of Hilbert spaces. For Hilbert spaces H , . . . , H d and ( h 1 , . . . , h d ) ∈ H 1 ×···× H d , the multi-linear operator h 1 ⊗··· ⊗ h d ∈ H 1 ⊗··· ⊗ H d is defined as

for all ( f 1 , . . . , f d ) ∈ H 1 ×··· × H d . By extending the inner product

to finite linear combinations of a 1 ⊗··· ⊗ a d -s

by linearity, and taking the topological completion one arrives at H 1 ⊗··· ⊗ H d . Specifically, if ( H 1 , k 1 ) , . . . , ( H d , k d ) are RKHSs, then so is H 1 ⊗ · · · ⊗ H d = H ⊗ d j =1 k j (Berlinet and ThomasAgnan, 2004, Theorem 13) with the tensor product kernel

where ( x 1 , . . . , x d ) , ( x ′ 1 , . . . , x ′ d ) ∈ X 1 ×··· × X d .

Tensor products of Banach spaces. For Banach spaces B 1 , . . . B d , the construction of B 1 ⊗···⊗B d is a little more involved (Lang, 2002) as one cannot rely on an inner product.

Direct sums of Hilbert and Banach spaces. Let ( H i ) i ∈ I be Hilbert or Banach spaces where I is some index set. The direct sum of H i -s- written as ⊕ i ∈ I H i -consists of ordered tuples h = ( h i ) i ∈ I such that h i ∈ H i for all i ∈ I and h i = 0 for all but a finite number of i ∈ I . Operations (addition, scalar multiplication) are performed coordinate-wise, and the inner product of a, b ∈ ⊕ i ∈ I H i is defined as ⟨ a, b ⟩ ⊕ i ∈ I H i = ∑ i ∈ I a i b i .

B.2 Tensor algebras

The tensor algebra T j over a Hilbert space H j is defined as the topological completion of the space

Note that it can equivalently be defined as the subset of ( h 0 , h 1 , h 2 , . . . ) ∈ ∏ m ≥ 0 H ⊗ m j such that ∑ m ≥ 0 ∥ h m ∥ 2 H ⊗ j m < ∞ , and as such it is a Hilbert space with norm

T j is also an algebra, endowed with the tensor product over H j as its product. For a = ( a 0 , a 1 , a 2 , a 2 . . . ) , b = ( b 0 , b 1 , b 2 , b 2 . . . ) ∈ T j , their product can be written down in coordinates as

For a sequence H 1 , . . . , H d of Hilbert spaces, we define

where T j = ∏ m ≥ 0 H ⊗ m j ( j = 1 , . . . , d ). Let H = H 1 ×··· × H d , and recall that given a tuple of integers i = ( i 1 , . . . , i d ) ∈ N d we define H ⊗ i : = H ⊗ i 1 1 ⊗··· ⊗ H ⊗ i d d . This allows us to write down a multi-grading for T as

Note that this gives credence to us using multi-indices i ∈ N d to describe elements of the tensor algebra, as the multi-indices form its multi-grading.

Furthermore, T is a multi-graded algebra when endowed with the (linear extension of the) following multiplication defined on the components of T

so that for a = ( a i ) i ∈ N d , b = ( b i ) i ∈ N d ∈ T , their product can be written down as

where addition of tuples i 1 , i 2 ∈ N d is defined as i 1 + i 2 = ( i 1 1 + i 2 1 , . . . , i 1 d + i 2 d ) . With the degree of a tuple defined as deg( i ) = i 1 + · · · + i d , T is also a graded algebra, with the grading written down as

so that if one multiplies two elements together, the degree of their product is the sum of their degree.

Finally we note that T is a unital algebra and the unit has the explicit form

i.e. the element consisting of only a 1 at degree 0.

C Proofs

This section is dedicated to proofs. The equivalence between the combinatorial expressions of cumulants and the definition via a moment generating function is proved in Section C.2. The derivation of our main results (Theorem 2 and Theorem 3) are detailed in Section C.3.

C.1 Equivalent definitions of cumulants in R d

Here we introduce a classical definition of cumulants via a moment generating function and its equivalence to the combinatorial expressions. If X = ( X 1 , . . . , X d ) is an R d -valued random variable distributed according to X ∼ γ , then recall that

for i = ( i 1 , . . . , i d ) ∈ N d .

The following definition of the cumulants κ i ( γ ) of γ are equivalent

where θ = ( θ 1 , . . . , θ d ) ∈ R d , c π = ( -1) | π | ( | π |-1)! . The equivalence between these two definitions of cumulants via a generating function and via their combinatorial definition, is classical (McCullagh, 2018) even if our notation here is non-standard in the classical case. This equivalence is also at the heart of many proofs about properties of cumulants since some properties are easier to prove via one or the other definition.

C.2 Equivalent definitions of cumulants in RKHS

In the main text, we defined cumulants in RKHS by mimicking the combinatorial definition of cumulants in R d . It is natural and useful to also have the analogous definition via a "generating function" for RKHS-valued random variables. However, to generalize the definition via the logarithm of the moment generating function to random variables in RKHS, requires to define a logarithm for tensor series of moments. In this part, we show that this can be done and that indeed the two definitions are equivalent.

We use the shorthand κ ( γ ) := κ k 1 ,...,k d ( γ ) , µ ( γ ) := µ k 1 ,...,k d ( γ ) , and we overload the notation ( X 1 , . . . , X d ) with ( k 1 ( · , X 1 ) , . . . , k d ( · , X d )) . With this notation, we show that given coordinates i ∈ N d , one may express the generalized cumulant κ i ( γ ) as either a combinatorial sum over moments indexed by partitions, or by using the cumulant generating function.

More specifically, we show that the generalized cumulant of a probability measure γ on H 1 ×···×H d defined as

where c π = ( -1) | π |-1 ( | π | -1)! can also be expressed as coordinates in the tensorized logarithm of the moment series. Motivated by the Taylor series expansion of the classic logarithm, we define

where ⋆ denotes the product as defined in (9) and for t ∈ T , t ⋆n is defined as

or coordinate-wise ( t ⋆n ) i = ∑ i 1 + ··· + i n = i t i 1 ⋆ · · · ⋆ t i n for i ∈ N d . Note that unlike the classical logarithm log : R + → R , the tensorized logarithm is defined on the whole space as a formal expression. This can be summarized in the following lemma:

Lemma 4.

where 1 m = (1 , . . . , 1) ∈ N m .

By iterating (10) we can express (11) as

and our goal is to express this as a sum over partitions. We will use the notation [ n ] = { 1 , . . . , n } . We can achieve our goal in two parts:

Show that for a fixed i ∈ N d with deg( i ) = m we can express (11) as a sum over all surjective functions from [ m ] to [ j ] .

Show that this sum over functions reduces to a sum over partitions.

Part 1. Note that given i 1 + · · · + i j = 1 m we may define h : [ m ] → [ j ] by the relation ( i h ( n ) ) n = 1 , that is, we take h ( n ) to be the index c for which the multi-index i c is 1 at n . Note that this function is necessarily surjective since the sum is taken over non-zero multi-indices. Equivalently, for any surjective function h : [ m ] → [ j ] we may define multi-indices by setting

Note that any such multi-index will be non-zero since the function is assumed to be surjective. With this identification we can write

Part 2. Recall that given a function h : [ m ] → [ j ] we can associate it to its corresponding partition π h ∈ P ( m ) by considering the set { h -1 (1) , . . . , h -1 ( j ) } , and there are exactly j ! different functions corresponding to a given partition, which are given by re-ordering the values 1 , . . . , j . This reordering of the blocks does not change the summands since the marginals of the partition measure are always copies of each other and hence self-commute, hence a product of moments like µ i h -1 (1) ( γ i ) ⋆ · · · ⋆ µ i h -1 ( j ) ( γ i ) can always be written as µ i ( γ i π h ) , the i -th coordinate of the moment sequence of the partition measure γ i π h . With this in mind we can write

From this it immediately follows that for two probability measures γ, η we can write

Lemma 1 then follows from the definition of the tensor products.

C.3 Proof of Theorem 2 and Theorem 3

In this section we present the proofs of Theorem 2 and Theorem 3. We do this in a slightly more abstract setting where the feature maps take values in Banach spaces for clarity, until the end when we again restrict our attention to RKHSs. We start out by showing that polynomial functions of the feature maps characterize measures (Lemma 5). From this result we show that cumulants have the same property (Theorem 4), and lastly that this also holds when working directly with the kernels (Proposition 1).

A monomial on separable Banach spaces B 1 , . . . , B d is any expression of the form

̸

for some ( i 1 , . . . , i d ) ∈ N d , where f i j ∈ B ⋆ i are elements of the dual space B ⋆ i and x i ∈ B i . 5 Finite linear combinations of monomials are called the polynomials . Recall that a set of functions F on a set S is said to separate the points of S if for every x = y ∈ S there exists f ∈ F such that f ( x ) = f ( y ) . Lemma 5 (Polynomial functions of feature maps characterize probability measures) . Let X 1 , . . . , X d be Polish spaces, B 1 . . . , B d separable Banach spaces and φ i : X i →B i be continuous, bounded, and injective functions. Then the set of functions on the Borel probability measures P ( ∏ d i =1 X i ) of ∏ d i =1 X i

where p ranges over all polynomials, separates the points of P ( ∏ d i =1 X i ) .

Proof. We first show that the pushforward map

is injective. This is done in two parts, first we show that every Borel measure on ∏ d i =1 X i is a Radon measure, then we show that the pushforward map is injective on Radon measures. To see the first part, note that since X 1 , . . . , X d are Polish spaces, so is their product space ∏ d i =1 X i (Dudley 2004, Theorem 2.5.7; Willard 1970, Theorem 16.4c), and since Borel measures on Polish spaces are Radon measures (Bogachev, 2007, Theorem 7.1.7), any γ ∈ P ( ∏ d i =1 X i ) must be a Radon measure.

For the second part, note that

is a norm bounded, continuous injection. Since ∏ d i =1 B i is a Hausdorff space, ∏ d i =1 φ i is a homeomorphism on compacts since continuous injections into Hausdorff spaces are homeomorphisms on compacts (Rudin, 1953, Theorem 4.17). Let µ, ν ∈ P ( ∏ d i =1 X i ) be two Radon measures such that their pushforwards are the same ∏ d i =1 φ i ( µ ) = ∏ d i =1 φ i ( ν ) , then for any compact C ⊆ ∏ d i =1 X i we have µ ( C ) = ν ( C ) as ∏ d i =1 φ i : C → ∏ d i =1 φ i ( C ) is a homeomorphism. Since Radon measures are characterized by their values on compacts, this implies that µ = ν . Hence the pushforward map is injective.

Denote by K the image of ∏ d i =1 X i under the mapping ∏ d i =1 φ i in ∏ d i =1 B i . Note that K is a bounded Polish space. It is enough to show that the polynomials separate the points of P ( K ) . To see this, note that the polynomials form an algebra of continuous functions that separate the points of ∏ d i =1 B i , and when restricted to K they are bounded, since K is norm bounded. Since K is Polish, any Borel measure is Radon, and we can apply the Stone-Weierstrass theorem for Radon measures (Bogachev, 2007, Exercise 7.14.79) to get the assertion.

In what follows we will use the following index notation for linear functionals. Fix some tuple i = ( i 1 , . . . , i d ) ∈ N d with deg( i ) = m . Given separable Banach spaces B 1 . . . , B d we use the notation

̸

and given an element x = ( x 1 , . . . , x d ) ∈ ∏ d i =1 B i we write x i : = x ⊗ i 1 1 ⊗ · · · ⊗ x ⊗ i d d so that x i ∈ B ⊗ i . If we have functions ( φ i ) d i =1 such that φ i : X i →B i on some Polish spaces X 1 , . . . , X d , then we write

Given a collection of linear functionals F ∈ ∏ d j =1 ( B ⋆ j ) i j such that F = ( f 1 , . . . , f d ) we write

Note the following trick: the monomials on ∏ d i =1 B i are exactly functions of the form

for F = ( f 1 , . . . , f d ) , this will be used in the proofs. We can now restate and prove the our theorem. Note that the cumulants here are defined like in Definition 4 which is a sensible definition even if the feature maps are not associated to kernels.

Theorem 4 (Generalization of Theorem 2 and Theorem 3) . Let X 1 , . . . , X d be Polish spaces and φ i : X i →B i be continuous, bounded and injective feature maps into separable Banach spaces B i for i = 1 , . . . d . Let γ and η be probability measures on X 1 ×··· × X d . Then

γ = η if and only if κ ( γ ) = κ ( η ) .

γ = ⊗ d i =1 γ | X i if and only if the cross cumulants vanish, that is κ i ( γ ) = 0 for all i ∈ N d + .

Proof.

Item 2: We want to show that the cross cumulants vanish if and only if γ = ⊗ d i =1 γ | X i . By Lemma 5 it is enough to show that

for any monomial function p . Let us take linear functionals F = ( f 1 , . . . , f d ) and note that

which is the classical cumulant of the vector-valued random variable

where ( X 1 , . . . , X d ) ∼ γ . Hence by classical results (Speed, 1983), all cross cumulants of ( ( f 1 ◦ φ 1 )( X 1 ) , . . . , ( f d ◦ φ d )( X d ) ) vanish if and only if the cross moments split, that is to say

for any monomial p on R d . Since f 1 , . . . , f d were arbitrary this holds for all monomials, which shows the assertion.

Item 1: By assumption κ i ( γ ) = κ i ( η ) for every i ∈ N d ; this implies that E γ p ( φ 1 , . . . , φ d ) = E η p ( φ 1 , . . . , φ d ) for any polynomial p , so we can apply Lemma 5.

Proposition 1 (Theorem 2 and Theorem 3) . Let X 1 , . . . , X d be Polish spaces and k i : X 2 i → R be a collection of bounded, continuous, point-separating kernels. Let γ and η be be probability measures on X 1 ×··· × X d . Then

Proof. We reduce the proof to the checking of the conditions of Theorem 4. Let φ i denote the canonical feature map of the kernel k i , and let B i : = H k i be the RKHS associated to k i ( i ∈ { 1 , . . . , d } ) . For all i ∈ { 1 , . . . , d } , φ i is (i) bounded by the boundedness of k i since ∥ φ i ( x ) ∥ 2 H k i = k i ( x, x ) ≤ sup x ∈X i | k i ( x, x ) | < ∞ , (ii) continuous by the continuity of k i (Steinwart and Christmann, 2008, Lemma 4.29), (iii) injective by the point-separating property of k i . The separability of H k i follows (Steinwart and Christmann, 2008, Lemma 4.33) from the separability of X i and the continuity of k i ( i ∈ { 1 , . . . , d } ). Note: Details on the expected kernel trick part of Theorem 2 and Theorem 3 are provided in Section E.

D Additional experiments and details

Here we give additional details on the experiments that were performed, and discuss some further experiments that did not fit into the main text.

Background on permutation testing. Permutation testing works by bootstrapping the distribution of a test statistic under the null hypothesis. This allows the user to estimate confidence intervals under the null, which is a powerful all-purpose way of doing so when analytic expressions are unavailable. As an example, assume we have two probability measures γ, η on X with i.i.d. samples x 1 , . . . , x N ∼ γ, y 1 , . . . , y N ∼ η . If the null hypothesis is that γ = η then we may set

so that for any permutation σ on 2 N elements, we get two different set of of i.i.d. samples from γ = η by using the empirical measures

and for any statistic S : P ( X ) 2 → R , we may estimate S ( γ, η ) under the null by sampling from S (˜ γ σ , ˜ η σ ) . If the null hypothesis were true, we might expect S ( γ, η ) to lie in a region with high probability of the permutation estimator, and we can use this as a criteria for rejecting the null. Under fairly weak assumptions, this yields a test at the appropriate level (Chung and Romano, 2013).

Comparing a uniform and a mixture. Any uniform random variable over a symmetric interval will have 0 mean and skewness, so a symmetric mixture only needs to match the variance. If X is a 50 / 50 mixture of U [ a, b ] and U [ -a, -b ] then

so if Y is distributed according to U [ -c, c ] then we only need to solve

which is straightforward for a given a and c .

Computational complexity of estimators. The V-statistic for d (2) as written in Lemma 2 is bottlenecked by the matrix multiplications. We may note however that for two matrices A , B it holds that

where ⟨·⟩ denotes the sum over elements and ◦ denotes the Hadamard product. We also note that for for H n = 1 n 1 n 1 ⊤ n we have ( AH n ) i,j = 1 n ∑ n c =1 A i,c . Using both of these tricks we may compute both d (2) and CSIC without any matrix multiplications, which brings the computational complexity down to O ( N 2 ) for both. For a comparison of actual computation time, see Fig. 6 and Fig. 7, where the average computational times for out methods are compared to the KME and and HSIC for N between 50 and 2000.

Type I error on the Seoul Bicycle data. The results when comparing the winter data to itself is presented in Fig. 8. As we see the performance is similar for both estimators and lies between 5 and 10% .

0

250

500

750

1000

1250

1500

1750

2000

N

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Average times (s)

0

250

500

750

1000

1250

1500

1750

2000

N

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Average times (s)

Classical vs. kernelized cumulants. Using the same distributions as in the synthetic independence testing experiment, we now compare X with Y 2 0 . 5 to contrast independence testing with classical cumulants with their kernelized counterpart. The results are summarized in Table 1 where they are displayed as the median value ± half the difference between the 75th and 25th percentile. We consider every combination of classical vs. kernelized, variance vs. skewness, and two different sample sizes. One can observe that the classical variance based test performs poorly compared to a classical skewness test, the kernelized variance test is almost as powerful as the kernelized skewness test, and in all cases the kernelized tests deliver higher power.

E Kernel trick computations

Here we show how to arrive at the expressions used for the V-statistics used in the experiments.

4

8

12

16

20

24

28

32

N

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Type I error (%)

Given a real analytic function f ( x, . . . , x d ) = ∑ i ∈ N d f i x i in m variables with nonzero radius of convergence and Hilbert spaces H 1 , . . . , H d we may (formally) extend f to a function

Moreover, if the Hilbert spaces are RKHSs then we have the following result.

Lemma 6 (Nonlinear kernel trick) . For any collection of RKHSs H 1 , . . . , H d with feature maps φ i : X i →H i , assume that f and g are real analytic functions with radii of convergence r ( f ) and r ( g ) such that max 1 ≤ i ≤ d sup x ∈X i | φ i ( x ) | < min( r ( f ) , r ( g )) . Then

Proof. Since the image of the φ i s lie inside the radius of convergence of f ⊗ and g ⊗ the power series converge absolutely and we can write

where H = H 1 ×··· × H d .

Using Lemma 6, we can choose kernels k i : X 2 i → R with associated RKHSs H i and feature maps φ i and some i ∈ N d with deg( i ) = m . We make the observation that with X = ( X 1 , . . . , X d ) ∼ γ , Y = ( Y 1 , . . . , Y d ) ∼ η and k ⊗ i and H ⊗ i as in (4), one has

Since

one gets the expected kernel trick statements of Theorem 2 and Theorem 3.

We are now interested in explicitly computing the expression ∥ κ (1 , 2) k,ℓ ( γ ) ∥ 2 H ⊗ 1 k ⊗H ⊗ 2 ℓ , ∥ κ (2) k ( γ )

-

a (w.l.o.g.) symmetric, measurable function h ( z 1 , . . . , z m ) , the V-statistic of h with N samples Z 1 , . . . , Z N is defined as

Under fairly general conditions, the V-statistic converges in distribution to E [ h ( Z 1 , . . . , Z m )] and a well-developed theory describes this convergence (Van der Waart, 2000; Serfling, 1980; Arcones and Giné, 1992).

Example E.1 (Estimating ∥ κ (2) k ( γ ) -κ (2) k ( η ) ∥ 2 H (1 , 1) ) . Let X,X ′ , X ′′ , X ′′′ denote independent copies of γ and Y, Y ′ , Y ′′ , Y ′′′ denote independent copies of η . The full expression for ∥ κ (2) k ( γ ) -κ (2) k ( η ) ∥ 2 H (1 , 1) is

Given samples ( x i ) N i =1 , ( y i ) i M =1 from γ and η respectively the corresponding V statistic is

Let us define the Gram matrices K x = [ k ( x i , x j )] N i,j =1 ∈ R N × N , K y = [ k ( y i , y j )] M i,j =1 ∈ R M × M , K x,y = [ k ( x i , y j )] N,M i,j =1 and let H N = 1 N 1 N 1 ⊤ N ∈ R N × N , H M = 1 M 1 M 1 ⊤ M ∈ R M × M be the centering, then (13) can be rewritten as

which simplifies to

This estimator can be computed in quadratic time.

Example E.2 (Estimating ∥ κ (1 , 2) k,ℓ ( γ ) ∥ 2 H ⊗ 1 k ⊗H ⊗ 2 ℓ ) . Let k denote the kernel on X 1 and ℓ denote the kernel on X 2 . Let ( X,Y ) , ( X ′ , Y ′ ) , ( X ′′ , Y ′′ ) , ( X (3) , Y (3) ) , ( X (4) , Y (4) ) , ( X (5) , Y (5) ) denote in-

dependent copies of γ ∈ P ( X 1 ×X 2 ) . The full expression for ∥ κ (1 , 2) k,ℓ ( γ ) ∥ 2 H ⊗ 1 k ⊗H ⊗ 2 ℓ is

Given samples ( x i , y i ) N i =1 from γ the corresponding V-statistic for this expression is

Using the shorthand notation K = K x , L = L y and H = H N and denoting by ◦ the Hadamard product [ A ◦ B ] i,j = A i,j B i,j and ⟨·⟩ the sum over all elements of a matrix ⟨ A ⟩ = ∑ N i,j =1 A i,j , the V-statistic above can be written in the simpler form

Again this estimator can be computed in quadratic time.

Example E.3 (Estimating ∥ κ (3) k ( γ ) -κ (3) k ( η ) ∥ 2 H ⊗ 3 k ) . In order to estimate d (3) ( γ, η ) we note that one can write

We can estimate the first two terms like in Example E.2, and the third term can be expressed as

For simplicity we will assume that we have an equal number of samples ( N ) from both measures ( x i ) N i =1 ∈ γ and ( y i ) N i =1 ∈ η . The V-statistic for ⟨ κ (3) k ( γ ) , κ (3) k ( η ) ⟩ H ⊗ 3 k can be expressed as

Using the notation K xy = [ k ( x i , y j )] N , this estimator simplifies to

We mention also that the first two terms ∥ κ (3) k ( γ ) ∥ 2 H ⊗ 3 k , ∥ κ (3) k ( η ) ∥ 2 H ⊗ 3 k can be computed a little more simply than in Example E.2 since the expressions have more symmetry, using the notation K x = [ k ( x i , x j )] N i,j =1 we can write down the V-statistic for ∥ κ (3) k ( γ ) ∥ 2 H ⊗ 3 k as

with a similar expression for ∥ κ (3) k ( η ) ∥ 2 H ⊗ 3 k . The estimator can be computed in quadratic time.

Peer reviews:
comments: SUMMARY:
This paper proposes kernelized cumulants to extend classical cumulants in $\mathbb{R}^d$ and shows that the kernelized cumulants provide a new set of all-purpose statistics and are computationally tractable. 
This paper also show advantages of kernelized cumulants both theoretically and empirically.

STRENGTHS:
* The paper is well-written and easy to follow.

* The method, that is, kernelized cumulant, proposed in this paper is novel and might be useful in real-world applications.
For example, this method can be used to provide metrics for distributions and measure independence.
Moreover, this method includes traditional MMD and HSIC as its special case and can even outperform the traditional ones.

WEAKNESSES:
* In the experiments, more kernels, such as the neural tangent kernel, could be considered.
* It would be better if the authors provide theoretical guarantees of the effectiveness of kernelized cumulants, such as consistency.

QUESTIONS:
1. In the experiments, what is the criteria of the 'optimal value' of $\sigma$ (Line 265)?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes the notion of cumulants to Hilbert-space-valued random variables. \nWhen these Hilbert spaces are RKHSs, the kernel trick applies so that computations can be performed with the kernel function.\nIt leads to higher-order two-sample and independence tests, which generalizes MMD and HSIC.\nThe efficiency of these tests is demonstrated numerically on both synthetic and real data.\n\nEDIT: I have read the author's rebuttal, which partially addressed my concerns.\n\nSTRENGTHS:\nThe paper is well written and clear, despite the fact that the notation is heavy due to the complexity of the considered objects.\nI appreciated that equivalent definitions of the kernelized cumulants are given, as well as special cases to build intuition and relate it to classical notions.\n\nBecause of this, I think the paper should be accepted.\n\nWEAKNESSES:\nA first weakness in my opinion is that the main contribution of the paper is a straightforward combination of two known concepts: (i) cumulants and (ii) the use of the kernel methods in statistical testing. Also, rather than using the tensor algebra framework used by the authors in Section 3 and Appendix C.2, it seems to me that the simpler route would be to use the generating function $K(\\theta_1, \\dots, \\theta_d) = \\log \\mathbb E[ \\exp(\\sum_{i=1}^d \\langle \\theta_i, X_i \\rangle ) ]$ and define the cumulants from its series expansion, or equivalently derivatives at zero (which would be the correct equivalent of item 1. in Appendix C.1, rather than eq. (11)) . The definitions are of course equivalent, but it would lighten the formalism in the main text.\n\nThe second main weakness is that the relationship between MMD/HSIC and the higher-order versions *when one is allowed to change the kernel* is not discussed. If I'm not mistaken, the second-moment embedding of a probability distribution with kernel $k$ coincides with the mean embedding with the squared kernel $k^2$. So one should expect a relationship between $d^{(2)}$ with kernel $k$ and MMD with some combination of $k$ and $k^2$ as kernel. If that is the case, then the use of higher-order cumulants can be equivalently rephrased as the use of different kernels. In particular, it is not clear whether one could be better off by sticking to MMD/HSIC but with well-designed kernels, as the estimation of higher-order moments has a higher variance. I think this point should be discussed in the text.\n\nTwo additional minor remarks:\n- Repetition of \"energy distance\" in lines 37-38, which I'm guessing is a typo.\n- Missing related work: [1] considers a second-moment kernel embedding and defines kernel information-theoretic quantities.\n\n[1] Bach, Francis. \"Information theory with kernel methods.\" IEEE Transactions on Information Theory 69.2 (2022): 752-775.\n\nQUESTIONS:\n- I am suspicious of the fact that the computational complexity of MMD/HSIC and the higher-order extensions are the same, and are bottlenecked by the computation of the kernel matrix. Surely the complexity must increase with $m = \\mathrm{deg}(\\mathbf{i})$, even if it remains quadratic in the sample size?\n\n- Is there a relationship between going to higher-order cumulants and changing the kernel? Can we achieve the same performance than the higher-order cumulants by adapting the kernel?\n\n- Isn't the the $V$ statistic estimator a straightforward replacement of the expectation with empirical averages over the sample? I think this could be mentioned in the text.\n\nLIMITATIONS:\nThe authors mentioned the two main limitations of their approach:\n- It is not clear how to design the kernels to maximize the performance of the statistical tests.\n- There is no theoretical analysis of the introduced tests.\n\nI agree that a complete resolution of these issues should be left to future work. However, I think the second weakness above, which is related to the first limitation, should be at least acknowledged.\n\nThere is no foreseeable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors introduce the kernelized cumulant and show that it can characterize distributions and statistical (in)dependence.

STRENGTHS:
1. The kernelized cumulant provides a natural generalization of the popular maximum mean discrepancy (MMD) as well as the Hilbert-Schmidt independence criterion (HSIC).
2. The authors illustrate several interesting properties of the kernelized cumulants and provide a two-sample test for non-characteristic feature maps as well as an independence test.
3. The authors demonstrate the utility of the proposed method on a variety of datasets with competitive results.

WEAKNESSES:
1. Estimating the kernelized cumulant may require a large sample. Convergence rate of the estimator is not discussed, but the empirical results suggest comparable performance as the HSIC.
2. The proposed method could potentially be vulnerable to kernel misspecification.

QUESTIONS:
1. Is the independence criterion kernel dependent? What happens if the kernel is misspecified?
2. What are the convergence rates for the finite-sample estimators?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This papers revisits advances in cumulants on real data, by extending them to provide cumulants for random variables in an RKHS.

STRENGTHS:
The idea of this work is interesting, as the paper proposed to go beyond conventional kernel mean and kernel covariance. Moreover, a proposed kernel trick allow to obtain efficiently the kernelized cumulants.
The paper well describes the contributions. The derivations seem to be good, and some experiments on synthetic and real datasets allow to understand the relevance of the proposed approach

WEAKNESSES:
While the paper includes some experimental results, there are only two settings : independence test and two-sample test (MMD-like). Is there any other  tests (or beyond test) where the proposed kernelized cumulants would be relevant ?

QUESTIONS:
see question in weaknesses

LIMITATIONS:
OK
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper generalises the notion of kernel mean embeddings to higher-order cumulants. It proposes kernelled cumulants in the RKHS. While kernelled cumulants reside in tensor product space of the RKHS, the paper shows that Hilbert space metric between cumulants can be exactly computed using the kernel trick. Based on this construction, the paper proposes: (1) a two-sample test statistics, that generalises MMD test statistics by considering distance between cumulants, and (2) a generalisation of HSIC statistic for independence testing, again by considering distant between cumulants of joint distribution and product of marginals. \nThe advantages of the construction and proposed tests include: (i) the test are applicable for a broader class of \"point-separating\" kernels (unlike MMD/HSIC that can only useful for characteristic kernels); (ii) the new statistics can be computed in quadratic time (same as MMD); and (iii) empirically achieve higher power that classical MMD/HSIC statistics (both on synthetic and real data).\n\nSTRENGTHS:\n- The idea of considering higher order moments/cumulants in RKHS is quite natural, and yet unexplored in the literature (apart from the recent work of Makigusa (2020) that consider only 2nd order moments). Hence, the contribution is novel and quite timely\n- The main strength of the paper is that the computational cost of the proposed statistics is still quadratic in sample size (same as MMD), which implies the advantages of higher order moments does not come at significant additional cost\n- The work is technically sound, and the construction of cumulant and use of kernel trick is reasonably involved.\n\nWEAKNESSES:\n- While it is easy to imagine in general that higher-order cumulants can distinguish between more distributions, the advantage of kernelled cumulants is difficult to grasp. If a characteristic kernel is used wouldn't mean embedding (MMD) suffice?\n- The line of argument used in the paper to demonstrate advantage of kernel cumulants is that (empirically) they have show higher power (can detect small differences better). Is there a theoretical justification for this? It would be sufficient if the authors provide a justification / reference that standard (non-kernel) cumulants are more sample efficient in some cases, where means already show separation\n\n-  For the synthetic experiments, the null rejection rate should also be plotted to show that cumulants do not have higher tendency to reject than MMD/HSIC. While this is true for the real data, unfortunately both 1st order and higher order terms reject at a rate higher than significance level\n\n- Overall, it is not clear when tests based on higher-order cumulants are indeed needed in practice. I still feel the work is relevant, but some discussions on this would certainly increase the significance of the work for the broader community \n- The paper, although well-written, is quite dense and at times bit difficult to follow, but this can be attributed to the content of the paper\n\nQUESTIONS:\n- see weakness\n- in addition, a precise statement on computational complexity (at least for d2, d3) would be useful\n\nMinor remarks: H in Lemma 3 is not defined in main paper (but in appendix), and E et al citation seems incorrect (there is a full name for E)\n\nLIMITATIONS:\nthe paper does not have immediate negative societal impact (although conclusions from hypothesis tests can always have). Hence, the work could benefit from:\n- consistency results (similar to kernel two-sample tests)\n- characterisation of whether higher-order cumulant based statistics typically tend to be larger than MMD (even under null). The comment is about sample estimates and not expected value (hence, more tied to concentration/consistency of test statistics)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}