A generalized probability kernel on discrete distributions and its application in two-sample test

We propose a generalized probability kernel ( GPK ) on discrete distributions with finite support . This probability kernel , defined as kernel between distributions instead of samples , generalizes the existing discrepancy statistics such as maximum mean discrepancy ( MMD ) as well as probability product kernels , and extends to more general cases . For both existing and newly proposed statistics , we estimate them through empirical frequency and illustrate the strategy to analyze the resulting bias and convergence bounds . We further propose power-MMD , a natural extension of MMD in the framework of GPK , illustrating its usage for the task of two-sample test . Our work connects the fields of discrete distribution-property estimation and kernel-based hypothesis test , which might shed light on more new possibilities . 1 INTRODUCTION . We focus on the two-sample problem , which is given two i.i.d samples { x1 , x2 , ... xn } , { y1 , y2 , ... , yn } , could we infer the discrepancy between underlying distributions they are drawn from . For such a problem , the option of hypothesis test ( two-sample test ) is most popular , and a variety of statistics in estimating the discrepancy is proposed . In recent years , RKHS based method such as maximum mean discrepancy ( MMD ) has gained a lot of attention . ( Gretton et al. , 2012 ) has shown that in a universal-RKHS F , MMD ( F , p , q ) = 0 if and only if p = q , thus could be used for the two-sample hypothesis test . ( Gretton et al. , 2012 ) further provides unbiased estimator of MMD with fast asymptotic convergence rate , illustrating its advantages . On the other hand , estimating distribution properties with plugin ( empirical ) estimators on discrete setting is an active research area in recent years , where people focus on problem settings with large support size but not so large sample size . The Bernstein polynomial technique is introduced to analyze the bias of the plugin estimators in ( Yi & Alon , 2020 ) , which provides remarkable progress on bias-reduction methods of the plugin estimators . It is thus interesting to ask if the plugin estimators could motivate new results for the RKHS-based two-sample test . Another interesting topic is about the probability kernel , defined as kernel function over probabilities , instead of over samples . As is easily seen , any discrepancy measure of distribution p and q could potentially be valid probability kernels , not so much work focuses on this . While ( Jebara et al. , 2004 ) introduced the so called probability product kernels which generalize a variety of discrepancy measures , its properties remain further study . Motivated by above observations , our work focuses on a specialized probability kernel function which is a direct generalization of sample-based RKHS methods such as MMD . We focus on using plugin-estimator as the default estimator of the kernel function we defined , and illustrate that with the help of Bernstein polynomial techniques , we could analyze the bias and convergence bounds of these plugin-estimators . Our work thus connects the fields of discrete distribution-property estimation and kernel-based hypothesis test , which brings interesting possibilities . 2 NOTATION . We use bold symbol p , q ∈ Rk to represent a probability function over a discrete support with support size k , and pi , qi represents the ith entry of p and q . We use { v1 , v2 , ... , vk } , vi ∈ Rd to represent the support of p , q . [ k ] : = { 1 , 2 , 3 ... , k } represents the set of indices of elements in { v1 , v2 , ... , vk } . We use φ ◦ ( p , q ) to denote an element-wise function from Rk × Rk to Rk , where ( φ ◦ ( p , q ) ) i = φ ◦ ( pi , qi ) and φ ◦ p to denote an element-wise function from Rk to Rk , where ( φ ◦ p ) i = φ ◦ pi . With a slight abuse of notation , we denote pρ , p − q as element-wise function defined above . We use kernel ( p , q ) to denote kernel function which maps from Rk × Rk to real value R. And kernel ( x , y ) , x , y ∈ Rd represents a kernel function from Rd × Rd to real value R. We use K to denote the gram matrix generated from kernel ( x , y ) on finite support { v1 , v2 , ... , vk } , where Kij = kernel ( vi , vj ) . We use { x1 , x2 , ... , xn } ∼ p and { y1 , y2 , ... , yn } ∼ q to denote the samples from distribution p and q , where n is the sample size . 3 GENERALIZED PROBABILITY KERNEL . Probability kernel function , defined as kernel function between distributions instead of samples , is a natural extension of the idea of kernel function in sample space . Definition 1 . Given distribution p and q belongs to a family of discrete distribution with the same finite support { v1 , v2 , ... , vk } , vi ∈ Rd , where k is the support size , we define the probability kernel function as PK ( p , q ) , which is a kernel function maps from Rk × Rk to real value R. Many discrepancy measures , such as MMD , can serve as probability kernel functions , but people usually don ’ t use the term of probability kernel function when describing them . The reason is that for most of the time , we only consider a limited number of distributions , and do not need or have the resources to navigate through all the distributions within the family . For example , when looking into the two-sample problem , we usually assume two samples { x1 , x2 , ... , xn } ∈ Rd and { y1 , y2 , ... , yn } ∈ Rd are i.i.d drawn from two distributions p and q , and use the discrepancy measure MMD [ F , p , q ] to determine if p and q are indistinguishable in the RKHS F . We do not consider all other distributions in F that is irrelevant to our samples ! So far the idea of kernel function between distributions is in practice not so much useful , however , here in this paper , we propose , when considering the plugin-estimator of many of the existing discrepancy measures , it is beneficial to view them as probability kernel functions . 3.1 DEFINATION OF GENERALIZED PROBABILITY KERNEL . Definition 2 ( Generalized probability kernel ) . Given the family S of discrete distribution on support { v1 , v2 , .. , vk } where vi ∈ Rd . Let F be a unit ball in a universal-RKHS H with associated continuous kernel RK ( x , y ) , where for any x ∈ Rd and y ∈ Rd , RK ( x , y ) maps from Rd ×Rd to R. We denote gram matrix Kij = RK ( vi , vj ) . The generalized probability kernel function on distribution p , q ∈ S is GPKF , φ ( p , q ) = φ ◦ ( p , q ) Kφ ◦ ( q , p ) T = ∑ i∈ [ k ] ∑ j∈ [ k ] φ ◦ ( pi , qi ) Kijφ ◦ ( qj , pj ) where φ ◦ ( p , q ) is an element-wise mapping function on discrete distribution p , q ∈ S , which maps from Rk × Rk to Rk , Obviously , under this definition , the GPK is a symmetric probability kernel function where GPKF , φ ( p , q ) = GPKF , φ ( q , p ) Mapping function φ represent a great amount of possibilities . For most cases , we need to narrow down the region and equipped it with some convenient properties so that the GPK measure could be useful . One example is for the measurement of discrepancy , where we want GPKF , φ ( p , q ) = 0 if and only if p = q . Definition 3 ( discrepancy probability kernel ) . Let S be a family of discrete distribution p ∈ S on support { v1 , v2 , ... , vk } . A discrepancy probability kernel is a kernel function PK ( p , q ) that PK ( p , q ) = 0 if and only if p = q Theorem 1 . GPKF , φ ( p , q ) with the mapping function φ that satisfies : 1. symmetric or antisymmetric with respect to p and q : φ ◦ ( p , q ) = φ ◦ ( q , p ) or φ ◦ ( p , q ) = −φ ◦ ( q , p ) 2 . ‖φ ◦ ( p , q ) ‖2 = ‖φ ◦ ( q , p ) ‖2 = 0 if and only if p = q , where ‖ · ‖2 represents L2 norm . is a discrepancy probability kernel . Proof . GPKF , φ ( p , q ) = ∑ i∈ [ k ] ∑ j∈ [ k ] φ ◦ ( pi , qi ) Kijφ ◦ ( qj , pj ) = φ ◦ ( p , q ) Kφ ◦ ( q , p ) T = ±φ ◦ ( p , q ) Kφ ◦ ( p , q ) T = ±vKvT K is a semipositive definite matrix , thus by definition of positive definite matrix , vKvT ≥ 0 , where equality holds if and only if v = 0 , and since v = φ ◦ ( p , q ) , this condition further means φ ◦ ( p , q ) = 0 , which holds if and only if p = q . Another example is the polynomial GPK , which is our main focus of this paper . Such a subclass of GPK is interesting since we can build unbiased estimators of it using techniques of Bernstein polynomial in ( Qian et al. , 2011 ) . As we will show in section 5. , we also have analyzable convergence bounds for the resulting unbiased estimators , illustrating its potential usage for applications such as two-sample test . Definition 4 ( polynomial GPK ) . The polynomial GPK is the subset of GPK that equipped with the mapping function φ that is polynomial in p and q : φ ◦ ( p , q ) = ∑o l=0 ∑o s=0 αl , sp lqs where o ∈ Z is the degree of the polynomial , and al , s ∈ R is the coefficient Below we give some examples of polynomial GPK , which include MMD proposed in ( Gretton et al. , 2012 ) , and the newly proposed power-MMD in this paper , which is a natural extension of MMD , from the view point of probability kernels . 3.1.1 EXAMPLE 1 : MMD AS MEMBER OF POLYNOMIAL GPK . Given discrete distribution p , q with support { v1 , v2 , ... , vk } , we can rewrite MMD with distribution probability function pi , qi : MMD2F ( p , q ) = ‖Ex∼pf ( x ) −Ex′∼qf ( x′ ) ‖ 2 H = ∥∥∥∥∥∥ ∑ i∈ [ k ] f ( vi ) pi − ∑ i∈ [ k ] f ( vi ) qi ∥∥∥∥∥∥ 2 H = ∥∥∥∥∥∥ ∑ i∈ [ k ] f ( vi ) pi − f ( vi ) qi ∥∥∥∥∥∥ 2 H = ∑ i∈ [ k ] ∑ j∈ [ k ] ( pi − qi ) f ( vi ) f ( vj ) ( pj − qj ) = ∑ i∈ [ k ] ∑ j∈ [ k ] ( pi − qi ) Kij ( pj − qj ) = −GPKF , φl ( p , q ) Where φl ◦ ( p , q ) = p − q , H is the RKHS defined in MMD literature , and f is the function that maps vi toH . GPKF , φl ( p , q ) is a special case of polynomial GPK where α1,0 = 1 , α0,1 = −1 , and all other coefficients are 0 .

Summary: The paper under review proposes to generalize MMD for discrete random variables whose labels take value in $\mathbb{R}^k$. They propose to estimate these generalized probability kernel distance using empirical estimators. Their properties are studied for two particular examples, namely a kernelized Stein discrenpancy and polynomials versions. Consistency and bias of both estimators are studied and bias corrected.