PSD Representations for Effective Probability Models

1 Introduction . Modeling probability distributions is a key task for many applications in machine learning [ 17 , 5 ] . To this end , several strategies have been proposed in the literature , such as adopting mixture models ( e.g . Gaussian mixtures ) [ 5 ] , exponential models [ 32 ] , implicit generative models [ 11 , 13 ] or kernel ( conditional ) mean embeddings [ 16 ] . An ideal probabilistic model should have two main features : i ) efficiently perform key operations for probabilistic inference , such as sum rule ( i.e . marginalization ) and product rule [ 5 ] and , ii ) concisely approximate a large class of probabilities . Finding models that satisfy these two conditions is challenging and current methods tend to tackle only one of the two . Exponential and implicit generative models have typically strong approximation properties ( see e.g . [ 32 , 29 ] ) but can not easily perform operations such as marginalization . On the contrary , mixture models are designed to efficient integrate and multiply probabilities , but tend to require a large number of components to approximate complicated distributions . In principle , mixture models would offer an appealing strategy to model probability densities since they allow for efficient computations when performing key operations such as sum and product rule . However , these advantages in terms of computations , come as a disadvantage in terms of expressiveness : even though mixture models are universal approximators ( namely they can approximate arbitrarily well any probability density ) , they require a significant number n of observations and of components to do that . Indeed it is known that models that are non-negative mixtures of non-negative components lead to learning rates that suffer from the curse of dimensionality . For example , when approximating probabilities on Rd , kernel density estimation ( KDE ) [ 21 ] with non-negative components has rates as slow as n 2/ ( 4+d ) ( see , e.g . [ 38 , page 100 ] ) , and can not improve with the regularity ( smoothness ) of the target probability ( see e.g . [ 9 , Thm . 16.1 ] for impossibility results in the one dimensional case ) . In the past decades , this limitation has been overcome by removing either the non-negativity of the weights of the mixture ( leading to RBF networks [ 27 ] [ 35 ] ) , or the non-negativity of the components 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . used in the mixture ( leading to KDE with oscillating kernel [ 35 ] ) , or both . On the positive side this allows to achieve a learning rate of n / ( 2 +d ) , for -times differentiable densities on Rd . Note that such rate is minimax optimal [ 10 ] and overcomes the curse of dimensionality when d. Additionally the resulting model is very concise , since only m = O ( nd/ ) centers are necessary to achieve the optimal rate [ 36 ] . However , on the negative side , the resulting model is not a probability , since it may attain negative values , and so it can not be used where a proper probability is required . In this paper , we show that the positive semidefinite ( PSD ) models , recently proposed in [ 14 ] , offer a way out of this dichotomy . By construction PSD models generalize mixture models by allowing also for negative weights , while still guaranteeing that the resulting function is non-negative everywhere . Here , we prove that they get the best of both worlds : expressivity ( optimal learning and approximation rates with concise models ) and flexibility ( exact and efficient sum and product rule ) . Contributions . The main contributions of this paper are : i ) Showing that PSD models can perform exact sum and product rules in terms of efficient matrix operations ( Sec . 2.1 ) . ii ) Characterizing the approximation and learning properties of PSD models with respect to a large family of probability densities ( Sec . 3 ) . iii ) Providing a “ compression ” method to control the number of components of a PSD model ( introduced in Sec . 2.2 and analyzed in Sec . 3.3 ) . iv ) Discussing a number of possible applications of PSD models ( Sec . 4 ) . Summary of the results . Table 1 summarizes all the desirable properties that a probability model should have and which of them are satisfied by state-of-the-art estimators . Among these we have : non-negativity , the model should be point-wise non-negative ; sum and product rule , the model should allow for efficient computation of such operations between probabilities ; concise approximation , a model with the optimal number of components m = O ( `` d/ ) is enough to approximate with error `` a non-negative function in a wide family of smooth -times differentiable functions [ 36 ] ; optimal learning , the model achieves optimal learning rates of order n / ( 2 +d ) when learning the unknown target probability from i.i.d . samples [ 35 ] ; efficient sampling the model allows to extract i.i.d . samples without incurring in the curse of dimensionality in terms of computational complexity . We note that for PSD models , this last property has been recently investigated in [ 15 ] . We consider under the umbrella of linear models all the models which corresponds to a mixture P m j=1 wjfj ( x ) of functions fj : X ! R , with either wj 0 , 8j = 1 , . . . , m or fj 0 , 8j = 1 , . . . , m or both . We denote by mixture models the mixture models defined as above , where wj 0 , fj 0 , 8j = 1 , . . . , m. By mean embeddings , we denote the kernel mean embedding estimator framework from [ 30 ] see also [ 16 ] ) . With exponential models we refer to models of the form exp ( P m j=1 wjfj ( x ) ) with wj 2 R , fj : X ! R [ 32 ] . With PSD models we refer to the framework introduced in [ 14 ] and studied in this paper . We note that this work contributes in showing that the latter are the only probability models to satisfy all requirements ( we recall that non-negativity and efficient sampling have been shown respectively in [ 14 ] and [ 15 ] ) . Notation . We denote by Rd++ the space vectors in Rd with positive entries , Rn⇥d the space of n ⇥ d matrices , Sn+ = S+ ( Rn ) the space of positive semidefinite n ⇥ n matrices . Given a vector ⌘ 2 Rd , we denote diag ( ⌘ ) 2 Rd⇥d the diagonal matrix associated to ⌘ . We denote by A B and A ⌦ B respectively the entry-wise and Kronecker product between two matrices A and B . We denote by kAkF , kAk , det ( A ) , vec ( A ) and A > respectively the Frobenius norm , the operator norm ( i.e . maximum singular value ) , the determinant , the ( column-wise ) vectorization of a matrix and the ( conjugate ) transpose of A . With some abuse of notation , where clear from context we write element-wise products and division of vectors u , v 2 Rd as uv , u/v . Given two matrices X 2 Rn⇥d1 , Y 2 Rn⇥d2 with same number of rows , we denote by [ X , Y ] 2 Rn⇥ ( d1+d2 ) their concatenation row-wise . The term 1n 2 Rn denotes the vector with all entries equal to 1 . 2 PSD Models . Following [ 14 ] , in this work we consider the family of positive semi-definite ( PSD ) models , namely non-negative functions parametrized by a feature map : X ! H from an input space X to a suitable feature space H ( a separable Hilbert space e.g . Rq ) and a linear operator M 2 S+ ( H ) , of the form f ( x ; M , ) = ( x ) > M ( x ) . ( 1 ) PSD models offer a general way to parametrize non-negative functions ( since M is positive semidefinite , f ( x ; M , ) 0 for any x 2 X ) and enjoy several additional appealing properties discussed in the following . In this work , we will focus on a special family of models of the form ( 1 ) to parametrize probability densities over X = Rd . In particular , we will consider the case where : i ) = ⌘ : Rd ! H⌘ is a feature map associated to the Gaussian kernel [ 26 ] k⌘ ( x , x0 ) = ⌘ ( x ) > ⌘ ( x ) = e ( x x 0 ) > diag ( ⌘ ) ( x x0 ) , with ⌘ 2 Rd++ and , ii ) the operator M lives in the span of ( x1 ) , . . . , ( xn ) for a given set of points ( xi ) ni=1 , namely there exists A 2 Sn+ such that M = P ij Aij ( xi ) ( xj ) > . We define a Gaussian PSD model by specializing the definition in ( 1 ) as f ( x ; A , X , ⌘ ) = nX i , j=1 Ai , jk⌘ ( xi , x ) k⌘ ( xj , x ) , 8x 2 Rd ( 2 ) in terms of the coefficient matrix A 2 Sn+ , the base points matrix X 2 Rn⇥d , whose i-th row corresponds to the point xi for each i = 1 , . . . , n and kernel parameter ⌘ . In the following , given two base point matrices X 2 Rn⇥d and X 0 2 Rm⇥d , we denote by KX , X0 , ⌘ 2 Rn⇥m the kernel matrix with entries ( KX , X0 , ⌘ ) ij = k⌘ ( xi , x0j ) where xi , x0j are the i-th and j-th rows of X , X 0 respectively . When clear from context , in the following we will refer to Gaussian PSD models as PSD models . Remark 1 ( PSD models generalize Mixture models ) . Mixture models ( a mixture of Gaussian distributions ) are a special case of PSD models . Let A = diag ( a ) be a diagonal matrix of n positive weights a 2 Rn++ . We have f ( x ; A , X , ⌘/2 ) = P n i=1 Aiik⌘/2 ( xi , x ) 2 = P n i=1 aik⌘ ( xi , x ) . Remark 2 ( PSD models allow negative weights ) . From ( 2 ) , we immediately see that PSD models generalize mixture models by allowing also for negative weights : e.g. , f ( · ; A , X , ⌘ ) with A = ( 1 , 12 ; 1 2 , 1 4 ) 2 S 2 + , ⌘ = 1 , X = ( x1 ; x2 ) and x1 = 0 , x2 = 2 , corresponding to f ( x ; A , X , ⌘ ) = e 2x 2 + 14e 2 ( x 2 ) 2 1 e e 2 ( x 1 ) 2 , i.e . a mixture of Gaussians with also negative weights .

Summary: The authors discuss an application of recent work in nonnegative function estimation to the problem of density estimation.  In particular, they argue that under mild assumptions, PSD models can concisely model a large range of interesting densities and that these densities can be (more or less) efficiently learned.  The work concludes with a qualitative comparison against existing density estimation strategies.