Fast Bayesian Inference for Gaussian Cox Processes via Path Integral Formulation

1 Introduction . Gaussian Cox processes constitute a class of doubly-stochastic point process models in which a flexible prior over a latent intensity function is established by a Gaussian process through a positive link function under which point events are generated . Gaussian Cox processes are the gold standard in analyzing event data in a Bayesian manner , and have a wide ranging list of applications in neuroscience [ 10 ] , finance [ 5 ] , and spatio-temporal analysis [ 30 ] . The observation process , i.e. , the likelihood function of the Gaussian Cox process , depends on the functional form of the latent intensity function over a compact domain , which makes Bayesian inference challenging as it imposes the intractable computation of infinite-dimensional distributions . Many algorithms have been proposed to deal with this difficulty . They include Markov Chain Monte Carlo ( MCMC ) sampling , domain discretization , and variational Bayesian ( VB ) approximation . MCMC approaches [ 2 , 22 ] provide exact inferencing for a specific link function , but they have excessive computation costs . Although domain discretization [ 10 , 43 ] achieves sub-linear computational scaling with discretization size , it suffers from poor scaling with data size and the dimension of the domain . To overcome these limitations , state-of-the-art algorithms employ sparse VB approximation with inducing points [ 3 , 31 ] ; they achieve improved scaling with the dimension of the domain as well as computation cost that is linear to data size . However , the computation costs of VB approaches scale at the rate of O ( NL2+L3 ) where L and N are the number of inducing points and data points , respectively , making them problematic in the multi-dimensional domain setting , where L tends to be large . Also , VB approaches achieve good approximations by aggressively focusing on the unique structures of individual kernel functions or link functions , and thus are not applicable to a wide range of Gaussian Cox processes . In this paper , we introduce a conceptually-intuitive path integral formulation of Gaussian processes , by which we derive a novel inference scheme for tackling intractable Gaussian Cox processes . The 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . formulation predicts that the maximum a posteriori ( MAP ) estimation of a latent intensity function coincides with the problem of finding the most likely path that maximizes the functional posterior , whose solution can be obtained by solving an explicit integral equation . The approximate predictive covariance is also acquired by solving the corresponding integral equation . We propose an efficient algorithm to solve the integral equations ; it offers O ( NL+ L2 ) computation scaling , that is , lower complexity than the state-of-the-art variational Bayesian schemes with respect to the number of inducing points , as well as linear computation scaling with data size . Furthermore , our inference scheme via path integral holds for any kernel function and a wide range of link functions , and thus provides a practical methodology against Gaussian Cox processes that conventional approaches have yet to tackle successfully . In Section 2 , we introduce the path integral formulation of Gaussian processes , and construct a new Bayesian inference scheme for Gaussian Cox processes . In Section 3 , we provide a practical methodology to perform inferencing . In Section 4 , we outline related work . In Section 5 , we perform comparative evaluations against reference methods on synthetic and real-world data , and confirm that our scheme achieves comparable predictive accuracy with substantial speed improvements over state-of-the-art algorithms . Finally , Section 6 states our conclusions . 2 Path Integral Formulation . 2.1 Gaussian Cox Process . We assume that a latent function on a compact space , T ⊂ RD , x ( t ) : T → R , is generated from a Gaussian Process ( GP ) , denoted by GP ( x ( t ) |µ , k ( t , t′ ) ) , and observations are generated from a process under latent function x ( t ) , where µ and k are the mean value and the kernel/covariance function for x ( t ) , respectively . Given data set D observed within T , we consider the problem of obtaining the maximum a posteriori ( MAP ) estimator of x ( t ) that maximizes the posterior probability p ( x ( t ) |D ) = p ( D|x ( t ) ) GP ( x ( t ) |µ , k ) ∫ Dx ( t ) p ( D|x ( t ) ) GP ( x ( t ) |µ , k ) , ( 1 ) where p ( D|x ( t ) ) is the likelihood function , and ∫ Dx ( t ) in the denominator represents the integral over the function or the infinite-dimensional variable x ( t ) . The GP prior , GP ( x ( t ) |µ , k ) , can be regarded as a normal distribution for an infinite-dimensional variable ( see Section 2.2 ) , and thus if the likelihood function depends only on a set of latent functions on finite N training points , p ( D|x ( t ) ) = p ( D| { x ( tn ) } Nn=1 ) , then MAP estimation for an infinitedimensional variable reduces to that for a finite N -dimensional variable by marginalizing out the variables on the points other than { x ( tn ) } Nn=1 ; this inference is tractable and has been examined extensively ( e.g . GP regression , GP classification ) . Here , we focus on a more challenging situation where the likelihood function includes a factor that depends on an integral of a latent function over T as follows , log p ( D|x ( t ) ) = N∑ n=1 log λ ( tn ) − ∫ T λ ( t ) dt , λ ( t ) = κ ( x ( t ) ) , ( 2 ) where D = { tn } Nn=1 is the set of point events occurring in the observation region T , κ ( x ) : R → R+ is a non-negative function called the link function , and λ ( t ) is the intensity function or the instantaneous probability of events occurring at each point in T . The probabilistic model defined by ( 1-2 ) is called the Gaussian Cox process . Bayesian inference ( 1 ) on situation ( 2 ) is often referred as “ doubly-intractable '' , because it requires solving the integral of the stochastic function x ( t ) as well as computing the intractable posterior probability of an infinite-dimensional variable . Below , we address the problem directly by introducing path integral , a mathematical tool familiar in the field of quantum physics , to treat probability distributions of stochastic functions properly and intuitively . 2.2 Gaussian Process Prior via Path Integral . In order to derive an explicit description of GP prior , we first review a well-known formulation of GP [ 7 ] , and reconstruct its explicit form of probability density distribution by using a path integral representation . Consider a model defined in terms of a linear combination of M fixed basis functions , { ϕm ( t ) } Mm=1 , such that x ( t ) = µ+ M∑ m=1 αmϕm ( t ) = µ+α ⊤ϕ ( t ) , t ∈ T , ( 3 ) where α ≜ ( α1 , · · · , αM ) ⊤ is the M -dimensional weight vector , ϕ ( t ) ≜ ( ϕ1 ( t ) , · · · , ϕM ( t ) ) ⊤ , and µ is a mean parameter . If α is generated from an M -dimensional Gaussian with a diagonal covariance matrix of the form p ( α ) = N ( α|0 , c−1IM ) , ( 4 ) the joint probability distribution for any finite ( even infinite ) set of variables , x ≜ ( x ( t1 ) , . . . , x ( tJ ) ) ⊤ for J ≤ ∞ , follows the J-dimensional Gaussian given by p ( x ) dx = N ( x|µ1 , K ) dx , Kjj′ = k ( tj , tj′ ) ≜ c−1ϕ ( tj ) ⊤ϕ ( tj′ ) , ( 5 ) where k ( t , t′ ) is a semi-definite kernel function , c is the precision parameter , 1 ≤ j , j′ ≤ J , and 1 ≜ ( 1 , . . . , 1 ) ⊤ . Because ( 5 ) holds for arbitrary numbers of dimensions , J , it provides an implicit representation of GP prior . Next , based on the expression ( 5 ) , we derive a more explicit representation of GP prior . For simplicity of explanation , we assume that the domain T is one-dimensional , t = t and T = [ 0 , T ] , and mean µ is zero . Let K be the integral operator corresponding to k ( t , t′ ) , and K∗ be the inverse operator of K , K ( ∗ ) x ( t ) ≜ ∫ T k ( ∗ ) ( t , t′ ) x ( t′ ) dt′ , K∗k ( t , s ) = ∫ T k∗ ( t , t′ ) k ( t′ , s ) dt′ = δ ( t− s ) , ( 6 ) where δ ( · ) represents the Dirac delta function . Note that K∗ and k∗ ( t , t′ ) should be described by a differential operator because relation ( 6 ) indicates that k ( t , s ) is the Green function [ 15 ] for the operator K∗ , but we do not go into detail here . Under the finite scenario ( 5 ) with [ t0 , tJ ] = [ 0 , T ] and J ≫ 1 , the second equation in ( 6 ) can be approximated by K∗∆K = ∆−1 , where ∆jj′ = ( tj − tj−1 ) δjj′ , K∗jj′ = k∗ ( tj , tj′ ) , and δjj′ is the Kronecker delta . Then we can write down the J-dimensional Gaussian ( 5 ) by using K∗ as p ( x ) dx = √ |∆K∗∆| ( 2π ) J exp [ −1 2 ( ∆x ) ⊤K∗ ( ∆x ) ] dx . ( 7 ) By taking the limit of the division number J → ∞ ( ∆ → 0 ) for the finite-dimensional Gaussian measure ( 7 ) , we achieve the following path integral representation of GP prior over the latent function x ( t ) : GP ( x ( t ) |0 , k ) Dx ≜ lim J→∞ p ( x ) J∏ j=1 dx ( tj ) = √ 1 |K| exp [ −1 2 ∫∫ T ×T k∗ ( t , s ) x ( t ) x ( s ) dtds ] Dx , ( 8 ) where |K| ≜ limJ→∞ |K∆| , Dx ≜ limJ→∞ ∏J j=1 √ ( tj−tj−1 ) / ( 2π ) dx ( tj ) , and the following path integral holds , ∫ exp [ −1 2 ∫∫ T ×T k∗ ( t , s ) x ( t ) x ( s ) dtds ] Dx = √ |K| . ( 9 ) |K| represents the Fredholm determinant or functional determinant [ 19 ] of the integral operator K , which is defined by the product of its eigenvalues . See the supplementary material ( §1 ) for the derivation . The expression for GP ( x ( t ) |µ , k ) is easily recovered by x ( t ) → x ( t ) − µ and t → t in ( 8 ) . The path integral representation ( 8-9 ) has a clear advantage over the standard one ( 5 ) in that the distribution of the latent function x ( t ) is written in terms of the explicit integral of x ( t ) over the domain T , as with the likelihood function ( 2 ) , which makes it possible to apply functional analyses , calculus of variation , to the Bayesian estimation ( 1 ) . To the best of our knowledge , this is the first proposal of a path integral representation for GP prior in the machine learning community . It should be noted that our derivation of the path integral representation is based on an intuitive view reminiscent of Feynman ’ s path integral formulation of the quantum field theory [ 17 ] , and thus is not mathematically rigorous . In this paper , we confirm that our path integral expression is valid by using experiments rather than taking a more mathematically rigorous approach ; we discuss this briefly in Section 4 . 2.3 Maximum A Posteriori Estimation Using the representation ( 8 ) and the likelihood function ( 2 ) , we can rewrite the posterior ( 1 ) as the functional , p ( x ( t ) |D ) Dx = 1 p ( D ) exp [ −S ( x ( t ) , x ( t ) ) ] Dx , ( 10 ) where S ( x ( t ) , x ( t ) ) is an action integral , defined by S ( x ( t ) , x ( t ) ) ≜ ∫ T [ 1 2 ( x ( t ) −µ ) x ( t ) +κ ( x ( t ) ) − N∑ n=1 log κ ( x ( t ) ) δ ( t−tn ) ] dt+ 1 2 log |K| , ( 11 ) and x ( t ) ≜ ∫ T k ∗ ( t , t′ ) ( x ( t′ ) −µ ) dt′ = K∗ ( x − µ ) . ( 10 ) indicates that MAP estimation of x ( t ) is equivalent to the problem of finding the most likely function or path x ( t ) that minimizes the action integral . Thus we now apply calculus of variations to the action integral , where the functional derivative of S ( x ( t ) , x ( t ) ) on the MAP estimator x̂ ( t ) is equal to zero : δSδx̂ ( t ) δx̂ ( t ) + δS δx̂ ( t ) δx̂ ( t ) = 0 . This leads us to realize the following equation for deriving the MAP estimator x̂ ( t ) , x̂ ( t ) + ∫ T k ( t , t′ ) κ̇ ( x̂ ( t′ ) ) dt′ = µ+ N∑ n=1 k ( t , tn ) κ̇ ( x̂ ( tn ) ) κ ( x̂ ( tn ) ) , t ∈ T , ( 12 ) where κ̇ ( x ) ≜ dκ ( x ) /dx . See the supplementary material ( §2 ) for the detailed derivations of ( 12 ) .

Summary: This outstanding paper presents an extremely novel approach (for the machine learning community) to the Gaussian process formulation. This formulation is interesting in its own right. But, the paper goes beyond the pure GP interpretation aspect and introduces this formulation for a very specific and precise purpose: to handle the challenging Poisson process likelihood function. This likelihood function is non-i.i.d. as it involves and integral term (equation 2 of the submission). The integral formulation leads to sneaky analytical tricks on the Poisson likelihood, and provides a new way to handle the interesting "doubly stochastic" inference problem.  The authors import some highly technical and novel concepts from physics, to attack the latent GP intensity function of the Poisson process. This leads to a highly efficient and principled algorithm. Moreover, the high degree of novelty is such that the technique may be applied to other problems with latent GP intensity. For example, their method would also allow to handle the likelihood function of survival analysis with latent GP + link function for the hazard function, which also includes an integral term.  The writing is crisp and compelling. This was a pleasure to behold. The experiments are more than sufficient. It is impressive that the authors made this work in practice, and I hope they see fit to release their code, but if they don't it is okay and the paper more than stands on its own.