Fast Bayesian Inference for Gaussian Cox Processes via Path Integral Formulation

1 Introduction . Gaussian Cox processes constitute a class of doubly-stochastic point process models in which a flexible prior over a latent intensity function is established by a Gaussian process through a positive link function under which point events are generated . Gaussian Cox processes are the gold standard in analyzing event data in a Bayesian manner , and have a wide ranging list of applications in neuroscience [ 10 ] , finance [ 5 ] , and spatio-temporal analysis [ 30 ] . The observation process , i.e. , the likelihood function of the Gaussian Cox process , depends on the functional form of the latent intensity function over a compact domain , which makes Bayesian inference challenging as it imposes the intractable computation of infinite-dimensional distributions . Many algorithms have been proposed to deal with this difficulty . They include Markov Chain Monte Carlo ( MCMC ) sampling , domain discretization , and variational Bayesian ( VB ) approximation . MCMC approaches [ 2 , 22 ] provide exact inferencing for a specific link function , but they have excessive computation costs . Although domain discretization [ 10 , 43 ] achieves sub-linear computational scaling with discretization size , it suffers from poor scaling with data size and the dimension of the domain . To overcome these limitations , state-of-the-art algorithms employ sparse VB approximation with inducing points [ 3 , 31 ] ; they achieve improved scaling with the dimension of the domain as well as computation cost that is linear to data size . However , the computation costs of VB approaches scale at the rate of O ( NL2+L3 ) where L and N are the number of inducing points and data points , respectively , making them problematic in the multi-dimensional domain setting , where L tends to be large . Also , VB approaches achieve good approximations by aggressively focusing on the unique structures of individual kernel functions or link functions , and thus are not applicable to a wide range of Gaussian Cox processes . In this paper , we introduce a conceptually-intuitive path integral formulation of Gaussian processes , by which we derive a novel inference scheme for tackling intractable Gaussian Cox processes . The 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . formulation predicts that the maximum a posteriori ( MAP ) estimation of a latent intensity function coincides with the problem of finding the most likely path that maximizes the functional posterior , whose solution can be obtained by solving an explicit integral equation . The approximate predictive covariance is also acquired by solving the corresponding integral equation . We propose an efficient algorithm to solve the integral equations ; it offers O ( NL+ L2 ) computation scaling , that is , lower complexity than the state-of-the-art variational Bayesian schemes with respect to the number of inducing points , as well as linear computation scaling with data size . Furthermore , our inference scheme via path integral holds for any kernel function and a wide range of link functions , and thus provides a practical methodology against Gaussian Cox processes that conventional approaches have yet to tackle successfully . In Section 2 , we introduce the path integral formulation of Gaussian processes , and construct a new Bayesian inference scheme for Gaussian Cox processes . In Section 3 , we provide a practical methodology to perform inferencing . In Section 4 , we outline related work . In Section 5 , we perform comparative evaluations against reference methods on synthetic and real-world data , and confirm that our scheme achieves comparable predictive accuracy with substantial speed improvements over state-of-the-art algorithms . Finally , Section 6 states our conclusions . 2 Path Integral Formulation . 2.1 Gaussian Cox Process . We assume that a latent function on a compact space , T ⊂ RD , x ( t ) : T → R , is generated from a Gaussian Process ( GP ) , denoted by GP ( x ( t ) |µ , k ( t , t′ ) ) , and observations are generated from a process under latent function x ( t ) , where µ and k are the mean value and the kernel/covariance function for x ( t ) , respectively . Given data set D observed within T , we consider the problem of obtaining the maximum a posteriori ( MAP ) estimator of x ( t ) that maximizes the posterior probability p ( x ( t ) |D ) = p ( D|x ( t ) ) GP ( x ( t ) |µ , k ) ∫ Dx ( t ) p ( D|x ( t ) ) GP ( x ( t ) |µ , k ) , ( 1 ) where p ( D|x ( t ) ) is the likelihood function , and ∫ Dx ( t ) in the denominator represents the integral over the function or the infinite-dimensional variable x ( t ) . The GP prior , GP ( x ( t ) |µ , k ) , can be regarded as a normal distribution for an infinite-dimensional variable ( see Section 2.2 ) , and thus if the likelihood function depends only on a set of latent functions on finite N training points , p ( D|x ( t ) ) = p ( D| { x ( tn ) } Nn=1 ) , then MAP estimation for an infinitedimensional variable reduces to that for a finite N -dimensional variable by marginalizing out the variables on the points other than { x ( tn ) } Nn=1 ; this inference is tractable and has been examined extensively ( e.g . GP regression , GP classification ) . Here , we focus on a more challenging situation where the likelihood function includes a factor that depends on an integral of a latent function over T as follows , log p ( D|x ( t ) ) = N∑ n=1 log λ ( tn ) − ∫ T λ ( t ) dt , λ ( t ) = κ ( x ( t ) ) , ( 2 ) where D = { tn } Nn=1 is the set of point events occurring in the observation region T , κ ( x ) : R → R+ is a non-negative function called the link function , and λ ( t ) is the intensity function or the instantaneous probability of events occurring at each point in T . The probabilistic model defined by ( 1-2 ) is called the Gaussian Cox process . Bayesian inference ( 1 ) on situation ( 2 ) is often referred as “ doubly-intractable '' , because it requires solving the integral of the stochastic function x ( t ) as well as computing the intractable posterior probability of an infinite-dimensional variable . Below , we address the problem directly by introducing path integral , a mathematical tool familiar in the field of quantum physics , to treat probability distributions of stochastic functions properly and intuitively . 2.2 Gaussian Process Prior via Path Integral . In order to derive an explicit description of GP prior , we first review a well-known formulation of GP [ 7 ] , and reconstruct its explicit form of probability density distribution by using a path integral representation . Consider a model defined in terms of a linear combination of M fixed basis functions , { ϕm ( t ) } Mm=1 , such that x ( t ) = µ+ M∑ m=1 αmϕm ( t ) = µ+α ⊤ϕ ( t ) , t ∈ T , ( 3 ) where α ≜ ( α1 , · · · , αM ) ⊤ is the M -dimensional weight vector , ϕ ( t ) ≜ ( ϕ1 ( t ) , · · · , ϕM ( t ) ) ⊤ , and µ is a mean parameter . If α is generated from an M -dimensional Gaussian with a diagonal covariance matrix of the form p ( α ) = N ( α|0 , c−1IM ) , ( 4 ) the joint probability distribution for any finite ( even infinite ) set of variables , x ≜ ( x ( t1 ) , . . . , x ( tJ ) ) ⊤ for J ≤ ∞ , follows the J-dimensional Gaussian given by p ( x ) dx = N ( x|µ1 , K ) dx , Kjj′ = k ( tj , tj′ ) ≜ c−1ϕ ( tj ) ⊤ϕ ( tj′ ) , ( 5 ) where k ( t , t′ ) is a semi-definite kernel function , c is the precision parameter , 1 ≤ j , j′ ≤ J , and 1 ≜ ( 1 , . . . , 1 ) ⊤ . Because ( 5 ) holds for arbitrary numbers of dimensions , J , it provides an implicit representation of GP prior . Next , based on the expression ( 5 ) , we derive a more explicit representation of GP prior . For simplicity of explanation , we assume that the domain T is one-dimensional , t = t and T = [ 0 , T ] , and mean µ is zero . Let K be the integral operator corresponding to k ( t , t′ ) , and K∗ be the inverse operator of K , K ( ∗ ) x ( t ) ≜ ∫ T k ( ∗ ) ( t , t′ ) x ( t′ ) dt′ , K∗k ( t , s ) = ∫ T k∗ ( t , t′ ) k ( t′ , s ) dt′ = δ ( t− s ) , ( 6 ) where δ ( · ) represents the Dirac delta function . Note that K∗ and k∗ ( t , t′ ) should be described by a differential operator because relation ( 6 ) indicates that k ( t , s ) is the Green function [ 15 ] for the operator K∗ , but we do not go into detail here . Under the finite scenario ( 5 ) with [ t0 , tJ ] = [ 0 , T ] and J ≫ 1 , the second equation in ( 6 ) can be approximated by K∗∆K = ∆−1 , where ∆jj′ = ( tj − tj−1 ) δjj′ , K∗jj′ = k∗ ( tj , tj′ ) , and δjj′ is the Kronecker delta . Then we can write down the J-dimensional Gaussian ( 5 ) by using K∗ as p ( x ) dx = √ |∆K∗∆| ( 2π ) J exp [ −1 2 ( ∆x ) ⊤K∗ ( ∆x ) ] dx . ( 7 ) By taking the limit of the division number J → ∞ ( ∆ → 0 ) for the finite-dimensional Gaussian measure ( 7 ) , we achieve the following path integral representation of GP prior over the latent function x ( t ) : GP ( x ( t ) |0 , k ) Dx ≜ lim J→∞ p ( x ) J∏ j=1 dx ( tj ) = √ 1 |K| exp [ −1 2 ∫∫ T ×T k∗ ( t , s ) x ( t ) x ( s ) dtds ] Dx , ( 8 ) where |K| ≜ limJ→∞ |K∆| , Dx ≜ limJ→∞ ∏J j=1 √ ( tj−tj−1 ) / ( 2π ) dx ( tj ) , and the following path integral holds , ∫ exp [ −1 2 ∫∫ T ×T k∗ ( t , s ) x ( t ) x ( s ) dtds ] Dx = √ |K| . ( 9 ) |K| represents the Fredholm determinant or functional determinant [ 19 ] of the integral operator K , which is defined by the product of its eigenvalues . See the supplementary material ( §1 ) for the derivation . The expression for GP ( x ( t ) |µ , k ) is easily recovered by x ( t ) → x ( t ) − µ and t → t in ( 8 ) . The path integral representation ( 8-9 ) has a clear advantage over the standard one ( 5 ) in that the distribution of the latent function x ( t ) is written in terms of the explicit integral of x ( t ) over the domain T , as with the likelihood function ( 2 ) , which makes it possible to apply functional analyses , calculus of variation , to the Bayesian estimation ( 1 ) . To the best of our knowledge , this is the first proposal of a path integral representation for GP prior in the machine learning community . It should be noted that our derivation of the path integral representation is based on an intuitive view reminiscent of Feynman ’ s path integral formulation of the quantum field theory [ 17 ] , and thus is not mathematically rigorous . In this paper , we confirm that our path integral expression is valid by using experiments rather than taking a more mathematically rigorous approach ; we discuss this briefly in Section 4 . 2.3 Maximum A Posteriori Estimation Using the representation ( 8 ) and the likelihood function ( 2 ) , we can rewrite the posterior ( 1 ) as the functional , p ( x ( t ) |D ) Dx = 1 p ( D ) exp [ −S ( x ( t ) , x ( t ) ) ] Dx , ( 10 ) where S ( x ( t ) , x ( t ) ) is an action integral , defined by S ( x ( t ) , x ( t ) ) ≜ ∫ T [ 1 2 ( x ( t ) −µ ) x ( t ) +κ ( x ( t ) ) − N∑ n=1 log κ ( x ( t ) ) δ ( t−tn ) ] dt+ 1 2 log |K| , ( 11 ) and x ( t ) ≜ ∫ T k ∗ ( t , t′ ) ( x ( t′ ) −µ ) dt′ = K∗ ( x − µ ) . ( 10 ) indicates that MAP estimation of x ( t ) is equivalent to the problem of finding the most likely function or path x ( t ) that minimizes the action integral . Thus we now apply calculus of variations to the action integral , where the functional derivative of S ( x ( t ) , x ( t ) ) on the MAP estimator x̂ ( t ) is equal to zero : δSδx̂ ( t ) δx̂ ( t ) + δS δx̂ ( t ) δx̂ ( t ) = 0 . This leads us to realize the following equation for deriving the MAP estimator x̂ ( t ) , x̂ ( t ) + ∫ T k ( t , t′ ) κ̇ ( x̂ ( t′ ) ) dt′ = µ+ N∑ n=1 k ( t , tn ) κ̇ ( x̂ ( tn ) ) κ ( x̂ ( tn ) ) , t ∈ T , ( 12 ) where κ̇ ( x ) ≜ dκ ( x ) /dx . See the supplementary material ( §2 ) for the detailed derivations of ( 12 ) .

Summary:    The manuscript tackles the problem of scalable and tractable approximate inference for Gaussian Cox models using a path integral approach. Experiments on synthetic 1D data and on 2D neural data illustrate certain properties of the approach.     Strengths    + Gaussian Cox processes are a relevant class of Bayesian models that are computationally intractable and hence a meaningful research objective.      Weaknesses    - The paper does not come with code; hence it will be hard to reproduce the results.    - The data sets considered in the main paper are rather small compared to [24]. A direct comparison to [24] in the main paper would have been instructive.    - The empirical evaluation does not support a solid claim on the runtime and memory footprint of the method.    - An empirical comparison to previous methods e.g. [24] is only partly available.