Additive Poisson Process: Learning Intensity of Higher-Order Interaction in Poisson Processes

1 INTRODUCTION . The Poisson process is a counting process used in a wide range of disciplines such as spatialtemporal sequential data in transportation ( Zhou et al. , 2018 ; 2021 ) , finance ( Ilalan , 2016 ) and ecology ( Thompson , 1955 ) to model the arrival rate by learning an intensity function . For a given time interval , the integral of the intensity function represents the average number of events occurring in that interval . The intensity function can be generalized to multiple dimensions . However , for most practical applications , learning the multi-dimensional intensity function becomes a challenge because of the sparse observations . Despite the recent advances of Poisson processes , most current Poisson process models are unable to learn the intensity function of a multi-dimensional Poisson process . Our research question is , “ Are there any good ways of approximating the high dimensional intensity function ? ” Our proposed model , the Additive Poisson Process ( APP ) , provides a novel solution to this problem . Throughout this paper , we will use a running example in a spatial-temporal setting . Say we want to learn the intensity function for a taxi to pick up customers at a given time and location . For this setting , each event is multi-dimensional ; that is , ( xi , yi , Wi ) , where a pair of xi and yi represents two spatial coordinates and Wi represents the day of the week . In addition , observation time is associated with each component of the event , which is represented as ti = ( txi , tyi , tWi ) . In this example , it is reasonable to assume txi = tyi = tWi as they are observed simultaneously . For any given location or time , we can expect at most a few pick-up events , which makes it difficult for any model to learn the low-valued intensity function . Figure 2b visualizes this problem . In this problem setup , if we would like to learn the intensity function at a given location ( x , y ) and day of the week W , the naı̈ve approach would be to learn the intensity at ( x , y , W ) directly from observations . This is extremely difficult because there could be only few events for a given location and day . However , there is useful information in lower-dimensional space ; for example , the marginalized observations at the location ( xi , yi ) across all days of the week , or on the day Wi at all locations . This information can be included into the model to improve the estimation of the joint intensity function . Using the information in lower-dimensional space provides a structured approach to include prior information based on the location or day of the week to improve the estimation of the joint intensity function . For example , a given location could be a shopping center or a hotel , where it is common for taxis to pick up passengers , and therefore we expect more passengers at this location . There could also be additional patterns that could be uncovered based on the day of the week . We can then use the observations of events to update our knowledge of the intensity function . Previous approaches , such as kernel density estimation ( KDE ) ( Rosenblatt , 1956 ) , are able to learn the joint intensity function by using information in lower dimensions . However , KDE suffers from the curse of dimensionality , which means that it requires a large size of samples to build an accurate model . In addition , the complexity of the model expands exponentially with respect to the number of dimensions , which makes it infeasible to compute . Bayesian approaches , such as using a mixture of beta distributions with a Dirichlet prior ( Kottas , 2006 ; Kottas & Sansó , 2007 ) and Reproducing Kernel Hilbert Space ( RKHS ) ( Flaxman et al. , 2017 ) , have been proposed to quantify the uncertainty for the intensity function . However , these approaches are often non-convex , making it difficult to obtain the globally optimal solution . Besides , if observations are sparse , it is hard for these approaches to learn a reasonable intensity function . Additional related work about Bayesian inference for Poisson processes and Poisson factorization can be found in Appendix A . In this paper , we propose a novel framework to learn the higher-order interaction effects of intensity functions in Poisson processes . Our model combines the techniques introduced by Luo & Sugiyama ( 2019 ) to model higher-order interactions between Poisson processes and by Friedman & Stuetzle ( 1981 ) in generalized additive models to learn the joint intensity function using samples in a lower dimensional space . Our proposed approach is to decompose a multi-dimensional Poisson process into lower-dimensional representations . For example , we have points ( xi ) Ni=1 in the x-dimension and ( yi ) Ni=1 in the y-dimension . Such data in the lower-dimensional space can be used to improve the estimation of the joint intensity function . This is different from the traditional approaches where only the joint occurrence of events is used to learn the joint intensity . We first show the connection between generalized additive models and Poisson processes , and then provide the connection between generalized additive models and the log-linear model ( Agresti , 2012 ) , which has a well-established theoretical background in information geometry ( Amari , 2016 ) . We draw parallels between the formulation of the generalized additive models and the binary loglinear model on a partially ordered set ( poset ) ( Sugiyama et al. , 2017 ) . The learning process in our model is formulated as a convex optimization problem to arrive at a unique optimal solution using natural gradient , which minimizes the Kullback-Leibler ( KL ) divergence from the sample distribution in a lower-dimensional space to the distribution modeled by the learned intensity function . This connection provides remarkable properties to our model : the ability to learn higher-order intensity functions using lower-dimensional projections , thanks to the Kolmogorov-Arnold representation theorem . This property makes it advantageous to use our proposed approach for cases where there are no observations , missing samples , or low event rates . Our model is flexible because it can capture the interaction effects between events in a Poisson process as a partial order structure in the log-linear model and the parameters of the model are fully customizable to meet the requirements of the application . Our empirical results show that our model effectively uses samples projected onto a lower dimensional space to estimate the higher-order intensity function . More importantly , our model is also robust to various sample sizes . 2 FORMULATION . We start this section by introducing the technical background in the Poisson process and its extension to a multi-dimensional Poisson process . We then introduce the Generalized Additive Model ( GAM ) and its connection to the Poisson process . This is followed by presenting our novel framework , called Additive Poisson Process ( APP ) , which is our main technical contribution and has a tight link to the Poisson process modeled by GAMs . We show that the learning of APP can be achieved via convex optimization using natural gradient . The Poisson process is characterized by an intensity function λ : R → R. An inhomogeneous Poisson process is an extension of a homogeneous Poisson process , where the arrival rate changes with time . The process with time-changing intensity λ ( t ) is defined as a counting process N ( t ) , which has an independent increment property . For any time t ≥ 0 and infinitesimal interval δ ≥ 0 , the probability of events count is p ( N ( t+δ ) −N ( t ) = 0 ) = 1−δλ ( t ) +o ( δ ) , p ( N ( t+δ ) −N ( t ) = 1 ) = δλ ( t ) +o ( δ ) , and p ( N ( t+δ ) −N ( t ) ≥ 2 ) = o ( δ ) , where o ( · ) denotes little-o notation ( Daley & VereJones , 2007 ) . This formulation can be generalized into the ( inhomogeneous ) multi-dimensional Poisson process , which can be used to learn the intensity function λ : RD → R from a realization of timestamps t1 , t2 , . . . , tN with ti ∈ [ 0 , T ] D. Each ti is the time of occurrence for the i-th event across D dimensions and T is the observation duration . The likelihood for the multi-dimensional Poisson process ( Daley & Vere-Jones , 2007 ) is given by p ( { ti } Ni=1 | λ ( t ) ) = exp ( − ∫ λ ( t ) dt ) N∏ i=1 λ ( ti ) , ( 1 ) where t = ( t ( 1 ) , . . . , t ( D ) ) ∈ RD . We define the functional prior on λ ( t ) as λ ( t ) : = g ( f ( t ) ) = exp ( f ( t ) ) . ( 2 ) The function g ( · ) is a positive function to guarantee the non-negativity of the intensity which we choose to be the exponential function , and our objective is to learn the function f ( · ) . The loglikelihood of the multi-dimensional Poisson process with the functional prior is described as log p ( { ti } Ni=1 |λ ( t ) ) = N∑ i=1 f ( ti ) − ∫ exp ( f ( t ) ) dt . ( 3 ) In the following sections , we introduce the generalized additive models and propose to model it by the log-linear model to learn f ( t ) and the normalizing term ( ∫ exp ( f ( t ) ) dt ) . 2.1 GENERALIZED ADDITIVE MODEL . In this section , we present the connection between Poisson processes and the Generalized Additive Model ( GAM ) proposed by Friedman & Stuetzle ( 1981 ) . The GAM projects higher-dimensional features into lower-dimensional space to apply smoothing functions to build a restricted class of non-parametric regression models . GAM is less affected by the curse of dimensionality compared to directly using smoothing in a higher-dimensional space . For a given set of processes J ⊆ [ D ] = { 1 , . . . , D } , the traditional GAM using one-dimensional projections is defined as log λJ ( t ) = ∑ j∈J fj ( t ( j ) ) − βJ with some smoothing function fj . In this paper , we extend it to include higher-order interactions between features in GAM by introducing terms that represent the multiplicative product between events . The k-th order GAM is defined as log λJ ( t ) = ∑ j∈J f { j } ( t ( j ) ) + ∑ j1 , j2∈J f { j1 , j2 } ( t ( j1 ) , t ( j2 ) ) + · · ·+ ∑ j1 , ... , jk∈J f { j1 , ... , jk } ( t ( j1 ) , . . . , t ( jk ) ) − βJ = ∑ I⊆J , |I|≤k fI ( t ( I ) ) − βJ , ( 4 ) where t ( I ) ∈ R|I| denotes the subvector ( t ( j ) ) j∈I of t with respect to I ⊆ [ D ] . The function fI : R|I| → R is a smoothing function to fit the data , and the normalization constant βJ for the intensity function is obtained as βJ = ∫ λJ ( t ) dt = ∫ exp ( ∑ I⊆J , |I|≤k fI ( t ( I ) ) ) dt . The definition of the additive model is in the same form as Equation ( 3 ) . In particular , if we compare Equation ( 3 ) and Equation ( 4 ) , we can see that the smoothing function f in Equation ( 3 ) is realized as the summation over lower-dimensional projections in Equation ( 4 ) . Learning of a continuous function using lower-dimensional projections is well known because of the Kolmogorov-Arnold representation theorem , which states that : Theorem 1 ( Kolmogorov–Arnold Representation Theorem ( Braun & Griebel , 2009 ; Kolmogorov , 1957 ) ) . Any multivariate continuous function can be represented as a superposition of one–dimensional functions ; that is , f ( t1 , . . . , tn ) = ∑2n+1 q=1 fq ( ∑n p=1 gq , p ( tp ) ) . Braun ( 2009 ) showed that the GAM is an approximation to the general form presented in Kolmogorov-Arnold representation theorem by replacing the range q ∈ { 1 , . . . , 2n + 1 } with I ⊆ J and the inner function gq , p by the identity if q = p and zero otherwise , yielding f ( t ) = ∑ I⊆J fI ( t ( I ) ) . Interestingly , the canonical form for additive models in Equation ( 4 ) can be rearranged to be in the same form as Kolmogorov-Arnold representation theorem . By letting f ( t ) = ∑ I⊆J fI ( t ( I ) ) = g−1 ( λ ( t ) ) and g ( · ) = exp ( · ) , we have λJ ( t ) = 1 exp ( βJ ) exp ( ∑ I⊆J fI ( t ( I ) ) ) ∝ exp ( ∑ I⊆J fI ( t ( I ) ) ) , ( 5 ) where we assume fI ( t ( I ) ) = 0 if |I| > k for the k-th order model and 1/ exp ( βJ ) is the normalization term for the intensity function . Based on the Kolmogorov-Arnold representation theorem , generalized additive models are able to learn the intensity of the higher-order interaction between Poisson processes by using projections into lower dimensional spaces . The log-likelihood function for a kth-order model is obtained by substituting λJ ( t ) in Equation ( 4 ) into λ ( t ) in Equation ( 3 ) , log p ( { t } Ni=1|λ ( t ) ) = N∑ i=1 ∑ I⊆J , |I|≤k fI ( t ( I ) ) − β′ , ( 6 ) where β′ is a constant given by β′ = ∫ λ ( t ) dt+ ∑ I⊆J βJ . In the following subsection we introduce a log-linear formulation equipped with partially ordered sample space , which aligns with the GAM formulation in Equation ( 5 ) .

Summary: This paper considers the problem of estimating the intensity function of a multi-dimensional Poisson process. Inspired by the generalized additive model, the authors propose a certain non-parametric form for the log-likelihood of the intensity function, involving multiple functions of groups of dimensions. Further, the higher-order interactions among the dimensions of the Poisson process are restricted to be within a small order k, so only functions of up to k of the dimensions are considered. Considering lower-order interactions naturally leads to a partial order, and hence the authors propose a log-linear model. For each element in the discretized state space, there is a parameter to be estimated. In turn, searching for the parameters amounts to solving a KL divergence minimization problem, which can be solved using convex optimization. Experiments on synthetic data as well as real data (New York taxis) are included. The experiments investigate the performance of the new approach, compared with three existing methods from the literature.