Additive Poisson Process: Learning Intensity of Higher-Order Interaction in Poisson Processes

1 INTRODUCTION . The Poisson process is a counting process used in a wide range of disciplines such as spatialtemporal sequential data in transportation ( Zhou et al. , 2018 ; 2021 ) , finance ( Ilalan , 2016 ) and ecology ( Thompson , 1955 ) to model the arrival rate by learning an intensity function . For a given time interval , the integral of the intensity function represents the average number of events occurring in that interval . The intensity function can be generalized to multiple dimensions . However , for most practical applications , learning the multi-dimensional intensity function becomes a challenge because of the sparse observations . Despite the recent advances of Poisson processes , most current Poisson process models are unable to learn the intensity function of a multi-dimensional Poisson process . Our research question is , “ Are there any good ways of approximating the high dimensional intensity function ? ” Our proposed model , the Additive Poisson Process ( APP ) , provides a novel solution to this problem . Throughout this paper , we will use a running example in a spatial-temporal setting . Say we want to learn the intensity function for a taxi to pick up customers at a given time and location . For this setting , each event is multi-dimensional ; that is , ( xi , yi , Wi ) , where a pair of xi and yi represents two spatial coordinates and Wi represents the day of the week . In addition , observation time is associated with each component of the event , which is represented as ti = ( txi , tyi , tWi ) . In this example , it is reasonable to assume txi = tyi = tWi as they are observed simultaneously . For any given location or time , we can expect at most a few pick-up events , which makes it difficult for any model to learn the low-valued intensity function . Figure 2b visualizes this problem . In this problem setup , if we would like to learn the intensity function at a given location ( x , y ) and day of the week W , the naı̈ve approach would be to learn the intensity at ( x , y , W ) directly from observations . This is extremely difficult because there could be only few events for a given location and day . However , there is useful information in lower-dimensional space ; for example , the marginalized observations at the location ( xi , yi ) across all days of the week , or on the day Wi at all locations . This information can be included into the model to improve the estimation of the joint intensity function . Using the information in lower-dimensional space provides a structured approach to include prior information based on the location or day of the week to improve the estimation of the joint intensity function . For example , a given location could be a shopping center or a hotel , where it is common for taxis to pick up passengers , and therefore we expect more passengers at this location . There could also be additional patterns that could be uncovered based on the day of the week . We can then use the observations of events to update our knowledge of the intensity function . Previous approaches , such as kernel density estimation ( KDE ) ( Rosenblatt , 1956 ) , are able to learn the joint intensity function by using information in lower dimensions . However , KDE suffers from the curse of dimensionality , which means that it requires a large size of samples to build an accurate model . In addition , the complexity of the model expands exponentially with respect to the number of dimensions , which makes it infeasible to compute . Bayesian approaches , such as using a mixture of beta distributions with a Dirichlet prior ( Kottas , 2006 ; Kottas & Sansó , 2007 ) and Reproducing Kernel Hilbert Space ( RKHS ) ( Flaxman et al. , 2017 ) , have been proposed to quantify the uncertainty for the intensity function . However , these approaches are often non-convex , making it difficult to obtain the globally optimal solution . Besides , if observations are sparse , it is hard for these approaches to learn a reasonable intensity function . Additional related work about Bayesian inference for Poisson processes and Poisson factorization can be found in Appendix A . In this paper , we propose a novel framework to learn the higher-order interaction effects of intensity functions in Poisson processes . Our model combines the techniques introduced by Luo & Sugiyama ( 2019 ) to model higher-order interactions between Poisson processes and by Friedman & Stuetzle ( 1981 ) in generalized additive models to learn the joint intensity function using samples in a lower dimensional space . Our proposed approach is to decompose a multi-dimensional Poisson process into lower-dimensional representations . For example , we have points ( xi ) Ni=1 in the x-dimension and ( yi ) Ni=1 in the y-dimension . Such data in the lower-dimensional space can be used to improve the estimation of the joint intensity function . This is different from the traditional approaches where only the joint occurrence of events is used to learn the joint intensity . We first show the connection between generalized additive models and Poisson processes , and then provide the connection between generalized additive models and the log-linear model ( Agresti , 2012 ) , which has a well-established theoretical background in information geometry ( Amari , 2016 ) . We draw parallels between the formulation of the generalized additive models and the binary loglinear model on a partially ordered set ( poset ) ( Sugiyama et al. , 2017 ) . The learning process in our model is formulated as a convex optimization problem to arrive at a unique optimal solution using natural gradient , which minimizes the Kullback-Leibler ( KL ) divergence from the sample distribution in a lower-dimensional space to the distribution modeled by the learned intensity function . This connection provides remarkable properties to our model : the ability to learn higher-order intensity functions using lower-dimensional projections , thanks to the Kolmogorov-Arnold representation theorem . This property makes it advantageous to use our proposed approach for cases where there are no observations , missing samples , or low event rates . Our model is flexible because it can capture the interaction effects between events in a Poisson process as a partial order structure in the log-linear model and the parameters of the model are fully customizable to meet the requirements of the application . Our empirical results show that our model effectively uses samples projected onto a lower dimensional space to estimate the higher-order intensity function . More importantly , our model is also robust to various sample sizes . 2 FORMULATION . We start this section by introducing the technical background in the Poisson process and its extension to a multi-dimensional Poisson process . We then introduce the Generalized Additive Model ( GAM ) and its connection to the Poisson process . This is followed by presenting our novel framework , called Additive Poisson Process ( APP ) , which is our main technical contribution and has a tight link to the Poisson process modeled by GAMs . We show that the learning of APP can be achieved via convex optimization using natural gradient . The Poisson process is characterized by an intensity function λ : R → R. An inhomogeneous Poisson process is an extension of a homogeneous Poisson process , where the arrival rate changes with time . The process with time-changing intensity λ ( t ) is defined as a counting process N ( t ) , which has an independent increment property . For any time t ≥ 0 and infinitesimal interval δ ≥ 0 , the probability of events count is p ( N ( t+δ ) −N ( t ) = 0 ) = 1−δλ ( t ) +o ( δ ) , p ( N ( t+δ ) −N ( t ) = 1 ) = δλ ( t ) +o ( δ ) , and p ( N ( t+δ ) −N ( t ) ≥ 2 ) = o ( δ ) , where o ( · ) denotes little-o notation ( Daley & VereJones , 2007 ) . This formulation can be generalized into the ( inhomogeneous ) multi-dimensional Poisson process , which can be used to learn the intensity function λ : RD → R from a realization of timestamps t1 , t2 , . . . , tN with ti ∈ [ 0 , T ] D. Each ti is the time of occurrence for the i-th event across D dimensions and T is the observation duration . The likelihood for the multi-dimensional Poisson process ( Daley & Vere-Jones , 2007 ) is given by p ( { ti } Ni=1 | λ ( t ) ) = exp ( − ∫ λ ( t ) dt ) N∏ i=1 λ ( ti ) , ( 1 ) where t = ( t ( 1 ) , . . . , t ( D ) ) ∈ RD . We define the functional prior on λ ( t ) as λ ( t ) : = g ( f ( t ) ) = exp ( f ( t ) ) . ( 2 ) The function g ( · ) is a positive function to guarantee the non-negativity of the intensity which we choose to be the exponential function , and our objective is to learn the function f ( · ) . The loglikelihood of the multi-dimensional Poisson process with the functional prior is described as log p ( { ti } Ni=1 |λ ( t ) ) = N∑ i=1 f ( ti ) − ∫ exp ( f ( t ) ) dt . ( 3 ) In the following sections , we introduce the generalized additive models and propose to model it by the log-linear model to learn f ( t ) and the normalizing term ( ∫ exp ( f ( t ) ) dt ) . 2.1 GENERALIZED ADDITIVE MODEL . In this section , we present the connection between Poisson processes and the Generalized Additive Model ( GAM ) proposed by Friedman & Stuetzle ( 1981 ) . The GAM projects higher-dimensional features into lower-dimensional space to apply smoothing functions to build a restricted class of non-parametric regression models . GAM is less affected by the curse of dimensionality compared to directly using smoothing in a higher-dimensional space . For a given set of processes J ⊆ [ D ] = { 1 , . . . , D } , the traditional GAM using one-dimensional projections is defined as log λJ ( t ) = ∑ j∈J fj ( t ( j ) ) − βJ with some smoothing function fj . In this paper , we extend it to include higher-order interactions between features in GAM by introducing terms that represent the multiplicative product between events . The k-th order GAM is defined as log λJ ( t ) = ∑ j∈J f { j } ( t ( j ) ) + ∑ j1 , j2∈J f { j1 , j2 } ( t ( j1 ) , t ( j2 ) ) + · · ·+ ∑ j1 , ... , jk∈J f { j1 , ... , jk } ( t ( j1 ) , . . . , t ( jk ) ) − βJ = ∑ I⊆J , |I|≤k fI ( t ( I ) ) − βJ , ( 4 ) where t ( I ) ∈ R|I| denotes the subvector ( t ( j ) ) j∈I of t with respect to I ⊆ [ D ] . The function fI : R|I| → R is a smoothing function to fit the data , and the normalization constant βJ for the intensity function is obtained as βJ = ∫ λJ ( t ) dt = ∫ exp ( ∑ I⊆J , |I|≤k fI ( t ( I ) ) ) dt . The definition of the additive model is in the same form as Equation ( 3 ) . In particular , if we compare Equation ( 3 ) and Equation ( 4 ) , we can see that the smoothing function f in Equation ( 3 ) is realized as the summation over lower-dimensional projections in Equation ( 4 ) . Learning of a continuous function using lower-dimensional projections is well known because of the Kolmogorov-Arnold representation theorem , which states that : Theorem 1 ( Kolmogorov–Arnold Representation Theorem ( Braun & Griebel , 2009 ; Kolmogorov , 1957 ) ) . Any multivariate continuous function can be represented as a superposition of one–dimensional functions ; that is , f ( t1 , . . . , tn ) = ∑2n+1 q=1 fq ( ∑n p=1 gq , p ( tp ) ) . Braun ( 2009 ) showed that the GAM is an approximation to the general form presented in Kolmogorov-Arnold representation theorem by replacing the range q ∈ { 1 , . . . , 2n + 1 } with I ⊆ J and the inner function gq , p by the identity if q = p and zero otherwise , yielding f ( t ) = ∑ I⊆J fI ( t ( I ) ) . Interestingly , the canonical form for additive models in Equation ( 4 ) can be rearranged to be in the same form as Kolmogorov-Arnold representation theorem . By letting f ( t ) = ∑ I⊆J fI ( t ( I ) ) = g−1 ( λ ( t ) ) and g ( · ) = exp ( · ) , we have λJ ( t ) = 1 exp ( βJ ) exp ( ∑ I⊆J fI ( t ( I ) ) ) ∝ exp ( ∑ I⊆J fI ( t ( I ) ) ) , ( 5 ) where we assume fI ( t ( I ) ) = 0 if |I| > k for the k-th order model and 1/ exp ( βJ ) is the normalization term for the intensity function . Based on the Kolmogorov-Arnold representation theorem , generalized additive models are able to learn the intensity of the higher-order interaction between Poisson processes by using projections into lower dimensional spaces . The log-likelihood function for a kth-order model is obtained by substituting λJ ( t ) in Equation ( 4 ) into λ ( t ) in Equation ( 3 ) , log p ( { t } Ni=1|λ ( t ) ) = N∑ i=1 ∑ I⊆J , |I|≤k fI ( t ( I ) ) − β′ , ( 6 ) where β′ is a constant given by β′ = ∫ λ ( t ) dt+ ∑ I⊆J βJ . In the following subsection we introduce a log-linear formulation equipped with partially ordered sample space , which aligns with the GAM formulation in Equation ( 5 ) .

Summary: This paper proposes a method to estimate the multi-dimensional intensity function of a Poisson process with a lower-dimensional projection. The proposed method is motivated by perspectives of information geometry and generalized additive models in statistics. The performance of the proposed method is extreme through synthetic examples and a real data analysis.