Additive Poisson Process: Learning Intensity of Higher-Order Interaction in Stochastic Processes

We present the Additive Poisson Process ( APP ) , a novel framework that can model the higher-order interaction effects of the intensity functions in point processes using lower dimensional projections . Our model combines the techniques in information geometry to model higher-order interactions on a statistical manifold and in generalized additive models to use lower-dimensional projections to overcome the effects from the curse of dimensionality . Our approach solves a convex optimization problem by minimizing the KL divergence from a sample distribution in lower dimensional projections to the distribution modeled by an intensity function in the point process . Our empirical results show that our model is able to use samples observed in the lower dimensional space to estimate the higher-order intensity function with extremely sparse observations . 1 INTRODUCTION . Consider two point processes which are correlated with arrival times for an event . For a given time interval , what is the probability of observing an event from both processes ? Can we learn the joint intensity function by just using the observations from each individual processes ? Our proposed model , the Additive Poisson Process ( APP ) , provides a novel solution to this problem . The Poisson process is a counting process used in a wide range of disciplines such as time-space sequence data including transportation ( Zhou et al. , 2018 ) , finance ( Ilalan , 2016 ) , ecology ( Thompson , 1955 ) , and violent crime ( Taddy , 2010 ) to model the arrival times for a single system by learning an intensity function . For a given time interval of the intensity function , it represents the probability of a point being excited at a given time . Despite the recent advances of modeling of the Poisson processes and its wide applicability , majority of the point processes model do not consider the correlation between two or more point processes . Our proposed approach learns the joint intensity function of the point process which is defined to be the simultaneous occurrence of two events . For example in a spatial-temporal problem , we want to learn the intensity function for a taxi to pick up customers at a given time and location . For this problem , each point is multi-dimensional , that is ( x , y , t ) Ni=1 , where a pair of x and y represents two spatial dimensions and t represents the time dimension . For any given location or time , we can only expect very few pick-up events occurring , therefore making it difficult for any model to learn the low valued intensity function . Previous approaches such as Kernel density estimation ( KDE ) ( Rosenblatt , 1956 ) are able to learn the joint intensity function . However , KDE suffers from the curse of dimensionality , which means that KDE requires a large size sample or a high intensity function to build an accurate model . In addition , the complexity of the model expands exponentially with respect to the number of dimensions , which makes it infeasible to compute . Bayesian approaches such as using a mixture of beta distributions with a Dirichlet prior ( Kottas , 2006 ) and Reproducing Kernel Hilbert Space ( RKHS ) ( Flaxman et al. , 2017 ) have been proposed to quantify the uncertainty with a prior for the intensity function . However , these approaches are often non-convex , making it difficult to obtain the global optimal solution . In addition , if observations are sparse , it is hard for these approaches to learn a reasonable intensity function . All previous models are unable to efficiently and accurately learn the intensity of the interaction between point processes . This is because the intensity of the joint process is often low , leading to sparse samples or , in an extreme case , no direct observations of the simultaneous event at all , making it difficult to learn the intensity function from the joint samples . In this paper , we propose a novel framework to learn the higher-order interaction effects of intensity functions in point processes . Our model combines the techniques introduced by Luo & Sugiyama ( 2019 ) to model higher-order interactions between point processes and by Friedman & Stuetzle ( 1981 ) in generalized additive models to learn the intensity function using samples in a lower dimensional space . Our proposed approach is to decompose a multi-dimensional point process into lower-dimensional representations . For example , in the x-dimension we have points ( xi ) Ni=1 , in the y-dimension , we have points ( yi ) N i=1 and in the time dimension we have ( ti ) Ni=1 . The data in these lower dimensional space can be used to improve the estimate of the joint intensity function . This is different from the traditional approach where we only use the simultaneous events to learn the joint intensity function . We first show the connection between generalized additive models and Poisson processes . We then provide the connection between generalized additive models and the log-linear model ( Agresti , 2012 ) , which has a well-established theoretical background in information geometry ( Amari , 2016 ) . We draw parallels between the formulation of the generalized additive models and the binary loglinear model on a partially ordered set ( poset ) ( Sugiyama et al. , 2017 ) . The learning process in our model is formulated as a convex optimization problem to arrive at a unique optimal solution using natural gradient , which minimizes the Kullback-Leibler ( KL ) divergence from the sample distribution in a lower dimensional space to the distribution modeled by the learned intensity function . This connection provides remarkable properties to our model : the ability to learn higher-order intensity functions using lower dimensional projections , thanks to the Kolmogorov-Arnold representation theorem . This property makes it advantageous to use our proposed approach for the cases where there are , no observations , missing samples , or low event rate . Our model is flexible because it can capture interaction between processes as a partial order structure in the log-linear model and the parameters of the model are fully customizable to meet the requirements for the application . Our empirical results show that our model effectively uses samples projected onto a lower dimensional space to estimate the higher-order intensity function . Our model is also robust to various sample sizes . 2 FORMULATION . In this section we first introduce the technical background in the Poisson process and its extension to a multi-dimensional Poisson process . We then introduce the Generalized Additive Model ( GAM ) and its connection to the Poisson process . This is followed by presenting our novel framework , called Additive Poisson Process ( APP ) , which is our main technical contribution and has a tight link to the Poisson process modelled by GAMs . We show that learning of APP can be achieved via convex optimization using natural gradient . The Poisson process is characterized by an intensity function λ : RD → R , where we assume multiple D processes . An inhomogeneous Poisson process is a general type of processes , where the arrival intensity changes with time . The process with time-changing intensity λ ( t ) is defined as a counting process N ( t ) , which has an independent increment property . For all time t ≥ 0 and changes in time δ ≥ 0 , the probability p for the observations is given as p ( N ( t+ δ ) −N ( t ) = 0 ) = 1− δλ ( t ) + o ( δ ) , p ( N ( t + δ ) − N ( t ) = 1 ) = δλ ( t ) + o ( δ ) , and p ( N ( t + δ ) − N ( t ) ≥ 2 ) = o ( δ ) , where o ( · ) denotes little-o notation ( Daley & Vere-Jones , 2007 ) . Given a realization of timestamps t1 , t2 , . . . , tN with ti ∈ [ 0 , T ] D from an inhomogeneous ( multi-dimensional ) Poisson process with the intensity λ . Each ti is the time of occurrence for the i-th event across D processes and T is the observation duration . The likelihood for the Poisson process ( Daley & Vere-Jones , 2007 ) is given by p ( { ti } Ni=1 | λ ( t ) ) = exp ( − ∫ λ ( t ) dt ) N∏ i=1 λ ( ti ) , ( 1 ) where t = [ t ( 1 ) , . . . , t ( D ) ] ∈ RD . We define the functional prior on λ ( t ) as λ ( t ) : = g ( f ( t ) ) = exp ( f ( t ) ) . ( 2 ) The function g ( · ) is a positive function to guarantee the non-negativity of the intensity which we choose to be the exponential function , and our objective is to learn the function f ( · ) . The log- likelihood of the multi-dimensional Poisson process with the functional prior is described as log p ( { ti } Ni=1 | λ ( t ) ) = N∑ i=1 f ( ti ) − ∫ exp ( f ( t ) ) dt . ( 3 ) In the following sections , we introduce generalized additive models and propose to model it by the log-linear model to learn f ( t ) and the normalizing term . 2.1 GENERALIZED ADDITIVE MODEL . In this section we present the connection between Poisson processes with Generalized Additive Model ( GAM ) proposed by Friedman & Stuetzle ( 1981 ) . GAM projects higher-dimensional features into lower-dimensional space to apply smoothing functions to build a restricted class of non-parametric regression models . GAM is less affected by the curse of dimensionality compared to directly using smoothing in a higher-dimensional space . For a given set of processes J ⊆ [ D ] = { 1 , . . . , D } , the traditional GAM using one-dimensional projections is defined as log λJ ( t ) = ∑ j∈J fj ( t ( j ) ) − βJ with some smoothing function fj . In this paper , we extend it to include higher-order interactions between features in GAM . The k-th order GAM is defined as log λJ ( t ) = ∑ j∈J f { j } ( t ( j ) ) + ∑ j1 , j2∈J f { j1 , j2 } ( t ( j1 ) , t ( j2 ) ) + · · ·+ ∑ j1 , ... , jk∈J f { j1 , ... , jk } ( t ( j1 ) , . . . , t ( jk ) ) − βJ = ∑ I⊆J , |I|≤k fI ( t ( I ) ) − βJ , ( 4 ) where t ( I ) ∈ R|I| denotes the subvector ( t ( j ) ) j∈I of t with respect to I ⊆ [ D ] . The function fI : R|I| → R is a smoothing function to fit the data , and the normalization constant βJ for the intensity function is obtained as βJ = ∫ λJ ( t ) dt = ∫ exp ( ∑ I⊆J , |I|≤k fI ( t ( I ) ) ) dt . The definition of the additive model is in the same form as Equation ( 3 ) . In particular , if we compare Equation ( 3 ) and ( 4 ) , we can see that the smoothing function f in ( 3 ) corresponds to the right-hand side of ( 4 ) . Learning of a continuous function using lower dimensional projections is well known because of the Kolmogorov-Arnold representation theorem , which states as follows : Theorem 1 ( Kolmogorov–Arnold Representation Theorem ( Braun & Griebel , 2009 ; Kolmogorov , 1957 ) ) . Any multivariate continuous function can be represented as a superposition of one–dimensional functions , i.e. , f ( t1 , . . . , tn ) = ∑2n+1 q=1 fq ( ∑n p=1 gq , p ( tp ) ) . Braun ( 2009 ) showed that the GAM is an approximation to the general form presented in Kolmogorov-Arnold representation theorem by replacing the range q ∈ { 1 , . . . , 2n + 1 } with I ⊆ J and the inner function gq , p by the identity if q = p and zero otherwise , yielding f ( t ) = ∑ I⊆J fI ( t ( I ) ) . Interestingly , the canonical form for additive models in Equation ( 4 ) can be rearranged to be in the same form as Kolmogorov-Arnold representation theorem . By letting f ( t ) = ∑ I⊆J fI ( t ( I ) ) = g−1 ( λ ( t ) ) and g ( · ) = exp ( · ) , we have λJ ( t ) = 1 exp ( βJ ) exp ( ∑ I⊆J fI ( t ( I ) ) ) ∝ exp ( ∑ I⊆J fI ( t ( I ) ) ) , ( 5 ) where we assume fI ( t ( I ) ) = 0 if |I| > k for the k-th order model and 1/ exp ( βJ ) is the normalization term for the intensity function . Based on the Kolmogorov-Arnold representation theorem , generalized additive models are able to learn the intensity of the higher-order interaction between point processes by using projections into lower dimensional space . The log-likelihood function for a kth-order model is obtained by substituting the Equation ( 4 ) into Equation ( 1 ) , log p ( { t } Ni=1|λ ( t ) ) = N∑ i=1 exp ∑ I⊆J , |I|≤k fI ( t ( I ) ) − β′ , where is a constant given by β′ = ∫ λ ( t ) dt + ∑ I⊆J βJ . In the following section we will detail a log-linear formulation that efficiently maximizes this log-likelihood equation .

Summary: The paper under review proposes a new model for multi-dimensional temporal Point processes, allowing efficient estimation of high order interactions. This new model, called additive Poisson process, relies on a log-linear structure of the intensity function that is motivated thanks to the Kolmogorov-Arnold theorem. Such structure is then linked to generalized additive models, a result that is used to devise an efficient estimation procedure with formal guarantees  of convergence.