Self-Adaptable Point Processes with Nonparametric Time Decays

1 Introduction . Events of multiple types are ubiquitous , such as in online shopping , social networking and biological signal transduction . Understanding the complex influences of those events on each other , including excitation , inhibition , and how the strength of these influences varies with time , is crucial to discover useful knowledge and to predict future events and their types , which can benefit many applications , e.g. , marketing analysis , biological study , and early warnings of catastrophes . While many excellent works have been proposed for event modeling and analysis ( Blundell et al. , 2012 ; Linderman and Adams , 2014 ; Xu et al. , 2016 ; Tan et al. , 2016 ; Xu and Zha , 2017 ; Zhang et al. , 2020a ) , most of them are inadequate to capture complex influences between the events . The classical Poisson processes simply ignore these influences and assume independent increments . Although Hawkes processes ( HP ) can estimate the mutual excitation between the events , they ignore the inhibition effect , which is common in real world . More important , existing methods ( Xu et al. , 2016 ; Zhang et al. , 2020a ) , even for those considering both the excitation and inhibition , mostly assume an exponential time decay of the influence strength , which is over-simplified and can miss many other decay patterns . While recent works have also used recurrent neural networks ( RNNs ) ( Du et al. , 2016 ; Mei and Eisner , 2017 ) to model the event rate and hence are highly expressive , they couple 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . all the temporal dependencies in a black box , and are difficult to distill meaningful and important knowledge , e.g. , influence types and strengths . To address these limitations , we propose SPRITE , a multi-variate self-adaptable point process with nonparametric time decays . Our method not only can discover the type of the influence ( i.e. , excitation or inhibition ) between every pair of the events , but also is flexible enough to estimate all kinds of influence strength decaying patterns , not limited to the exponential decay or other particular forms . Specifically , we first introduce an embedding to represent each event type , and model the influence between any two events as an unknown function of the event type embeddings and time span . In so doing , we avoid separately estimating the influence between every pair of the events , and largely reduce the parameters . Next , we propose a general construction that covers all possible time decaying functions . We assign Gaussian process ( GP ) priors over the free latent functions and use Gauss-Legendra quadrature to calculate the integral in the construction . In this way , we can flexibly estimate a variety of time decaying influences while not needing to impose any derivative constraint , which can bring learning difficulties . For efficient inference , we use weight-space augmentation of GPs to prevent computing huge covariances matrices , and neural network feature mapping to enhance the learning capacity . We then use the re-reparameterization trick to develop a scalable stochastic variational learning algorithm . For evaluation , we first examined SPRITE in an ablation study . We tested the case of a single event type with excitation effects only and three time decays , and a bi-type event case with mixed excitation and inhibition effects . In both cases , our method accurately recovered the rate function of each type of events and the influence between the events , performing better than state-of-the-art methods based on or extending Hawkes processes and RNNs . Next , we examined SPRITE in three real-world and one synthetic benchmark datasets . We examined the accuracy in predicting the occurrence time and type of future events . In both tasks , our method nearly always outperforms the competing approaches , often by a large margin . Finally , by looking into the embeddings and influence functions learned by our model , we found interesting patterns within the event types and their interactions . 2 Background . Temporal Point Processes . We denote a sequence of events and their types by Γ = [ ( t1 , s1 ) , . . . , ( tM , sM ) ] , where each ti is the time point when the i-th event occurred ( t1 ≤ . . . ≤ tM ) , si is the type of the i-th event . Suppose we have K types of events in total ; so each si ∈ { 1 , . . . , K } . Temporal point processes ( TPPs ) ( Daley and Vere-Jones , 2007 ) provide a fundamental framework to characterize and model the events . A TPP is defined via the rate ( or conditional intensity ) function of the events , λs ( t ) . For example , a homogeneous Poisson process assumes each λs ( t ) is a constant λ0s , and the past events { ( tj , sj ) |tj < t } do not have any influence on generating a new event at t. While being simple and convenient , Poisson processes completely ignore the interactions between the events . Hawkes processes ( HPs ) ( Hawkes , 1971 ) overcome this issue by modeling the triggering effects between the events , λs ( t ) = λ0s + ∑ tj < t γsj→s ( t− tj ) where γsj→s ( ∆ ) > 0 is called “ triggering kernel ” , and characterizes how largely the previous event at tj encourages a new event of type s to occur at t. Naturally , this effect is supposed to decay with time ∆ . The most commonly used choice is the exponential decay , γs′→s ( ∆ ) = αs′s · exp ( −τs′s∆ ) , ( 1 ) where αs′s > 0 and τs′s > 0 . Given the definition of λs ( · ) , the probability of the event sequence is p ( Γ ) = ∏K k=1 exp ( − ∫ T 0 λk ( t ) dt ) ∏M i=1 λsi ( ti ) , ( 2 ) where T is the total span of the events in Γ and so tM ≤ T . RNN based Point Processes . HPs only consider the triggering effects and hence can be quite restricted . In order to estimate arbitrarily varying rate functions , recent works Du et al . ( 2016 ) ; Mei and Eisner ( 2017 ) use recurrent neural networks ( RNNs ) to model the event dependencies and rates . The key idea is as follows . For each event i , we introduce a hidden state vector β ( ti ) . The event dependency is modeled through the updating rule of the hidden states , β ( ti+1 ) = ρ ( β ( ti ) , ti+1 ) ( 3 ) where ρ ( · ) is a complicated mapping , fulfilled by a combination of neural network operations , such as linear transformation , nonlinear activation , and gating . Note that the event type si can also be incorporated into ρ ( · ) . The rate function is then defined as a transformation of β ( t ) . We can therefore evaluate and maximize the point process likelihood ( 2 ) for model estimation . 3 Model . While RNN based methods are highly expressive and can capture arbitrarily complex dependencies , due to the black-box updating rule ( see ( 3 ) ) , they can hardly distill meaningful knowledge , such as the influence type and strength between the events , which are critical for data analysis and knowledge discovery . On the other hand , while HPs enjoy excellent interpretability due to the additive structure in the intensity , they overlook the inhibition effect among the events , which is common in real world . More important , nearly all the HP based methods adopt exponential time-decaying kernels ( or a combination of them ) ( see ( 1 ) ) , including the recent works ( Mei and Eisner , 2017 ; Zhang et al. , 2020a ) that extend HPs to allow a negative triggering kernel ( αs′s < 0 in ( 1 ) ) . However , an exponential decay can be overly simplistic for various , complex real-world applications . These methods can therefore miss other important decaying patterns ( e.g. , polynomial decays ) , resulting in inferior influence estimation and predictions . To overcome these issues , we propose SPRITE , a multi-variate self-adaptable point process model that can explicitly estimate the influence type and strength between every pair of the events , and flexibly capture all kinds of time decays of the influence strength , not limited to a particular form . Specifically , we first define a raw rate function for each event type s ( 1 ≤ s ≤ K ) to characterize both the excitation and inhibition effects from the previous events , λ̃s ( t ) = λ 0 s + ∑M i=1 1 ( ti < t ) hsi→s ( t− ti ) , ( 4 ) where 1 ( · ) is an indicator function , λ0s is the background ( or base ) rate , and hsi→s ( · ) is an influence term that characterizes the effect of a previous event of type si happened at ti on the occurrence of a new event of type s at t. When hsi→s > 0 , it indicates a triggering effect while hsi→s < 0 means inhibition . The absolute value of hsi→s is the strength of the influence . Since the raw rate can be negative , we then apply a scaled soft-plus transformation ( Mei and Eisner , 2017 ) to ensure we obtain a positive rate function , λs ( t ) = β log ( 1 + exp ( λ̃s ( t ) /β ) ) where β > 0 . Next , we introduce a set of embeddings U = { us } Ks=1 to represent the event types . We sample these embeddings from the standard Gaussian distribution , p ( U ) = ∏ sN ( us|0 , I ) . Then we model the influence term as a function of the event type embeddings and time span , hsi→s ( ∆ ) = h ( usi , us , ∆ ) . ( 5 ) The advantage of doing so is that we only need to estimate one function h to capture the influence among all the events ( including those within the same type ) . The embeddings can further enable us to discover the hidden structures of the event types ( e.g. , cluster and outliers ) . Otherwise , if we estimate one function for each pair of event types , we have to learn K ( K + 1 ) /2 functions , which can be much more challenging and costly , especially when the event types are many ( i.e. , large K ) . We want our model to flexibly learn the influence function h ( · ) and its strength decaying patterns with time , rather than restrict to any pre-specified , parametric form , e.g. , the exponential decay in ( 1 ) . In other words , we only require when h > 0 , the time derivative ∂h∂∆ < 0 , and when h < 0 , ∂h ∂∆ > 0 , i.e. , h · ∂h∂∆ < 0 , to ensure |h| decreases over time . While it is standard to assign a Gaussian process ( GP ) prior ( Rasmussen and Williams , 2006 ) over h to enable a flexible , nonparametric function estimation , incorporating the time derivative constraints is difficult . The derivative of a GP is still a GP ( provided the kernel is differentiable ) . Since the support of the Gaussian distribution ( GP finite projection ) is the entire real space , imposing a monotonic constraint is infeasible ( there are always nonzero probabilities for positive/negative derivatives ) . Existing work ( Riihimäki and Vehtari , 2010 ) introduces a set of virtual points and encourage the derivative at these points to be as positive ( or negative ) as possible . This essentially is a soft regularization , not an actual constraint . To address this issue , we observe that the time-decaying influence functions have a general structure . Lemma 3.1. h · ∂h∂∆ < 0 if and only if h has the form h ( usi , us , ∆ ) = g ( usi , us ) ( 1− f ( usi , us , ∆ ) ) , ( 6 ) where g ( usi , us ) = h ( usi , us , 0 ) , 0 ≤ f ( · ) < 1 , ∂f ∂∆ > 0 , and f ( usi , us , 0 ) = 0 . We leave the proof in the Appendix . Next , to construct f , we represent f ( · ) = tanh ( z ( usi , us , ∆ ) ) where z ( · ) is another function . To satisfy the requirements for f in ( 6 ) , we need to ensure z ≥ 0 , z ( usi , us , 0 ) = 0 , and ∂z ∂∆ > 0 . According to these constraints , we can represent z as an integral function , z ( usi , us , ∆ ) = ∫∆ 0 exp ( η ( usi , us , r ) ) dr where η ( · ) is a free function . Combining with ( 6 ) , we now derive a general construction of our influence function , Corollary 3.1.1. h · ∂h∂∆ < 0 if and only if h ( usi , us , ∆ ) = g ( usi , us ) ( 1− tanh ( ∫ ∆ 0 exp ( η ( usi , us , r ) ) dr ) ) ( 7 ) where both g ( · ) and η ( · ) 1 are free latent functions . Note that our construction ( 7 ) also covers the special case that h ( · ) = 0 ( when g = 0 ) , i.e. , no influence is from the previous event . To flexibly estimate h ( · ) , we can place GP priors over g and η . However , we no longer need to incorporate any derivative constraint and hence the learning is much easier . Note that the prior of h ( · ) is not GP anymore . One might be concerned that the integral in ( 7 ) is analytically intractable and will bring troubles in model estimation . However , since the integration is one dimensional , we can use Gauss-Legendre quadrature to calculate the integral quite accurately and evaluate h ( · ) . We will discuss the details in Sec . 4.2 .

Summary: This paper proposes a multi-variate self-adaptable point process with nonparametric time decays. First, to flexibly represent arbitrary types of influences between events, they introduce an embedding to represent each event type, and model the influence between any two events as an unknown function of the event type embeddings and time span. Second, they also propose a general construction that covers all possible time decaying functions by assigning Gaussian process (GP) priors over the free latent functions and using Gauss Legendra quadrature to derive the integral in the construction. This gives their model more representative power while being computationally efficient. For efficient inference, they also use weight-space augmentation of GPs to prevent computing huge covariances matrices, and neural network feature mapping to enhance the learning capacity. Lastly, a scalable stochastic variational learning algorithm is developed by taking advantage of the re-reparameterization trick.