Self-Adaptable Point Processes with Nonparametric Time Decays

1 Introduction . Events of multiple types are ubiquitous , such as in online shopping , social networking and biological signal transduction . Understanding the complex influences of those events on each other , including excitation , inhibition , and how the strength of these influences varies with time , is crucial to discover useful knowledge and to predict future events and their types , which can benefit many applications , e.g. , marketing analysis , biological study , and early warnings of catastrophes . While many excellent works have been proposed for event modeling and analysis ( Blundell et al. , 2012 ; Linderman and Adams , 2014 ; Xu et al. , 2016 ; Tan et al. , 2016 ; Xu and Zha , 2017 ; Zhang et al. , 2020a ) , most of them are inadequate to capture complex influences between the events . The classical Poisson processes simply ignore these influences and assume independent increments . Although Hawkes processes ( HP ) can estimate the mutual excitation between the events , they ignore the inhibition effect , which is common in real world . More important , existing methods ( Xu et al. , 2016 ; Zhang et al. , 2020a ) , even for those considering both the excitation and inhibition , mostly assume an exponential time decay of the influence strength , which is over-simplified and can miss many other decay patterns . While recent works have also used recurrent neural networks ( RNNs ) ( Du et al. , 2016 ; Mei and Eisner , 2017 ) to model the event rate and hence are highly expressive , they couple 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . all the temporal dependencies in a black box , and are difficult to distill meaningful and important knowledge , e.g. , influence types and strengths . To address these limitations , we propose SPRITE , a multi-variate self-adaptable point process with nonparametric time decays . Our method not only can discover the type of the influence ( i.e. , excitation or inhibition ) between every pair of the events , but also is flexible enough to estimate all kinds of influence strength decaying patterns , not limited to the exponential decay or other particular forms . Specifically , we first introduce an embedding to represent each event type , and model the influence between any two events as an unknown function of the event type embeddings and time span . In so doing , we avoid separately estimating the influence between every pair of the events , and largely reduce the parameters . Next , we propose a general construction that covers all possible time decaying functions . We assign Gaussian process ( GP ) priors over the free latent functions and use Gauss-Legendra quadrature to calculate the integral in the construction . In this way , we can flexibly estimate a variety of time decaying influences while not needing to impose any derivative constraint , which can bring learning difficulties . For efficient inference , we use weight-space augmentation of GPs to prevent computing huge covariances matrices , and neural network feature mapping to enhance the learning capacity . We then use the re-reparameterization trick to develop a scalable stochastic variational learning algorithm . For evaluation , we first examined SPRITE in an ablation study . We tested the case of a single event type with excitation effects only and three time decays , and a bi-type event case with mixed excitation and inhibition effects . In both cases , our method accurately recovered the rate function of each type of events and the influence between the events , performing better than state-of-the-art methods based on or extending Hawkes processes and RNNs . Next , we examined SPRITE in three real-world and one synthetic benchmark datasets . We examined the accuracy in predicting the occurrence time and type of future events . In both tasks , our method nearly always outperforms the competing approaches , often by a large margin . Finally , by looking into the embeddings and influence functions learned by our model , we found interesting patterns within the event types and their interactions . 2 Background . Temporal Point Processes . We denote a sequence of events and their types by Γ = [ ( t1 , s1 ) , . . . , ( tM , sM ) ] , where each ti is the time point when the i-th event occurred ( t1 ≤ . . . ≤ tM ) , si is the type of the i-th event . Suppose we have K types of events in total ; so each si ∈ { 1 , . . . , K } . Temporal point processes ( TPPs ) ( Daley and Vere-Jones , 2007 ) provide a fundamental framework to characterize and model the events . A TPP is defined via the rate ( or conditional intensity ) function of the events , λs ( t ) . For example , a homogeneous Poisson process assumes each λs ( t ) is a constant λ0s , and the past events { ( tj , sj ) |tj < t } do not have any influence on generating a new event at t. While being simple and convenient , Poisson processes completely ignore the interactions between the events . Hawkes processes ( HPs ) ( Hawkes , 1971 ) overcome this issue by modeling the triggering effects between the events , λs ( t ) = λ0s + ∑ tj < t γsj→s ( t− tj ) where γsj→s ( ∆ ) > 0 is called “ triggering kernel ” , and characterizes how largely the previous event at tj encourages a new event of type s to occur at t. Naturally , this effect is supposed to decay with time ∆ . The most commonly used choice is the exponential decay , γs′→s ( ∆ ) = αs′s · exp ( −τs′s∆ ) , ( 1 ) where αs′s > 0 and τs′s > 0 . Given the definition of λs ( · ) , the probability of the event sequence is p ( Γ ) = ∏K k=1 exp ( − ∫ T 0 λk ( t ) dt ) ∏M i=1 λsi ( ti ) , ( 2 ) where T is the total span of the events in Γ and so tM ≤ T . RNN based Point Processes . HPs only consider the triggering effects and hence can be quite restricted . In order to estimate arbitrarily varying rate functions , recent works Du et al . ( 2016 ) ; Mei and Eisner ( 2017 ) use recurrent neural networks ( RNNs ) to model the event dependencies and rates . The key idea is as follows . For each event i , we introduce a hidden state vector β ( ti ) . The event dependency is modeled through the updating rule of the hidden states , β ( ti+1 ) = ρ ( β ( ti ) , ti+1 ) ( 3 ) where ρ ( · ) is a complicated mapping , fulfilled by a combination of neural network operations , such as linear transformation , nonlinear activation , and gating . Note that the event type si can also be incorporated into ρ ( · ) . The rate function is then defined as a transformation of β ( t ) . We can therefore evaluate and maximize the point process likelihood ( 2 ) for model estimation . 3 Model . While RNN based methods are highly expressive and can capture arbitrarily complex dependencies , due to the black-box updating rule ( see ( 3 ) ) , they can hardly distill meaningful knowledge , such as the influence type and strength between the events , which are critical for data analysis and knowledge discovery . On the other hand , while HPs enjoy excellent interpretability due to the additive structure in the intensity , they overlook the inhibition effect among the events , which is common in real world . More important , nearly all the HP based methods adopt exponential time-decaying kernels ( or a combination of them ) ( see ( 1 ) ) , including the recent works ( Mei and Eisner , 2017 ; Zhang et al. , 2020a ) that extend HPs to allow a negative triggering kernel ( αs′s < 0 in ( 1 ) ) . However , an exponential decay can be overly simplistic for various , complex real-world applications . These methods can therefore miss other important decaying patterns ( e.g. , polynomial decays ) , resulting in inferior influence estimation and predictions . To overcome these issues , we propose SPRITE , a multi-variate self-adaptable point process model that can explicitly estimate the influence type and strength between every pair of the events , and flexibly capture all kinds of time decays of the influence strength , not limited to a particular form . Specifically , we first define a raw rate function for each event type s ( 1 ≤ s ≤ K ) to characterize both the excitation and inhibition effects from the previous events , λ̃s ( t ) = λ 0 s + ∑M i=1 1 ( ti < t ) hsi→s ( t− ti ) , ( 4 ) where 1 ( · ) is an indicator function , λ0s is the background ( or base ) rate , and hsi→s ( · ) is an influence term that characterizes the effect of a previous event of type si happened at ti on the occurrence of a new event of type s at t. When hsi→s > 0 , it indicates a triggering effect while hsi→s < 0 means inhibition . The absolute value of hsi→s is the strength of the influence . Since the raw rate can be negative , we then apply a scaled soft-plus transformation ( Mei and Eisner , 2017 ) to ensure we obtain a positive rate function , λs ( t ) = β log ( 1 + exp ( λ̃s ( t ) /β ) ) where β > 0 . Next , we introduce a set of embeddings U = { us } Ks=1 to represent the event types . We sample these embeddings from the standard Gaussian distribution , p ( U ) = ∏ sN ( us|0 , I ) . Then we model the influence term as a function of the event type embeddings and time span , hsi→s ( ∆ ) = h ( usi , us , ∆ ) . ( 5 ) The advantage of doing so is that we only need to estimate one function h to capture the influence among all the events ( including those within the same type ) . The embeddings can further enable us to discover the hidden structures of the event types ( e.g. , cluster and outliers ) . Otherwise , if we estimate one function for each pair of event types , we have to learn K ( K + 1 ) /2 functions , which can be much more challenging and costly , especially when the event types are many ( i.e. , large K ) . We want our model to flexibly learn the influence function h ( · ) and its strength decaying patterns with time , rather than restrict to any pre-specified , parametric form , e.g. , the exponential decay in ( 1 ) . In other words , we only require when h > 0 , the time derivative ∂h∂∆ < 0 , and when h < 0 , ∂h ∂∆ > 0 , i.e. , h · ∂h∂∆ < 0 , to ensure |h| decreases over time . While it is standard to assign a Gaussian process ( GP ) prior ( Rasmussen and Williams , 2006 ) over h to enable a flexible , nonparametric function estimation , incorporating the time derivative constraints is difficult . The derivative of a GP is still a GP ( provided the kernel is differentiable ) . Since the support of the Gaussian distribution ( GP finite projection ) is the entire real space , imposing a monotonic constraint is infeasible ( there are always nonzero probabilities for positive/negative derivatives ) . Existing work ( Riihimäki and Vehtari , 2010 ) introduces a set of virtual points and encourage the derivative at these points to be as positive ( or negative ) as possible . This essentially is a soft regularization , not an actual constraint . To address this issue , we observe that the time-decaying influence functions have a general structure . Lemma 3.1. h · ∂h∂∆ < 0 if and only if h has the form h ( usi , us , ∆ ) = g ( usi , us ) ( 1− f ( usi , us , ∆ ) ) , ( 6 ) where g ( usi , us ) = h ( usi , us , 0 ) , 0 ≤ f ( · ) < 1 , ∂f ∂∆ > 0 , and f ( usi , us , 0 ) = 0 . We leave the proof in the Appendix . Next , to construct f , we represent f ( · ) = tanh ( z ( usi , us , ∆ ) ) where z ( · ) is another function . To satisfy the requirements for f in ( 6 ) , we need to ensure z ≥ 0 , z ( usi , us , 0 ) = 0 , and ∂z ∂∆ > 0 . According to these constraints , we can represent z as an integral function , z ( usi , us , ∆ ) = ∫∆ 0 exp ( η ( usi , us , r ) ) dr where η ( · ) is a free function . Combining with ( 6 ) , we now derive a general construction of our influence function , Corollary 3.1.1. h · ∂h∂∆ < 0 if and only if h ( usi , us , ∆ ) = g ( usi , us ) ( 1− tanh ( ∫ ∆ 0 exp ( η ( usi , us , r ) ) dr ) ) ( 7 ) where both g ( · ) and η ( · ) 1 are free latent functions . Note that our construction ( 7 ) also covers the special case that h ( · ) = 0 ( when g = 0 ) , i.e. , no influence is from the previous event . To flexibly estimate h ( · ) , we can place GP priors over g and η . However , we no longer need to incorporate any derivative constraint and hence the learning is much easier . Note that the prior of h ( · ) is not GP anymore . One might be concerned that the integral in ( 7 ) is analytically intractable and will bring troubles in model estimation . However , since the integration is one dimensional , we can use Gauss-Legendre quadrature to calculate the integral quite accurately and evaluate h ( · ) . We will discuss the details in Sec . 4.2 .

Summary: The paper proposes a semi-parametric class of point processes which is intensity-based while using kernel-based,non-parametric functions to model the effects between types of events in the intensity function. This reduces the number of the learnable parameters hence making the model applicable to datasets with a large number of event types.  In particular, it   (1) formulates in Lemma 3.1 the property of such kernel functions that guarantee a time-decaying effect of the influence functions when there is present either an excitatory effect or an inhibitory effect   (2) proposes a class of kernel functions in corollary 3.1.1 . that respects the property in (1)  (3) suggests a training algorithm that leverages a bayesian linear regression model for constructing the kernel functions and stochastic variational inference on batches of types and event sequences for accelerating inference.