Neural Spectral Marked Point Processes

1 INTRODUCTION . Event sequence data are ubiquitous in our daily life , ranging from traffic incidents , 911 calls , social media posts , earthquake catalog data , and COVID-19 data ( see , e.g. , Bertozzi et al . ( 2020 ) ) . Such data consist of a sequence of events indicating when and where each event occurred , with additional descriptive information ( called marks ) about the event ( such as category , volume , or free-text ) . The distribution of events is of scientific and practical interest , both for prediction purposes and for inferring events ’ underlying generative mechanism . A popular framework for modeling events is point processes ( Daley & Vere-Jones , 2008 ) , which can be continuous over time and the space of marks . An important aspect of this model is capturing the event ’ s triggering effect on its subsequent events . Since the distribution of point processes is completely specified by the conditional intensity function ( the occurrence rate of events conditioning on the history ) , such triggering effect has been captured by an influence kernel function embedded in the conditional intensity . In statistical literature , the kernel function usually assumes a parametric form . For example , the original work by Hawkes ( Hawkes , 1971 ) considers an exponential decaying influence function over time , and the seminal work ( Ogata , 1998 ) introduces epidemic-type aftershock sequence ( ETAS ) model , which considers an influence function that exponentially decays over space and time . With the increasing complexity of modern applications , there has been much recent effort in developing recurrent neural network ( RNN ) -based point processes , leveraging the rich representation power of RNNs ( Du et al. , 2016 ; Mei & Eisner , 2017 ; Xiao et al. , 2017b ) . However , there are several limitations of existing RNN-based models . First , such models typically do not consider the kernel function ( Du et al. , 2016 ; Li et al. , 2018 ; Mei & Eisner , 2017 ; Upadhyay et al. , 2018 ; Xiao et al. , 2017a ; b ) ; thus , the RNN approach does not enjoy the interpretability of the kernel function based models . Second , the popular RNN models such as Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) still implicitly discounts the influence of events over time ( due to their recursive structure ) ( Vaswani et al. , 2017 ; Zhu et al. , 2021d ) . Such assumptions may not hold in many real-world applications . Take the earthquake catalog as an example , which is a typical type of discrete event data ; most aftershocks occur along the fault plane or other faults within the volume affected by the mainshock ’ s strain ( Zhu et al. , 2021b ) . This means that different regions may be correlated to their surrounding area differently according to their geological structure , which creates a complex non-stationary spatial profile that we would like to capture through the model . Third , a majority of the existing works mainly focus on one-dimensional temporal point processes . Although there are works on marked point processes ( Du et al. , 2016 ; Mei & Eisner , 2017 ; Reinhart , 2018 ) , they are primarily based on simplifying assumptions that the marks are conditionally independent of the event ’ s time and location , which is equivalent to assuming the kernel is separable ; these assumptions may fail to capture some complex non-stationary , time- and location-dependent triggering effects for various types of events , as observed for many real-world applications ( see , e.g. , ( Bertozzi et al. , 2020 ) ) . Contribution . In this paper , we present a novel general non-stationary point process model based on neural networks , referred to as the neural spectral marked point process ( NSMPP ) . The key component is a new powerful representation of the kernel function using neural networks , which enables us to go beyond stationarity ( and thus go beyond standard Hawkes processes ) and has the capacity to model high-dimensional marks . Figure 1 gives an example of non-stationary influence kernel that measures the influence of the past events to the future time t. The premise of the model design is that the conditional intensity function uniquely specifies the distribution of the point process , and the most important component in the intensity function is the influence kernel . In summary , the novelty of our approach includes the following : • The kernel function is represented by a spectral decomposition of the influence kernel with a finite-rank truncation in practice . Such a kernel representation will enable us to capture the most general non-stationary process as well as high-dimensional marks . The model also allows the distribution of marks to depend on time , which is drastically different from the separable kernels considered in the existing literature ( Reinhart , 2018 ) . • The spectral decomposition of asymmetric influence kernel consists of a sum of the product of feature maps , which can be parameterized by neural networks . This enable us to harvest the powerful expressiveness and scalability to high-dimensional input of neural networks for complicated tasks involving discrete events data . • We establish theoretical guarantees of the maximum likelihood estimate for the true kernel function based on functional variational analysis and finite-dimensional asymptotic analysis , which shed light on theoretical understanding of neural network-based kernel functions . • Using synthetic and real data ( seismic and police data ) , we demonstrate the superior performance of our proposed method in complex situations ; the performance gain is particularly outstanding for cases involving non-stationary point processes . Related work . Seminal works in point processes modeling ( Ogata , 1988 ; 1998 ) assume parametric forms of the intensity functions . Such methods enjoy good interpretability and are efficient to estimate . However , classical parametric models are not expressive enough to capture the events ’ dynamics in modern applications . Recent research interests aim to improve the expressive power of point process models , where a recurrent neural networks ( RNNs ) -based structure is introduced to represent the conditional intensity function ( Du et al. , 2016 ; Mei & Eisner , 2017 ; Xiao et al. , 2017b ) . However , most of these works either explicitly or implicitly specify the inter-event dependence in a limited form with restrained representative power . For example , Du et al . ( 2016 ) expresses the influence of two consecutive events in a form of exp { w ( ti+1 − ti ) } , which is an exponential function with respect to the length of the time interval ti+1 − ti with weights w ; Mei & Eisner ( 2017 ) enhances the expressiveness of the model and represents the entire history using the hidden state of an LSTM , which still implicitly assumes the influence of the history decays over time due to the recurrent structure of LSTM . Another line of research uses neural networks to directly model dependence of sequential events without specifying the conditional intensity function explicitly ( Li et al. , 2018 ; Xiao et al. , 2017a ) . Some studies consider non-stationary influence kernel using neural networks in spatio-temporal point processes ( Zhu et al. , 2021b ; a ) . Recent work ( Omi et al. , 2019 ) also uses a neural network to parameterize the hazard function , the derivative of which gives the conditional intensity function . However , the above approaches either capture the temporal dependence or assume the temporal and mark dependence are separable rather than jointly accounting for marked-temporal dependence . Recently , attention models have become popular in computer vision and sequential data modeling ( Britz et al. , 2017 ; Luong et al. , 2015 ; Vaswani et al. , 2017 ) . This motivates works including Zhang et al . ( 2019 ) ; Zhu et al . ( 2021c ; d ) ; Zuo et al . ( 2020 ) to model the conditional intensity of point processes using the attention mechanism and characterize the inter-event dependence by a score function . The attention mechanism has proven to be more flexible in capturing long-range dependence regardless of how far apart two events are separated and greatly enhances the performance in practice . However , the main limitation of Zhang et al . ( 2019 ) ; Zuo et al . ( 2020 ) is that they rely on a conventional score function – dot-product between linear mappings of events , which is still limited in representing non-linear dependence between events for some applications . Zhu et al . ( 2019 ; 2021c ; d ) used a more flexible and general Fourier kernel as a substitution for the dot-product score ; however , the expressive power of the proposed Fourier kernel is still limited , and the spectrum of the Fourier basis is represented by a generative neural network , which is difficult to learn in some cases ( Arjovsky & Bottou , 2017 ) . There are also works considering point processes with non-stationary intensities . Chen & Hall ( 2013 ) proposed time-varying background intensity for point process , while we focus on the non-stationary triggering kernel depicting complex events dependency . Remes et al . ( 2017 ; 2018 ) studied nonstationary kernels combined with Gaussian processes , assuming specific structures of the kernels in the Fourier domain . Such kernels are more restricted than ours since the nature of Gaussian processes requires that the kernel is positive semidefinite . 2 METHOD . 2.1 BACKGROUND : MARKED TEMPORAL POINT PROCESS . Marked temporal point processes ( MTPPs ) ( Reinhart , 2018 ) consist of a sequence of events over time . Each event is associated with a ( possibly multi-dimensional ) mark that contains detailed information of the event , such as location , nodal information ( if the observations are over networks , such as sensor or social networks ) categorical data , and contextual information ( such as token , image , and text descriptions ) . Let T > 0 be a fixed time-horizon , andM⊆ Rd be the space of marks . We denote the space of observation as X = [ 0 , T ) ×M and a data point in the discrete event sequence as x = ( t , m ) , t ∈ [ 0 , T ) , m ∈M , ( 1 ) where t is the event time and m represents the mark . Let Nt be the number of events up to time t < T ( which is random ) , and Ht : = { x1 , x2 , . . . , xNt } denote historical events . Let N be the counting measure on X , i.e. , for any measurable S ⊆ X , N ( S ) = |HT ∩ S| . For any function f : X → R , the integral with respect to the counting measure is defined as∫ S f ( x ) dN ( x ) = ∑ xi∈HT∩S f ( xi ) . The events ’ distribution in MTPPs can be characterized via the conditional intensity function λ ( x ) , which is defined to be the conditional probability of observing an event in the marked temporal space X given the events ’ historyHt ( x ) , that is , E ( dN ( x ) |Ht ( x ) ) = λ ( x ) dx . ( 2 ) Above , t ( x ) extracts the occurrence time of event x , and we omit the dependence on Ht ( x ) in the notation of λ ( x ) for simplicity . As self- and mutual-exciting point processes , Hawkes processes ( Hawkes , 1971 ) have been widely used to capture the mutual excitation dynamics among temporal events . The model assumes that influences from past events are linearly additive towards the current event . The conditional intensity function for a self-exciting point process takes the form of λ [ k ] ( x ) = µ+ ∑ x′∈Ht ( x ) k ( x′ , x ) , ( 3 ) where µ > 0 stands for the background intensity , and the so-called “ influence kernel ” k : X ×X → R is crucial in capturing the influence of past events on the likelihood of event occurrence at the current time . Here we use the notation [ k ] to stress the dependence of the conditional intensity function on the kernel function k ( x′ , x ) . Written in the form of the integral over counting measure , we have that λ [ k ] ( x ) = µ+ ∫ x′∈Xt ( x ) k ( x′ , x ) dN ( x′ ) , ( 4 ) where Xt is the subset of X with the first component smaller than t. The most commonly made assumption in the literature is that the process is stationary , where the influence of the past events is shift-invariant , such that k ( x′ , x ) = f ( x− x′ ) for a influence function f : Rd → R+ ; a common influence function in one-dimensional cases is f ( t ) = α exp { −βt } , where β controls the decay rate and α > 0 controls the magnitude of the influence of an event . The current work aims at going beyond stationary point processes , which enables us to better capture the heterogeneity in the events ’ influence across the spatial-temporal space , which naturally arises in many applications .

Summary: This paper propose a point process with a non-stationary kernel to model complex event data. The kernel represented by its finite rank decomposition and the basis functions (feature functions) are models using a neural network architecture. To learn the model parameters they used stochastic gradient to maximize the resulting likelihood function. Moreover, they give a theoretical guarantee by showing that under some assumptions, true kernel function is identifiable by solving the MLE problem. They also compared their method against state-of-the-art and baseline on synthetic and real datasets.