Title: Integration-free Training for Spatio-temporal Multimodal Covariate Deep Kernel Point Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spatio-temporal Point Processes

Deep Kernel

Covariate

Integration-free
Abstract: In this study, we propose a novel deep spatio-temporal point process model, Deep Kernel Mixture Point Processes (DKMPP), that incorporates multimodal covariate information. DKMPP is an enhanced version of Deep Mixture Point Processes (DMPP), which uses a more flexible deep kernel to model complex relationships between events and covariate data, improving the model's expressiveness. To address the intractable training procedure of DKMPP due to the non-integrable deep kernel, we utilize an integration-free method based on score matching, and further improve efficiency by adopting a scalable denoising score matching method. Our experiments demonstrate that DKMPP and its corresponding score-based estimators outperform baseline models, showcasing the advantages of incorporating covariate information, utilizing a deep kernel, and employing score-based estimators.

Integration-free Training for Spatio-temporal Multimodal Covariate Deep Kernel Point Processes

Abstract

In this study, we propose a novel deep spatio-temporal point process model, Deep Kernel Mixture Point Processes (DKMPP), that incorporates multimodal covariate information. DKMPP is an enhanced version of Deep Mixture Point Processes (DMPP), which uses a more flexible deep kernel to model complex relationships between events and covariate data, improving the model's expressiveness. To address the intractable training procedure of DKMPP due to the non-integrable deep kernel, we utilize an integration-free method based on score matching, and further improve efficiency by adopting a scalable denoising score matching method. Our experiments demonstrate that DKMPP and its corresponding score-based estimators outperform baseline models, showcasing the advantages of incorporating covariate information, utilizing a deep kernel, and employing score-based estimators.

1 Introduction

Point processes are widely used statistical tools for modeling event occurrence patterns within continuous spatio-temporal domains. They have garnered extensive attention across various fields, such as criminology [17; 38], neuroscience [24; 40], financial engineering [11; 2], and epidemiology [20; 25]. The core task of point processes is to derive an intensity function from event sequences, which represents the rate of event occurrences at any given time or location within the observation domain.

There are two common approaches for modeling intensity functions: traditional and covariatebased methods. The traditional approach [13; 10; 18] considers only event information, such as time and position, while ignoring the effects of contextual factors. Conversely, the covariatebased method [31; 33; 3] incorporates covariates from the observation domain, providing insights into factors that may cause event occurrences. In real applications, event occurrence rates are often correlated with contextual factors, making covariate-based methods more effective for event prediction [21]. For example, in crime data analysis, the covariate-based method considers factors like income level, education status, and public security to estimate crime intensity. Thus, utilizing a covariate-based point process model is significant for leveraging rich contextual information to predict events and explore various factors influencing event occurrence.

Quyu Kong

Okawa et al. [21] proposed Deep Mixture Point Processes (DMPP) to incorporate covariate information into point process models. DMPP models intensity as a deep mixture of parametric kernels, representing the influence of nearby representative points on the target event. Mixture weights are modeled by a neural network using covariates from representative points as input, avoiding intractable intensity integrals for log-likelihood evaluation and enabling simple parameter estimation. However, DMPP requires integrable parametric kernels like radial basis function (RBF) or polynomial kernels, which have closed-form solutions for intensity integrals. With more complex, flexible kernels (e.g., deep kernels [35]), DMPP cannot compute the intensity integral tractably. In real-world applications, flexible kernels are usually preferred to capture complex influences of covariates on intensity, which can be challenging for restricted parametric kernels.

In this study, we aim to learn the intricate influence of covariate data on event occurrence using a data-driven approach. We propose replacing the simple parametric kernel in DMPP with a deep kernel [35], which transforms the inputs of a base kernel using a deep neural network. The deep kernel significantly enhances expressive power by automatically learning a flexible input transformation metric, thus transcending Euclidean and absolute distance-based metrics. This results in a substantial improvement in DMPP's expressiveness, leading to our proposed model, Deep Kernel Mixture Point Processes (DKMPP). However, due to the deep architecture embedded in the base kernel, the deep kernel is non-integrable, causing an intractable training procedure as intensity integral evaluation is required for likelihood-based parameter estimation. To address parameter estimation in DKMPP, we adopt an integration-free method based on score matching [12], which estimates model parameters by matching the gradient of the log-density of the model and data. Score matching avoids intensity integral computation and allows tractable training. However, the naive score matching estimator is computationally expensive as it requires the computation of second derivatives of the log-density. To improve efficiency, we further adopt a scalable denoising score matching (DSM) method [32], with the resulting models referred to as score-DKMPP and score-DKMPP+ respectively.

Score-based estimators, such as Score-DKMPP and Score-DKMPP+, offer an advantage over traditional likelihood-based estimators as they do not require numerical integration, which is often computationally expensive and prone to errors. Additionally, Score-DKMPP+ has a high efficiency due to the utilization of denoising score matching. Our experiments serve to highlight the advantages of DKMPP and its corresponding score-based estimators. In the majority of our experiments, both Score-DKMPP and Score-DKMPP+ outperform the baseline models, indicating that incorporating covariate information can enhance the modeling and prediction of spatio-temporal events. Furthermore, our results strongly suggest that the adoption of a deep kernel significantly enhances the model's expressiveness when compared to a simple parametric kernel.

Specifically, our contributions are as follows: (1) We propose DKMPP, which replaces the simple parametric kernel in DMPP with a more flexible deep kernel, enhancing model expressiveness and allowing for the learning of complex influence from covariate factors on event occurrence. (2) We use the score matching estimator to overcome the intractable intensity integral in DKMPP during training and extend the denoising score matching method to DKMPP, significantly improving computational efficiency. (3) We analyze the difference between likelihood estimator and score matching estimators for DKMPP, demonstrate the superiority of DKMPP over baseline models, and show the impact of deep kernel and various hyperparameters through experiments.

2 Related Work

Point processes are statistical tools for modeling event occurrence patterns, with intensity functions characterizing event rates. Essential models include Poisson processes [13], renewal processes [5], and Hawkes processes [10]. Poisson processes feature independent point occurrences and have been widely used in economics [8], ecology [34], and astronomy [28]. Renewal processes generalize Poisson processes for arbitrary event intervals. However, neither model is suitable for self-excitation in event patterns. Hawkes processes, containing a triggering kernel, are suitable for applications with excitation effects, such as seismology [19], finance [11], criminology [16], and neuroscience [39].

Deep point processes have gained traction due to the growing popularity of deep neural networks. While traditional models capture simple event patterns, deep point processes leverage neural networks to describe complex dependencies. The first deep point process work by Du et al. [7] utilized RNNs to encode history information. Later, Xiao et al. [36] and Mei & Eisner [14] proposed LSTM-based

versions. However, RNNs struggle with long-term dependencies, training difficulties, and lack of parallelism. To address these issues, Zuo et al. [43] and Zhang et al. [37] proposed attention-based models. Additionally, there are some point process models directly relevant to our work, such as Okawa et al. [22] and Zhu et al. [42], which employ the use of deep kernels to model the intensity function. Besides directly modeling intensity functions with neural networks, other approaches use them to represent probability density functions [29] or cumulative intensity functions [23; 30]. However, to the best of our knowledge, most deep point process models are limited to temporal point processes, and few works have extended them to spatio-temporal point processes.

Covariate point processes differ from traditional point processes by incorporating contextual factors and establishing connections between these factors and event patterns. This integration of covariate information enables a more comprehensive explanation of studied phenomena and improves predictive performance. Meyer et al. [15] proposed a model for epidemic prediction using population density and other covariates. Adelfio & Chiodi [1] proposed a spatio-temporal self-exciting point process for earthquake forecasting using geological features. Gajardo & Müller [9] analyzed COVID-19 cases with point processes, studying their relation to covariates such as mobility restrictions and demographic factors. However, these works model the influence of covariate in a handcrafted parametric form, limiting flexibility.

Different from existing works, our approach emphasizes covariate point processes and enhances both model flexibility and parameter estimation. While our work is closely related to [21], we extend it by introducing the deep kernel and score-based estimation methods in our proposed DKMPP model.

3 Preliminaries

We offer a brief overview of three fundamental components that serve as the foundation for our work.

3.1 Spatio-temporal Point Processes

A spatio-temporal point process is a stochastic model whose realization is a random collection of points representing the time and location of events [6]. Consider a sequence of N events S = { ( t 1 , x 1 ) , ( t 2 , x 2 ) , . . . , ( t N , x N ) } , where ( t n , x n ) denotes the time and location of the n -th event in T × X ⊆ R × R 2 . The intensity function λ ( t, x ) represents the instantaneous event occurrence rate:

where N (∆ t × ∆ x ) is the number of events in ∆ t × ∆ x , ∆ t is a small time interval around t , ∆ x is a small spatial region around x , and E denotes expectation w.r.t. realizations. Given a sequence S , the probability density function, or likelihood, of the point process can be written as [6]:

3.2 Deep Kernel

Wilson et al. [35] combined kernel methods and neural networks to create a deep kernel, capturing the expressive power of deep neural networks. The deep kernel extends traditional covariance kernels by embedding a deep architecture into the base kernel:

where k ϕ is a base kernel with parameters ϕ . To enhance flexibility, the inputs x and x ′ are transformed by a deep neural network g parameterized by weights w . The base kernel offers various options, such as the common RBF kernel. The deep kernel parameters include base kernel parameters ϕ and neural network weights w . An advantage of deep kernels is their ability to learn metrics by optimizing input space transformation in a data-driven manner, rather than relying on Euclidean distance-based metrics, which are common in traditional kernels but may not always be suitable, especially in high-dimensional input spaces.

3.3 Score Matching

The classic maximum likelihood estimation (MLE) minimizes the Kullback-Leibler (KL) divergence between the parameterized model distribution and the data distribution, but requires an often intractable normalizing constant. Numerical integration can approximate the intractable integral, but its complexity grows exponentially with the input dimension. An alternative is to minimize the Fisher divergence, which does not require the normalizing constant. The Fisher divergence is defined as:

where p ( x ) and q ( x ) are two smooth distributions of the same dimension, and ∥ · ∥ is a suitable norm (e.g., ℓ 2 norm). Minimizing the Fisher divergence estimates model parameters by matching the gradient of the log-density of the model to the log-density of the data. The parameterized model density is defined as p θ ( x ) = 1 Z ( θ ) ˜ p ( x | θ ) , where θ represents the model parameters, and the normalizing constant Z ( θ ) = ∫ ˜ p ( x | θ ) dx is intractable. With Fisher divergence, we do not need to compute the normalizing constant, as the matched gradients do not depend on it. The gradient of log-density is referred to score, so the aforementioned method is also called score matching [12].

4 Methodology

In this section, we introduce our proposed DKMPP and two score-based parameter estimation methods. Following Okawa et al. [21], we use the same notation. We assume a sequence of N events S = { s 1 = ( t 1 , x 1 ) , s 2 = ( t 2 , x 2 ) , . . . , s N = ( t N , x N ) } , where s n = ( t n , x n ) represents the time and location of the n -th event in T × X ⊆ R × R 2 . We also assume K contextual covariates on the spatio-temporal region T × X : D = { Z 1 , Z 2 , . . . , Z K } , where Z k is the k -th covariate.

4.1 Deep Kernel Mixture Point Processes

Okawa et al. [21] developed a deep spatio-temporal covariate point process model called DMPP, capable of incorporating multimodal contextual data like temperature, humidity, text and image. The intensity of DMPP is designed in a kernel convolution form:

where u = ( τ, r ) is a point in the region T × X , and k ϕ ( s , u ) is a kernel with parameters ϕ . f w , a deep neural network, inputs contextual data and outputs nonnegative mixture weights, with w representing the network parameters. Z ( u ) = { Z 1 ( u ) , . . . , Z K ( u ) } are covariate values at point u .

Although the intensity formulation integrates multimodal contextual data, in reality, covariate data is only available on a regular grid. DMPP introduces finite representative points { u j } J j =1 on the spatio-temporal domain, resulting in a discrete version 2 :

The discrete version of DMPP offers two significant advantages: (1) the intensity itself is tractable since it requires summation instead of integration; (2) the intensity integral is tractable as long as the kernel integral is tractable, as given by ∫ λ ( s | D ) d s = ∑ J j =1 f w ( u j , Z ( u j )) ∫ k ϕ ( s , u j ) d s . For this reason, DMPP employs simple integrable kernels like RBF kernels, which can be integrated analytically, to simplify parameter estimation. This design choice ensures that the intensity integral is tractable, which is crucial for evaluating log-likelihood. Consequently, parameter estimation can be performed using straightforward back-propagation.

However, this approach has notable drawbacks. Firstly, using limited parametric kernels constrains the intensity's expressive power. Secondly, Euclidean distance in the kernel may not be a suitable measure of similarity, particularly in high-dimensional input spaces. In many applications, the

relationship between covariates and event occurrence is complex and often unknown, making it desirable to model this influence using data-driven kernels with non-Euclidean distance metrics.

To address the issues mentioned earlier, we propose DKMPP, which replaces the simple integrable kernel in DMPP with the deep kernel [35]. The deep kernel captures the similarity between representative points and event points in both time and location. The deep kernel has greater expressiveness than the traditional kernel and can flexibly learn the metric's functional form through a neural network-based nonlinear transformation. Specifically, the intensity of DKMPP is designed as:

where k ϕ,w 2 ( x , x ′ ) = k ϕ ( g w 2 ( x ) , g w 2 ( x ′ )) is the deep kernel, with ϕ as the base kernel parameters and w 2 as the weights of a nonlinear transformation g implemented by a deep neural network. In subsequent experiments, we found that, even for this three-dimensional problem, the deep kernel's ability to learn metrics from data outperforms the traditional Euclidean distance. Compared to DMPP, DKMPP introduces a link function η ( · ) to ensure non-negativity of the intensity function 3 . Various functions, such as softplus, exponential, and ReLU, can be utilized as the link function. Unless otherwise specified, we choose softplus as η ( · ) in this work.

Representative Points In this study, we adopt a fixed grid of representative points, which are evenly spaced both along the temporal axis and across the spatial region. The complete set of representative points is derived as the Cartesian product of the temporal and spatial representatives.

Implementation Description Both the kernel mixture weight network f and the non-linear transformation g in the deep kernel are implemented using MLPs with ReLU activation functions. To handle multimodal covariates such as numerical values, categorical values, and text, etc., we perform necessary feature extraction. If the covariates are numerical, we normalize them as necessary. If the covariates are categorical, we convert them to numerical type or one-hot encoding. If the covariates are textual, we use a Transformer architecture to extract features. If the covariates are multimodal, we fuse the multimodal information by concatenating the extracted features from multiple modules.

4.2 Parameter Estimation with Score Matching

The traditional approach to estimate point process is based on MLE where the log-likelihood is:

The intensity integral is a compensator which can be understood as a normalizing constant. The integration is intractable in most cases, and our proposed DKMPP is no exception. To solve this problem, we normally resort to numerical integration, e.g., Monte Carlo or quadrature, to obtain an approximation. However, these numerical methods introduce additional errors; more importantly, they are not scalable to the high-dimensional problem.

Score-DKMPP In this work, we propose an integration-free estimation method based on score matching [12] to estimate the model parameters in our proposed DKMPP. Surprisingly, the scorebased estimator for point process model is largely unexplored in recent years. As far as we know, few works attempted to utilize score matching for the estimation of point processes. Sahani et al. [26] proposed a pioneering estimator in this field, linking score matching with temporal point processes. Here, we extend the score matching estimator to our covariate-based spatio-temporal DKMPP. We assume the ground-truth process generating the data has a density p ( S ) and design a parameterized model with density p θ ( S ) where θ is the model parameter to estimate. A Fisher-divergence objective for our covariate-based spatio-temporal DKMPP is designed as:

where s n is any entry in vector s n , i.e., s n ∈ s n = ( t n , x n ) , ˜ N is the number of equivalent variables, ˜ N = N for 1-D point process, e.g., temporal point process; ˜ N = 2 N for 2-D point process, e.g., spatial point process; and ˜ N = 3 N for 3-D point process, e.g., spatio-temporal point process, etc. The optimal estimate of model parameter is given by ˆ θ = arg min θ F ( θ ) .

However, the above loss cannot be minimized directly as it depends on the gradient of the ground-truth data distribution which is unknown. Following the derivation in Hyvärinen [12], this dependence can be eliminated by using a trick of integration by parts. Sahani et al. [26] assumed the log density of point process satisfies the proper smoothness and obtained a concise empirical loss. In this work, we prove that without those smoothness assumptions, we can still obtain the same empirical loss (proof provided in Appendix A):

where we take M sequences { S m } M m =1 from p ( S ) , ˜ N m is the number of equivalent variables on the m -th sequence, s m,n is the n -th variable on the m -th sequence, the constant C 1 does not depend on θ and can be discarded. It is easy to see that, in Eq. (10), the intensity integral has been removed by the operation of gradient. We refer to our proposed DKMPP estimated by Eq. (10) as score-DKMPP.

Score-DKMPP+ A serious drawback of the objective in Eq. (10) is it requires the second derivative which is computationally expensive. To improve efficiency, we derive the denoising score matching (DSM) method [32] for our proposed DKMPP to avoid second derivatives. We add a small noise to the sequence S to obtain a noisy sequence ˜ S (we add noise to each variable ˜ s n = s n + ϵ ), which is distributed as p ( ˜ S ) = ∫ p ( ˜ S | S ) p ( S ) dS . The DSM method uses the Fisher divergence between the noisy data distribution p ( ˜ S ) and model distribution p θ ( ˜ S ) as an objective:

where ˜ s n is any entry in the noisy vector ˜ s n , i.e., ˜ s n ∈ ˜ s n = ( ˜ t n , ˜ x n ) . The optimal parameter estimate is given by ˆ θ = arg min θ F DSM ( θ ) . To facilitate computation, the Fisher divergence in Eq. (11) can be further rewritten as (proof provided in Appendix B):

where we take M clean and noisy sequences { S m , ˜ S m } M m =1 from p ( S, ˜ S ) , ˜ s m,n is the n -th variable on the m -th noisy sequence, the constant C 2 does not depend on θ and can be discarded. With a Gaussian noise ϵ ∼ N (0 , σ 2 ) , the conditional gradient term can be written in a closed form: ∂ log p ( ˜ S m | S m ) /∂ ˜ s m,n = -(˜ s m,n -s m,n ) /σ 2 . Equation (12) avoids the unknown ground-truth data distribution, the intractable intensity integral and the tedious second derivatives.

It is worth noting that the denoising score matching is not equivalent to the original score matching, because it is easy to see from Eq. (11) that the model is trained to match the score of the noisy data distribution p ( ˜ S ) instead of the original p ( S ) . However, when the noise is small enough, we can consider p ( ˜ S ) ≈ p ( S ) such that Eq. (12) is approximately equivalent to Eq. (10). We refer to our proposed DKMPP estimated by Eq. (12) as score-DKMPP+.

5 Experiments

In the experimental section, we mainly analyze the difference between MLE and score matching for DKMPP, the improvement in performance of DKMPP over baseline models, as well as the impact of various hyperparameters.

5.1 DKMPP: MLE vs. Score Matching

Since our proposed DKMPP model employs a score-based parameter estimation method instead of traditional MLE, a natural question arises regarding the comparison between the two approaches. To

100% Data

10% Data

Datasize

0

1

2

3

4

5

6

RMSE

MLE-DKMPP

Score-DKMPP

Score-DKMPP+

(a) RMSE w.r.t. Data Size

0

20

40

60

80

100

120

140

Number of Monte Carlo Samples

3.7

3.8

3.9

4.0

4.1

4.2

RMSE

runtime: 150.8s

runtime: 356.79s

runtime: 29.41s

MLE-DKMPP

Score-DKMPP

Score-DKMPP+

(b) RMSE w.r.t. MC

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

DSM: t = 10

0.00

3.67

7.33

11.00

14.67

18.33

22.00

25.67

29.33

33.00

(c)

σ

2

= 0

.

01

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

DSM: t = 10

0.00

3.33

6.67

10.00

13.33

16.67

20.00

23.33

26.67

30.00

(d)

σ

2

= 10

evaluate this question, we generate a 3-D spatio-temporal point process synthetic dataset. The spatial observation x spans the area of [0 , 1] × [0 , 1] , while the temporal observation window covers the time interval of [0 , 10] . We design a 1-D covariate function on the domain, a kernel mixture weight function f w 1 ( u j , Z ( u j )) , and a deep kernel k ϕ,w 2 ( s , u j ) with the RBF base kernel. More details are provided in Appendix C.1. We fix the representative points on a regular grid: 5 representative points evenly spaced on each axis, so there are 5 3 = 125 representative points in total. We use the thinning algorithm [19] to generate 5,000 sequences according to the ground truth specified above. The statistics of the synthetic data are provided in Appendix C.1. We try to fit a DKMPP model to the synthetic data with the ground-truth representative points and an RBF base kernel. Both the kernel mixture weight network f and the non-linear transformation g in the deep kernel are implemented using MLPs with ReLU activation functions. Therefore, the learnable parameters are w 1 , w 2 , ϕ .

Two aspects of particular interest when comparing MLE and score matching estimators are consistency and computational cost. Consistency refers to the property that as the data size varies, the estimates produced by these methods converge to the true parameter values. The computational costs of these methods refer to the amount of time required to achieve a certain level of accuracy.

For consistency, we use the root mean square error (RMSE) between the estimated intensity and the ground-truth intensity as a metric. During the training of MLE, numerical integration is required to approximate the intractable intensity integral. In our experiments, we use Monte Carlo methods and fix the number of Monte Carlo samples to 1,000, which is sufficient to obtain good results. For Score-DKMPP+, we use a Gaussian noise ϵ ∼ N (0 , σ 2 ) with σ 2 = 0 . 01 . Refer to Appendix C.1 for the intensity functions learned by different estimators. We test the RMSE of MLE-DKMPP, Score-DKMPP and Score-DKMPP+ under different data sizes ( 100% and 10% ), as shown in Fig. 1a. Three estimators exhibit very similar RMSE performance; however, as the data size decreases, the errors of all three estimators increase slightly. Besides, we examine the effect of Monte Carlo samples on RMSE. Specifically, we use 100% data and increase the number of Monte Carlo samples from 10 to 150. The results are shown in Fig. 1b. Due to the reliance on Monte Carlo integration, MLE suffers from poor RMSE when the number of Monte Carlo samples is insufficient. As the number of Monte Carlo samples increases, the parameter estimation performance of MLE improves and eventually converges. In comparison, score matching estimators (Score-DKMPP and Score-DKMPP+) have an advantage because they only require event location and do not rely on Monte Carlo integration. Therefore, the parameter estimation performance of score matching estimators is independent of the number of Monte Carlo samples.

For computational efficiency, we record the running time required by three estimators to achieve the same level of accuracy. The results are shown in Fig. 1b, indicating that to reach the same RMSE, the Score-DKMPP requires 356.8 seconds, while the MLE is faster, taking 150.8 seconds. Furthermore, the Score-DKMPP+ is extremely fast, completing the task in only 29.4 seconds.

5.2 Real-world Data

In this section, we validate our proposed DKMPP model and its corresponding score-based estimators on real-world data from various domains. We also compare DKMPP against other popular spatialtemporal point process models.

Datasets We analyze three datasets from the fields of transportation and crime, with details of the datasets shown below. Each dataset is divided into training, validation and test data using a 50% / 40% / 10% split ratio based on time. The data preprocessing is provided in Appendix C.2.

Crimes in Vancouver 4 This dataset is composed of more than 530 thousand crime records, including all categories of crimes committed in Vancouver. Each crime record contains the time and location (latitude and longitude) of the crime.

NYC Vehicle Collisions 5 The New York City vehicle collision dataset contains about 1.05 million vehicle collision records. Each collision record includes the time and location (latitude and longitude).

NYC Complaint Data 6 This dataset contains over 228 thousand complaint records in New York City. Each record includes the date, time, and location (latitude and longitude) of the complaint.

Covariates These three datasets contain not only the spatio-temporal information of the events but also multimodal covariate information. For example, the Crime in Vancouver dataset includes information on the cause and type of crime records; the NYC Vehicle Collisions dataset includes textual descriptions of the accident scene, such as borough and contributing factors; the NYC Complaint Data includes numerical and textual covariate information, such as the police department number and offense description. For every representative point, we identify the closest numerical/categorical/textual features in both time and space, which are used as covariates.

Baselines Deep temporal point process models have gained widespread use in recent years, e.g., Mei & Eisner [14]; Omi et al. [23]; Zhang et al. [37]; Zuo et al. [43]. These works primarily focus on modeling time information while neglecting spatial information. As far as we know, there has been limited research on deep learning approaches for spatial-temporal point process modeling. Here, for a fair comparison, we compare DKMPP against deep (covariate) spatial-temporal point process models. Specifically, the following baselines are considered: (1) the homogeneous Poisson process (HomoPoisson) [13]; (2) the neural spatio-temporal point processes (NSTPP) [4]; (3) the deep spatio-temporal point processes (DeepSTPP) [41]; (4) the DMPP [21].

HomoPoisson is a traditional statistical spatial-temporal point process model; NSTPP and DeepSTPP are deep spatial-temporal point process models that do not incorporate covariates, while DMPP and DKMPP are deep covariate spatial-temporal point process models that allow the inclusion of covariate information. For DMPP and DKMPP, we experiment with three different kernels: RBF kernel, rational quadratic (RQ) kernel and Ornstein-Uhlenbeck (OU) kernel (See Appendix C.2).

Metrics We use two metrics to evaluate the performance: test log-likelihood (TLL) and prediction accuracy (ACC). The TLL measures the log-likelihood on the test data, indicating how well the model captures the distribution of the data. The ACC evaluates the absolute difference between the predicted number of events and the actual number of events on the test data, 1 -| predicted # -actual # | / actual #, assessing how accurately the model predicts the number of events on the test data.

Results We evaluate the performance of all baseline models in terms of TLL and ACC. In our experiments, we use Monte Carlo methods for the baselines which need numerical integration and fix the number of Monte Carlo samples to 1,000 which is sufficient to obtain good results. For Score-DKMPP+, we use a Gaussian noise ϵ ∼ N (0 , σ 2 ) with σ 2 = 0 . 01 . More training details are provided in Appendix C.2. The performance results are presented in Table 1, which show that Score-DKMPP and Score-DKMPP+ achieve similar performance with the same kernel, and both outperform other baseline models. As expected, HomoPoisson performs the worst due to its lack of flexibility in modeling varying intensity functions over time and space. In comparison to NSTPP and DeepSTPP, DKMPP performs better in two aspects. Firstly, DKMPP incorporates rich multimodal covariate information, which aids in modeling event occurrences. Secondly, DKMPP employs a score-based estimator instead of the MLE estimator, which avoids unnecessary errors caused by numerical integration. The comparison between DKMPP and DMPP with the same kernel constitutes an ablation study, which again verifies that using a deep kernel can provide stronger representational power and achieve better performance in both data fitting and prediction. This is because the deep

kernel can flexibly learn the functional form of the metric by optimizing input space transformation in a data-driven manner, rather than relying on a fixed Euclidean distance-based metric.

5.3 Hyperparameter Analysis

Representative Points The effect of the number of representative points on the performance of DKMPP on synthetic and real-world data is presented in Table 2 and Appendix C.2, respectively. In general, the accuracy of DKMPP tends to improve with an increase in the number of representative points, although the change in performance is not significant. However, having more representative points leads to higher computational costs. Therefore, in the experiment, the number of representative points needs to be chosen as a trade-off between accuracy and efficiency.

Layer and Batch Size We examine how the number of layers in the kernel mixture weight network f and the non-linear transformation g of the deep kernel, as well as batch size, affect model performance on synthetic and real data in Table 2 and Appendix C.2, respectively. In general, varying batch sizes do not produce a significant difference in model performance, so we set a batch size of 100 because both overly large and excessively small batch sizes may harm optimization. For the number of layers, we find that the number of layers is correlated with the dimension of covariates. In the case of low-dimensional covariates, a simple hidden layer can provide better performance than multiple layers. Thus, we set the number of layers to 1 for the synthetic data and 2 for the real data.

Noise Variance The noise added by the Score-DKMPP+ is an important hyperparameter. If the noise is too large, the resulting noisy distribution p ( ˜ S ) will deviate significantly from the true data distribution p ( S ) , leading to severe biases in the learned results. We conduct experiments with two different noise variance settings, σ 2 = 0 . 01 and σ 2 = 10 . When σ 2 = 0 . 01 , the model can learn the intensity function that is close to the ground truth, as shown in Fig. 1c. However, as the noise level increases, when σ 2 = 10 , the learned results deviate significantly from the true values, as shown in Fig. 1d. In practice, setting σ 2 too small can also lead to numerical instability.

6 Limitations

One limitation of the current study is that the proposed multimodal covariate spatio-temporal point process model is constrained to covariates of numerical, categorical, and textual types during the experiments, primarily due to the availability of data. However, future research could greatly benefit from the incorporation of image-type covariates, as this holds immense potential for enhancing the model's ability to capture a wider range of diverse and intricate effects. By including image data, the model could potentially uncover richer patterns and relationships, leading to even more fascinating and insightful results.

7 Conlusions

In conclusion, we have introduced a novel deep spatio-temporal point process model, the DKMPP, which incorporates multimodal covariate information and captures complex relationships between events and covariate data by utilizing a more flexible deep kernel leading to improved expressiveness. We address the intractable training procedure of DKMPP by using an integration-free method based on score matching and improving efficiency through a scalable denoising score matching method. Through our experiments, we show that DKMPP and its corresponding score-based estimators outperform baseline models, demonstrating the advantages of incorporating covariate information, utilizing a deep kernel, and employing score-based estimators. Our results suggest that DKMPP is a promising approach for modeling complex spatio-temporal events with multimodal covariate data.

Acknowledgments and Disclosure of Funding

This work was supported by NSFC Project (No. 62106121), the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD110001), the fund for building world-class universities (disciplines) of Renmin University of China (No. KYGJC2023012), China-Austria Belt and Road Joint Laboratory on Artificial Intelligence and Advanced Manufacturing of Hangzhou Dianzi University, and the Public Computing Cloud, Renmin University of China.

References

Adelfio, G. and Chiodi, M. Including covariates in a space-time point process with application to seismicity. Statistical Methods & Applications , 30(3):947-971, 2021.

Bacry, E., Delattre, S., Hoffmann, M., and Muzy, J.-F. Modelling microstructure noise with mutually exciting point processes. Quantitative Finance , 13(1):65-77, 2013.

Baddeley, A., Chang, Y.-M., Song, Y., and Turner, R. Nonparametric estimation of the dependence of a spatial point process on spatial covariates. Statistics and its interface , 5(2): 221-236, 2012.

Chen, R. T. Q., Amos, B., and Nickel, M. Neural spatio-temporal point processes. In International Conference on Learning Representations , 2021.

Cinlar, E. Markov renewal theory. Advances in Applied Probability , 1(2):123-187, 1969.

Daley, D. J. and Vere-Jones, D. An introduction to the theory of point processes. vol. i. probability and its applications, 2003.

Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Krishnapuram, B., Shah, M., Smola, A. J., Aggarwal, C. C., Shen, D., and Rastogi, R. (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , pp. 1555-1564. ACM, 2016.

Dufresne, F. and Gerber, H. U. Risk theory for the compound poisson process that is perturbed by diffusion. Insurance: mathematics and economics , 10(1):51-59, 1991.

Gajardo, Á. and Müller, H.-G. Point process models for covid-19 cases and deaths. Journal of Applied Statistics , pp. 1-16, 2021.

Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika , 58(1):83-90, 1971.

Hawkes, A. G. Hawkes processes and their applications to finance: a review. Quantitative Finance , 18(2):193-198, 2018.

Hyvärinen, A. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res. , 6:695-709, 2005.

Kingman, J. F. C. Poisson processes , volume 3. Clarendon Press, 1992.

Mei, H. and Eisner, J. The neural Hawkes process: A neurally self-modulating multivariate point process. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pp. 6754-6764, 2017.

Meyer, S., Elias, J., and Höhle, M. A space-time conditional intensity model for invasive meningococcal disease occurrence. Biometrics , 68(2):607-616, 2012.

Mohler, G. Modeling and estimation of multi-source clustering in crime and security data. The Annals of Applied Statistics , pp. 1525-1539, 2013.

Mohler, G. O., Short, M. B., Brantingham, P. J., Schoenberg, F. P., and Tita, G. E. Self-exciting point process modeling of crime. Journal of the American Statistical Association , 106(493): 100-108, 2011.

Møller, J., Syversveen, A. R., and Waagepetersen, R. P. Log Gaussian Cox processes. Scandinavian journal of statistics , 25(3):451-482, 1998.

Ogata, Y. Space-time point-process models for earthquake occurrences. Annals of the Institute of Statistical Mathematics , 50(2):379-402, 1998.

Ogata, Y., Matsu'ura, R. S., and Katsura, K. Fast likelihood computation of epidemic type aftershock-sequence model. Geophysical research letters , 20(19):2143-2146, 1993.

Okawa, M., Iwata, T., Kurashima, T., Tanaka, Y., Toda, H., and Ueda, N. Deep mixture point processes: Spatio-temporal event prediction with rich contextual information. In Teredesai, A., Kumar, V., Li, Y., Rosales, R., Terzi, E., and Karypis, G. (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019 , pp. 373-383. ACM, 2019.

Okawa, M., Iwata, T., Tanaka, Y., Toda, H., Kurashima, T., and Kashima, H. Dynamic hawkes processes for discovering time-evolving communities' states behind diffusion processes. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pp. 1276-1286, 2021.

Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pp. 2120-2129, 2019.

Paninski, L. Maximum likelihood estimation of cascade point-process neural encoding models. Network: Computation in Neural Systems , 15(4):243-262, 2004.

Rizoiu, M., Mishra, S., Kong, Q., Carman, M. J., and Xie, L. Sir-hawkes: Linking epidemic models and hawkes processes to model diffusions in finite populations. In Champin, P., Gandon, F., Lalmas, M., and Ipeirotis, P. G. (eds.), Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018 , pp. 419-428. ACM, 2018.

Sahani, M., Bohner, G., and Meyer, A. Score-matching estimators for continuous-time pointprocess regression models. In Palmieri, F. A. N., Uncini, A., Diamantaras, K. I., and Larsen, J. (eds.), 26th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2016, Vietri sul Mare, Salerno, Italy, September 13-16, 2016 , pp. 1-5. IEEE, 2016.

Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.

Scargle, J. D. Studies in astronomical time series analysis. v. bayesian blocks, a new method to analyze structure in photon counting data. The Astrophysical Journal , 504(1):405, 1998.

Shchur, O., Bilos, M., and Günnemann, S. Intensity-free learning of temporal point processes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Shchur, O., Gao, N., Biloš, M., and Günnemann, S. Fast and flexible temporal point processes with triangular maps. Advances in Neural Information Processing Systems , 33:73-84, 2020.

Truccolo, W., Eden, U. T., Fellows, M. R., Donoghue, J. P., and Brown, E. N. A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. Journal of neurophysiology , 93(2):1074-1089, 2005.

Vincent, P. A connection between score matching and denoising autoencoders. Neural Comput. , 23(7):1661-1674, 2011.

Waagepetersen, R. Estimating functions for inhomogeneous spatial point processes with incomplete covariate data. Biometrika , 95(2):351-363, 2008.

Warton, D. I. and Shepherd, L. C. Poisson point process models solve the" pseudo-absence problem" for presence-only data in ecology. The Annals of Applied Statistics , pp. 1383-1402, 2010.

Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P. Deep kernel learning. In Gretton, A. and Robert, C. C. (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016 , volume 51 of JMLR Workshop and Conference Proceedings , pp. 370-378. JMLR.org, 2016.

Xiao, S., Yan, J., Yang, X., Zha, H., and Chu, S. M. Modeling the intensity function of point process via recurrent neural networks. In Singh, S. and Markovitch, S. (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA , pp. 1597-1603. AAAI Press, 2017.

Zhang, Q., Lipani, A., Kirnap, Ö., and Yilmaz, E. Self-attentive Hawkes process. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pp. 11183-11193. PMLR, 2020.

Zhou, F., Li, Z., Fan, X., Wang, Y., Sowmya, A., and Chen, F. Efficient inference for nonparametric Hawkes processes using auxiliary latent variables. Journal of Machine Learning Research , 21(241):1-31, 2020.

Zhou, F., Zhang, Y., and Zhu, J. Efficient inference of flexible interaction in spiking-neuron networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Zhou, F., Kong, Q., Deng, Z., Kan, J., Zhang, Y., Feng, C., and Zhu, J. Efficient inference for dynamic flexible interactions of neural populations. Journal of Machine Learning Research , 23 (211):1-49, 2022.

Zhou, Z., Yang, X., Rossi, R., Zhao, H., and Yu, R. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference , pp. 777-789. PMLR, 2022.

Zhu, S., Wang, H., Dong, Z., Cheng, X., and Xie, Y. Neural spectral marked point processes. In International Conference on Learning Representations , 2021.

Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer Hawkes process. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pp. 11692-11702. PMLR, 2020.

A Score Matching Estimator

We provide the proof of the score matching estimator for temporal point processes and spatial point processes below, respectively. Generally speaking, the derivation for temporal point processes is more complex than spatial point processes because as we can see later there exists some complications arising from different limits of integration due to the order constraint on timestamps.

A.1 Temporal Point Processes

Given an observation window [0 , T ] , a sequence from a temporal point process is composed of a random number of timestamps arranged in a sequential order S = { t n } N n =1 where t 1 < t 2 < . . . < t N and t n ∈ [0 , T ] is the n -th event timestamp. We assume the ground-truth process generating the data has a density p ( S ) and design a parameterized model with density p θ ( S ) where θ is the model parameter to estimate. Following Sahani et al. [26], we define a Fisher-divergence objective:

The above loss can be understood as matching the variational derivatives of log-density w.r.t. the counting process N ( t ) given that the counting process is non-decreasing and piece-wise constant with unit steps.

However, the above loss cannot be minimized directly as it depends on the gradient of the ground-truth data distribution which is unknown. Following the derivation of Hyvärinen [12], this dependence can be eliminated by using a trick of integration by parts. Let us expand Eq. (13), discard ( ∂ log p ( S ) ∂t n ) 2 which does not depend on parameter θ , and examine the cross-term:

where S t -n represents the sequence excluding t n , the fourth line uses integration by parts, the fifth line uses delta function to evaluate the limits of t n given the order constraint of timestamps. Therefore, the loss in Eq. (13) can be rewritten as:

where the constant C 1 does not depend on θ and can be discarded. To construct the final empirical loss, we replace the expectation by the empirical average, which eliminates the delta functions as any two timestamps cannot overlap with each other, and obtain the final version:

where we take M sequences { S m } M m =1 from p ( S ) , t m,n is the n -th timestamp on the m -th sequence.

It is worth noting that Sahani et al. [26] assumed the parametric density satisfies the smoothness property: ∂ t n log p θ ( S ) | t n = t n +1 = ∂ t n +1 log p θ ( S ) | t n +1 = t n to cancel most delta functions. Here, we emphasize that this smoothness assumption is not necessary. In our derivation, we do not utilize this smoothness property, but just take advantage of the non-overlapping of timestamps to eliminate all delta functions and obtain the same objective.

A.2 Spatial Point Processes

Let us consider a planar point process for example. Given a 2-D observation region X ⊆ R 2 , a realization from a 2-D spatial point process is composed of a random number of points S = { x n } N n =1 where x n ∈ X is the n -th event location (2-D coordinate). It is worth noting that these points are in no order. Similarly, we define a Fisher-divergence objective:

where x n is any entry in vector x n , i.e., x n ∈ x n .

Similarly, we use the trick of integration by parts to eliminate the dependence of the loss on the gradient of the unknown ground-truth data distribution. Let us expand Eq. (17), discard ( ∂ log p ( S ) ∂x n ) 2 which does not depend on parameter θ , and examine the cross-term:

where S x -n represents the realization excluding x n , the fourth line uses integration by parts, the fifth line assumes a weak regularity condition: p ( S ) ∂ x n log p θ ( S ) goes to zero for any θ when | x n |→∞ . It is worth noting that in spatial point processes the limits of x n are no longer constrained because there is no order for the points. Therefore, the loss in Eq. (17) can be rewritten as:

where the constant C 1 does not depend on θ and can be discarded. Replacing the expectation by the empirical average, we obtain the final empirical loss:

where we take M realizations { S m } M m =1 from p ( S ) , x m,n is the n -th variable on the m -th realization.

A.3 Spatio-temporal Point Processes

It is interesting to see that both score matching estimators for temporal point processes and spatial point processes have the same empirical loss (see Eq. (16) and Eq. (20)) regardless of whether the points are sequential or not. Therefore, it is easy to draw the conclusion that for spatio-temporal point processes the empirical score matching estimator is:

where s m,n is the n -th variable on the m -th sequence, ˜ N m is the number of equivalent variables on the m -th sequence, ˜ N m = N m for 1-D point process, e.g., temporal point process; ˜ N m = 2 N m for 2-D point process, e.g., spatial point process; and ˜ N m = 3 N m for 3-D point process, e.g., spatio-temporal point process, etc.

B Denoising Score Matching Estimator

In Appendix A, we provide an estimator trying to match the gradient of the log-density of the point process model to the log-density of the point process data. However, the estimator requires the second derivatives, which is computationally expensive. To avoid this issue, following the derivation of Vincent [32], we derive a denoising score matching estimator.

Differently, the denoising score matching estimator tries to match the gradient of the log-density of the model to the log-density of the noisy point process data. We add a small noise to the sequence S to obtain a noisy sequence ˜ S (we add noise to each variable ˜ s n = s n + ϵ ), which is distributed as p ( ˜ S ) = ∫ p ( ˜ S | S ) p ( S ) dS . Therefore, the Fisher divergence between the noisy data distribution p ( ˜ S ) and model distribution p θ ( ˜ S ) is:

where ˜ s n is any entry in the noisy vector ˜ s n , i.e., ˜ s n ∈ ˜ s n = ( ˜ t n , ˜ x n ) , ˜ N is the number of equivalent variables. Let us expand Eq. (22), discard ( ∂ log p ( ˜ S ) ∂ ˜ s n ) 2 which does not depend on parameter θ , and examine the cross-term:

Therefore, the loss in Eq. (22) can be rewritten as:

where the constants C and C 2 do not depend on θ and can be discarded. Replacing the expectation by the empirical average, we obtain the final empirical loss:

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Ground Truth: t = 10

0.00

3.67

7.33

11.00

14.67

18.33

22.00

25.67

29.33

33.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

LL: t = 10

0.00

3.67

7.33

11.00

14.67

18.33

22.00

25.67

29.33

33.00

(b) MLE-DKMPP

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

SM: t = 10

0.00

3.67

7.33

11.00

14.67

18.33

22.00

25.67

29.33

33.00

(c) Score-DKMPP

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

DSM: t = 10

0.00

3.67

7.33

11.00

14.67

18.33

22.00

25.67

29.33

33.00

(d) Score-DKMPP+

where we take M clean and noisy sequences { S m , ˜ S m } M m =1 from p ( S, ˜ S ) , ˜ s m,n is the n -th variable on the m -th noisy sequence, ˜ N m is the number of equivalent variables on the m -th noisy sequence.

C Experimental Details

C.1 Synthetic Data

Data Simulation We generate a 3-D spatio-temporal point process synthetic dataset. The spatial observation x spans the area of [0 , 1] × [0 , 1] , while the temporal observation window covers the time interval of [0 , 10] . We assume a 1-D covariate function Z ( u = ( τ, r )) = ( N ( r 1 | 0 . 5 , 0 . 5) + N ( r 2 | 0 . 5 , 0 . 5)) on the domain. We set f w 1 ( u j , Z ( u j )) = 20 Z ( u j ) + 0 . 1 , k ϕ,w 2 ( s , u j ) = k ϕ ( g w 2 ( s ) , g w 2 ( u j )) where k ϕ is the RBF kernel k ϕ ( x , x ′ ) = exp ( -ϕ ∥ x -x ′ ∥ 2 ) with ϕ = 100 and g w 2 is a linear transformation g w 2 ( s ) = s +0 . 1 . We fix the representative points on a regular grid: 5 representative points evenly spaced on each axis, so there are 5 3 = 125 representative points in total. We use the thinning algorithm to generate 5,000 sequences according to the intensity function specified above. The statistics of the synthetic data are shown in Table 3.

Training Details We fit a DKMPP model to the synthetic data with the ground-truth representative points and an RBF base kernel. Both the kernel mixture weight network f and the non-linear transformation g in the deep kernel are implemented using MLPs with ReLU activation functions. Therefore, the learnable parameters are w 1 , w 2 , ϕ . The intensity functions at t = 10 estimated by three different estimators are shown in Fig. 2.

C.2 Real-world Data

Data Preprocessing The preprocessing details of three real-world datasets are shown below. We listed the statistics of three real-world datasets after preprocessing in Table 3.

Crimes in Vancouver This dataset is composed of more than 530 thousand crime records, including all categories of crimes committed in Vancouver from 2003 to 2017. Each crime record contains the time and location (latitude and longitude) of the crime. We split the data into multiple sequences by year, month and day. Then, we select events from 2013 to 2016, drop the NaN value, and scale the time and space into a volume of [0 , 1] × [0 , 1] × [0 , 10] . We select the categorical feature 'Crime Type' as the descriptive feature for each event. We convert the categorical feature into the corresponding numerical feature as the covariate. Finally, the dimension of covariate Z is 1.

NYC Vehicle Collisions The New York City vehicle collision dataset contains about 1.05 million vehicle collision records. Each collision record includes the time and location (latitude and longitude). Wesplit the data into multiple sequences by day. We select the records from 01/01/2019 to 02/03/2019, drop the NaN value and scale the time and space into a volume of [0 , 1] × [0 , 1] × [0 , 10] . We select 'BOROUGH', 'CONTRIBUTING FACTOR VEHICLE 1', 'CONTRIBUTING FACTOR VEHICLE 2' and 'VEHICLE TYPE CODE 1' as the descriptive features for each event. We concatenate several textual features and use a pre-trained DistilBERT [27] to extract the textual features, and concatenate the textual features with other numerical/categorical features as the covariate. Finally, the dimension of covariate Z is 768.

NYC Complaint Data This dataset contains over 228 thousand complaint records in New York City. Each record includes the date, time, and location (latitude and longitude) of the complaint. We split the data into multiple sequences by hour. We select the records from 01/11/2022 to 13/11/2022, drop the NaN value and scale the time and space into a volume of [0 , 1] × [0 , 1] × [0 , 10] . We select 'OFNS_DESC', 'JURIS_DESC', 'LAW_CAT_CD', 'PD_DESC', 'VIC_RACE', 'VIC_SEX' and 'PREM_TYP_DESC' as the descriptive features for each event. We concatenate several textual features and use a pre-trained DistilBERT [27] to extract the textual features, and concatenate the textual features with other numerical/categorical features as the covariate. Finally, the dimension of covariate Z is 768.

Training Details Each dataset is divided into training, validation and test data using a 50% / 40% / 10% split ratio based on time. For the real-world data, we fix the representative points on a regular grid: 5 representative points evenly spaced on each axis, so there are 5 3 = 125 representative points in total. We use three different kernel functions for comparisons: the RBF kernel k ϕ ( x , x ′ ) = exp ( -ϕ ∥ x -x ′ ∥ 2 ) , the rational quadratic (RQ) kernel k ϕ ( x , x ′ ) = (1+ ϕ ∥ x -x ′ ∥ 2 ) -1 2 , and the Ornstein-Uhlenbeck (OU) kernel k ϕ ( x , x ′ ) = exp ( -ϕ ∥ x -x ′ ∥ ) . For the three real-world datasets, the kernel mixture weight network f and the non-linear transformation g in the deep kernel are implemented using MLPs with ReLU activation functions and we fixed the number of layers in f and g as 2.

Hyperparameters Wetested the performance of Score-DKMPP and Score-DKMPP+ with different hyperparameters on three real-world datasets. We tested the number of representation points and network structures. For the representation points, we tested three different values, 64, 125, and 216, respectively. For the network structure, we tested the number of layers of 1, 2, and 4. When we tested the effect of representation points, we fix the network with 2 hidden layers and when we tested the effect of network structure, we fix the number of representation points as 125.

For the representation points, the accuracy of Score-DKMPP and Score-DKMPP+ overall have better performance when the number of representation points is 125. The accuracy tends to increase when

64

125

216

# of Representation points

30

40

50

60

70

80

90

100

ACC

Crimes in Vncouver

NYC Vehicle Collisions

NYC Complaint Data

(a) Score-DKMPP

64

125

216

# of Representation points

30

40

50

60

70

80

90

100

ACC

Crimes in Vncouver

NYC Vehicle Collisions

NYC Complaint Data

(b) Score-DKMPP+

1

2

4

# of Layers

30

40

50

60

70

80

90

100

ACC

Crimes in Vncouver

NYC Vehicle Collisions

NYC Complaint Data

(c) Score-DKMPP

1

2

4

# of Layers

30

40

50

60

70

80

90

100

ACC

Crimes in Vncouver

NYC Vehicle Collisions

NYC Complaint Data

(d) Score-DKMPP+

the number of representation points increases from 64 to 125, however, except for the performance of Score-DKMPP on the complaint dataset, for other real-world datasets, the accuracy starts to drop.

For the network structure, the accuracy using Score-DKMPP on the Crimes in Vancouver and NYC Complaint data tends to increase as the number of layers increases. However, for Score-DKMPP+, we only capture a similar trend on the NYC Vehicle Collisions data. Other than those mentioned above, we do not observe a significant pattern when increasing the number of layers.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to fuse Deep Kernel Learning (DKL) into the Deep Mixture Point Processes (DMPP), resulting in Deep Kernel Mixture Point Processes (DKMPP) which can handle complex relationships between events and covariates in a more flexible and expressive manner. The authors also leverage the denoising score matching technique and come up with a training procedure that does not require integration and computation of second-order derivative. The proposed method is evaluated on a variety of datasets and compared with other baseline point process models to demonstrate its advantages.\n\nSTRENGTHS:\n• The authors extend the score matching to point processes and make the training more scalable by leveraging denoising technique, which, I believe, is a fair contribution to the field of point process learning and can be applied to a variety of scenarios.\n\n• The authors conduct a wide range of experiments using both synthetic and real-world datasets, demonstrating the efficiency and effectiveness of the proposed method. The choice of model hyperparameters is appropriately studied.\n\n• The organization and the presentation of the proposed method are clear and easy to understand.\n\nWEAKNESSES:\nOn lines 158-159, the authors argue that Euclidean distance in the kernel may not be a suitable measure of similarity, especially for **high-dimensional inputs**. I think this statement needs more clarification. If I understand correctly, the input $\\textbf{s}$ to the kernel function $k_{\\phi}$ in the original DMPP lies in the space $\\mathcal{R} \\times \\mathcal{R}^2$, but in DKMPP, based on (7), the input to the kernel function is first mapped by a deep neural network $g_{w_2}: \\mathcal{R} \\times \\mathcal{R}^2 \\rightarrow \\mathcal{R}^D$ where $D$ can be much larger than the original dimension of $\\textbf{s}$. In other words, the **high-dimension** of the input is induced by our design choice of $g_{w_2}$ instead of the data, is that right? Can $\\textbf{s}$ itself be high-dimensional in spatio-temporal point processes?\n\nMinor comments:\n\n•\tSince $\\lambda(\\textbf{s}|\\mathcal{D})$ in (6) and (7) is an approximation of the true intensity of DMPP, I recommend using a different notation such as $\\hat{\\lambda}(\\textbf{s}|\\mathcal{D})$.\n\n•\tFigures 1(c)-(d) are displayed but never referred to. I suggest the authors either remove the figures or compare 1(c)-(d) to the true intensity function.\n\nQUESTIONS:\n• If $f_w$ is a network that outputs nonnegative mixture weights and $k_{\\phi}$ is positive semi-definite, why do we need the link function $\\eta(\\cdot)$ to ensure non-negativity in (7) but not in (6)?\n\n• Why do all point process models yield relatively low accuracy on the NYC Complaint data? Does this mean point process models are not good at processing textual information?\n\nLIMITATIONS:\nThe limitations are adequately addressed. I’m not aware of any direct negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that there are two common approaches for modeling intensity functions: traditional and covariate based methods, and this paper focuses on the latter one. In detail, the intensity function is designed in a kernel convolution form: $\\lambda(s|\\mathcal{D})=\\int f_w(\\mathbf{u},\\mathbf{Z(u)})k_{\\phi}(\\mathbf{s,\\mathbf{u}})d\\mathbf{u}$, in which contextual information is embedded in $\\mathbf{Z(u)}$. In practice, its integration is replaced by summation. This method makes integration $\\int \\lambda(s|\\mathcal{D})ds$ intractable. However, the kernels lack expressiveness both because of the unknown relationship between covariates and event occurrence and improper usage of Euclidean distance. To solve this issue, authors propose to use deep kernels which are modeled by neural networks. To address parameter estimation, the authors further propose to use of a score matching-based estimator to estimate parameters.\n\nSTRENGTHS:\n1. The theoretical part is reasonable and complete, and the proposed model effectively takes into account both solving the difficulty of integrating the intensity function and maintaining the strong expressiveness of the intensity function.\n\n3. The score matching-based modeling method is novel and interesting and plays a positive role in promoting research in the field of point processes.\n\n3. The advantages of the model are adequately and effectively demonstrated by experiments.\n\nWEAKNESSES:\n1. The motivation for using a score-based approach is not clear. In fact, the score-based approach is a special generation model. In that case, why not use another generation model, such as GAN or VAE? The authors point out that the score-based approach can effectively solve the parameter estimation problem, but it seems that other generative models can also solve the problem.\n\n2. The proposed model seems to be a combination of existing frameworks, which actually hinders its nolvity to some extend.\n\nQUESTIONS:\n1. I'm confused about the reason why the authors use score-based methods.\n\n2. Actually, there are some efforts concerning the embedding of generative models into point processes. Is there any comparison between the proposed model and some existing methods, see, for example [1-2].\n\n\n[1] Xiao S, Farajtabar M, Ye X, et al. Wasserstein learning of deep generative point process models[J]. Advances in neural information processing systems, 2017, 30.\n[2] Mehrasa N, Jyothi A A, Durand T, et al. A variational auto-encoder model for stochastic point processes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 3165-3174.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an enhanced version of Deep Mixture Point Processes with a flexible neural network-based kernel. The intractable training process of the point process with deep kernel is handled by a score-matching technique with the denoising method.\n\nSTRENGTHS:\n1. The proposed deep kernel goes beyond the parametric kernel and substantially improves the model flexibility and expressiveness. \n2. Address the learning challenges of MLE (a long-lasting problem in neural pp training) by proposing a score-matching technique which achieves better modeling performance and computational efficiency. This is also supported by the experimental results.\n\nWEAKNESSES:\n1. Related works are not comprehensive enough. There is plenty of work on point processes equipped with deep kernel, such as Okawa [1] and Zhu. [2]. These works can be reviewed to make the paper more comprehensive.\n2. More experiments can be included.  For example, one baseline of Hawkes process equipped with a deep kernel can be included. Also model performance on synthetic point process data (such as self-exciting point process or self-correcting process) would make the numerical results more convincing.\n\n---\n[1] Maya Okawa, Tomoharu Iwata, Yusuke Tanaka, Hiroyuki Toda, Takeshi Kurashima, and Hisashi Kashima. Dynamic hawkes processes for discovering time-evolving communities’ states behind diffusion processes. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1276–1286, 2021.\n[2] Shixiang Zhu, Haoyun Wang, Zheng Dong, Xiuyuan Cheng, and Yao Xie. Neural spectral marked point processes. In International Conference on Learning Representations, 2022.\n\nQUESTIONS:\n1. How do we calculate the model density $p_\\theta(\\tilde{S}_m)$ in equation 12?\n2. Could the authors include additional synthetic experiments, such as modeling self-exciting point process data, because such data is becoming ubiquitous and gaining popularity in recent research and real-world application.\n\nLIMITATIONS:\nThe current structure still imposes parametric constraints on the kernel. An alternative form of the kernel can be considered [1].\n\n---\n[1] Shixiang Zhu, Haoyun Wang, Zheng Dong, Xiuyuan Cheng, and Yao Xie. Neural spectral marked point processes. In International Conference on Learning Representations, 2022.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}