SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixuan Zhang
China-Austria Belt and Road Joint Laboratory on Artificial Intelligence and Advanced Manufacturing Hangzhou Dianzi University yixuan.zhang@hdu.edu.cn
Quyu Kong
Alibaba Group kongquyu.kqy@alibaba-inc.com
Feng Zhou *
Center for Applied Statistics and School of Statistics
feng.zhou@ruc.edu.cn
In this study, we propose a novel deep spatio-temporal point process model, Deep Kernel Mixture Point Processes (DKMPP), that incorporates multimodal covariate information. DKMPP is an enhanced version of Deep Mixture Point Processes (DMPP), which uses a more flexible deep kernel to model complex relationships between events and covariate data, improving the model's expressiveness. To address the intractable training procedure of DKMPP due to the non-integrable deep kernel, we utilize an integration-free method based on score matching, and further improve efficiency by adopting a scalable denoising score matching method. Our experiments demonstrate that DKMPP and its corresponding score-based estimators outperform baseline models, showcasing the advantages of incorporating covariate information, utilizing a deep kernel, and employing score-based estimators.
Point processes are widely used statistical tools for modeling event occurrence patterns within continuous spatio-temporal domains. They have garnered extensive attention across various fields, such as criminology [17; 38], neuroscience [24; 40], financial engineering [11; 2], and epidemiology [20; 25]. The core task of point processes is to derive an intensity function from event sequences, which represents the rate of event occurrences at any given time or location within the observation domain.
There are two common approaches for modeling intensity functions: traditional and covariate-based methods. The traditional approach [13; 10; 18] considers only event information, such as time and position, while ignoring the effects of contextual factors. Conversely, the covariate-based method [31; 33; 3] incorporates covariates from the observation domain, providing insights into factors that may cause event occurrences. In real applications, event occurrence rates are often correlated with contextual factors, making covariate-based methods more effective for event prediction [21]. For example, in crime data analysis, the covariate-based method considers factors like income level, education status, and public security to estimate crime intensity. Thus, utilizing a covariate-based point process model is significant for leveraging rich contextual information to predict events and explore various factors influencing event occurrence.
Okawa et al. [21] proposed Deep Mixture Point Processes (DMPP) to incorporate covariate information into point process models. DMPP models intensity as a deep mixture of parametric kernels, representing the influence of nearby representative points on the target event. Mixture weights are modeled by a neural network using covariates from representative points as input, avoiding intractable intensity integrals for log-likelihood evaluation and enabling simple parameter estimation. However, DMPP requires integrable parametric kernels like radial basis function (RBF) or polynomial kernels, which have closed-form solutions for intensity integrals. With more complex, flexible kernels (e.g., deep kernels [35]), DMPP cannot compute the intensity integral tractably. In real-world applications, flexible kernels are usually preferred to capture complex influences of covariates on intensity, which can be challenging for restricted parametric kernels.
In this study, we aim to learn the intricate influence of covariate data on event occurrence using a data-driven approach. We propose replacing the simple parametric kernel in DMPP with a deep kernel [35], which transforms the inputs of a base kernel using a deep neural network. The deep kernel significantly enhances expressive power by automatically learning a flexible input transformation metric, thus transcending Euclidean and absolute distance-based metrics. This results in a substantial improvement in DMPP's expressiveness, leading to our proposed model, Deep Kernel Mixture Point Processes (DKMPP). However, due to the deep architecture embedded in the base kernel, the deep kernel is non-integrable, causing an intractable training procedure as intensity integral evaluation is required for likelihood-based parameter estimation. To address parameter estimation in DKMPP, we adopt an integration-free method based on score matching [12], which estimates model parameters by matching the gradient of the log-density of the model and data. Score matching avoids intensity integral computation and allows tractable training. However, the naive score matching estimator is computationally expensive as it requires the computation of second derivatives of the log-density. To improve efficiency, we further adopt a scalable denoising score matching (DSM) method [32], with the resulting models referred to as score-DKMPP and score-DKMPP+ respectively.
Score-based estimators, such as Score-DKMPP and Score-DKMPP+, offer an advantage over traditional likelihood-based estimators as they do not require numerical integration, which is often computationally expensive and prone to errors. Additionally, Score-DKMPP+ has a high efficiency due to the utilization of denoising score matching. Our experiments serve to highlight the advantages of DKMPP and its corresponding score-based estimators. In the majority of our experiments, both Score-DKMPP and Score-DKMPP+ outperform the baseline models, indicating that incorporating covariate information can enhance the modeling and prediction of spatio-temporal events. Furthermore, our results strongly suggest that the adoption of a deep kernel significantly enhances the model's expressiveness when compared to a simple parametric kernel.
Specifically, our contributions are as follows: (1) We propose DKMPP, which replaces the simple parametric kernel in DMPP with a more flexible deep kernel, enhancing model expressiveness and allowing for the learning of complex influence from covariate factors on event occurrence. (2) We use the score matching estimator to overcome the intractable intensity integral in DKMPP during training and extend the denoising score matching method to DKMPP, significantly improving computational efficiency. (3) We analyze the difference between likelihood estimator and score matching estimators for DKMPP, demonstrate the superiority of DKMPP over baseline models, and show the impact of deep kernel and various hyperparameters through experiments.
Point processes are statistical tools for modeling event occurrence patterns, with intensity functions characterizing event rates. Essential models include Poisson processes [13], renewal processes [5], and Hawkes processes [10]. Poisson processes feature independent point occurrences and have been widely used in economics [8], ecology [34], and astronomy [28]. Renewal processes generalize Poisson processes for arbitrary event intervals. However, neither model is suitable for self-excitation in event patterns. Hawkes processes, containing a triggering kernel, are suitable for applications with excitation effects, such as seismology [19], finance [11], criminology [16], and neuroscience [39].
Deep point processes have gained traction due to the growing popularity of deep neural networks. While traditional models capture simple event patterns, deep point processes leverage neural networks to describe complex dependencies. The first deep point process work by Du et al. [7] utilized RNNs to encode history information. Later, Xiao et al. [36] and Mei & Eisner [14] proposed LSTM-based
versions. However, RNNs struggle with long-term dependencies, training difficulties, and lack of parallelism. To address these issues, Zuo et al. [43] and Zhang et al. [37] proposed attention-based models. Additionally, there are some point process models directly relevant to our work, such as Okawa et al. [22] and Zhu et al. [42], which employ the use of deep kernels to model the intensity function. Besides directly modeling intensity functions with neural networks, other approaches use them to represent probability density functions [29] or cumulative intensity functions [23; 30]. However, to the best of our knowledge, most deep point process models are limited to temporal point processes, and few works have extended them to spatio-temporal point processes.
Covariate point processes differ from traditional point processes by incorporating contextual factors and establishing connections between these factors and event patterns. This integration of covariate information enables a more comprehensive explanation of studied phenomena and improves predictive performance. Meyer et al. [15] proposed a model for epidemic prediction using population density and other covariates. Adelfio & Chiodi [1] proposed a spatio-temporal self-exciting point process for earthquake forecasting using geological features. Gajardo & Muller [9] analyzed COVID-19 cases with point processes, studying their relation to covariates such as mobility restrictions and demographic factors. However, these works model the influence of covariate in a handcrafted parametric form, limiting flexibility.
Different from existing works, our approach emphasizes covariate point processes and enhances both model flexibility and parameter estimation. While our work is closely related to [21], we extend it by introducing the deep kernel and score-based estimation methods in our proposed DKMPP model.
We offer a brief overview of three fundamental components that serve as the foundation for our work.
3.1 Spatio-temporal Point Processes
A spatio-temporal point process is a stochastic model whose realization is a random collection of points representing the time and location of events [6]. Consider a sequence of  $N$  events  $S = \{(t_1, \mathbf{x}_1), (t_2, \mathbf{x}_2), \ldots, (t_N, \mathbf{x}_N)\}$ , where  $(t_n, \mathbf{x}_n)$  denotes the time and location of the  $n$ -th event in  $\mathcal{T} \times \mathcal{X} \subseteq \mathcal{R} \times \mathcal{R}^2$ . The intensity function  $\lambda(t, \mathbf{x})$  represents the instantaneous event occurrence rate:
$$
\lambda (t, \mathbf {x}) = \lim  _ {| \Delta_ {t} |, | \Delta_ {\mathbf {x}} | \rightarrow 0} \frac {\mathbb {E} [ \mathbb {N} (\Delta_ {t} \times \Delta_ {\mathbf {x}}) ]}{| \Delta_ {t} | | \Delta_ {\mathbf {x}} |}, \tag {1}
$$
where  $\mathbb{N}(\Delta_t\times \Delta_{\mathbf{x}})$  is the number of events in  $\Delta_t\times \Delta_{\mathbf{x}}$ $\Delta_t$  is a small time interval around  $t$ $\Delta_{\mathbf{x}}$  is a small spatial region around  $\mathbf{x}$ , and  $\mathbb{E}$  denotes expectation w.r.t. realizations. Given a sequence  $S$  the probability density function, or likelihood, of the point process can be written as [6]:
$$
p (S \mid \lambda (t, \mathbf {x})) = \prod_ {n = 1} ^ {N} \lambda \left(t _ {n}, \mathbf {x} _ {n}\right) \exp \left(- \int_ {\mathcal {T} \times \mathcal {X}} \lambda (t, \mathbf {x}) d t d \mathbf {x}\right). \tag {2}
$$
3.2 Deep Kernel
Wilson et al. [35] combined kernel methods and neural networks to create a deep kernel, capturing the expressive power of deep neural networks. The deep kernel extends traditional covariance kernels by embedding a deep architecture into the base kernel:
$$
k _ {\phi} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right)\rightarrow k _ {\phi} \left(g _ {w} (\mathbf {x}), g _ {w} \left(\mathbf {x} ^ {\prime}\right)\right), \tag {3}
$$
where  $k_{\phi}$  is a base kernel with parameters  $\phi$ . To enhance flexibility, the inputs  $x$  and  $x'$  are transformed by a deep neural network  $g$  parameterized by weights  $w$ . The base kernel offers various options, such as the common RBF kernel. The deep kernel parameters include base kernel parameters  $\phi$  and neural network weights  $w$ . An advantage of deep kernels is their ability to learn metrics by optimizing input space transformation in a data-driven manner, rather than relying on Euclidean distance-based metrics, which are common in traditional kernels but may not always be suitable, especially in high-dimensional input spaces.
3.3 Score Matching
The classic maximum likelihood estimation (MLE) minimizes the Kullback-Leibler (KL) divergence between the parameterized model distribution and the data distribution, but requires an often intractable normalizing constant. Numerical integration can approximate the intractable integral, but its complexity grows exponentially with the input dimension. An alternative is to minimize the Fisher divergence, which does not require the normalizing constant. The Fisher divergence is defined as:
$$
F (p (x), q (x)) = \frac {1}{2} \mathbb {E} _ {p (x)} \| \nabla \log p (x) - \nabla \log q (x) \| ^ {2}, \tag {4}
$$
where  $p(x)$  and  $q(x)$  are two smooth distributions of the same dimension, and  $\| \cdot \|$  is a suitable norm (e.g.,  $\ell^2$  norm). Minimizing the Fisher divergence estimates model parameters by matching the gradient of the log-density of the model to the log-density of the data. The parameterized model density is defined as  $p_{\theta}(x) = \frac{1}{Z(\theta)} \tilde{p}(x \mid \theta)$ , where  $\theta$  represents the model parameters, and the normalizing constant  $Z(\theta) = \int \tilde{p}(x \mid \theta) dx$  is intractable. With Fisher divergence, we do not need to compute the normalizing constant, as the matched gradients do not depend on it. The gradient of log-density is referred to score, so the aforementioned method is also called score matching [12].
In this section, we introduce our proposed DKMPP and two score-based parameter estimation methods. Following Okawa et al. [21], we use the same notation. We assume a sequence of  $N$  events  $S = \{\mathbf{s}_1 = (t_1,\mathbf{x}_1),\mathbf{s}_2 = (t_2,\mathbf{x}_2),\ldots ,\mathbf{s}_N = (t_N,\mathbf{x}_N)\}$ , where  $\mathbf{s}_n = (t_n,\mathbf{x}_n)$  represents the time and location of the  $n$ -th event in  $\mathcal{T} \times \mathcal{X} \subseteq \mathcal{R} \times \mathcal{R}^2$ . We also assume  $K$  contextual covariates on the spatio-temporal region  $\mathcal{T} \times \mathcal{X}$ :  $\mathcal{D} = \{Z_1,Z_2,\dots,Z_K\}$ , where  $Z_{k}$  is the  $k$ -th covariate.
4.1 Deep Kernel Mixture Point Processes
Okawa et al. [21] developed a deep spatio-temporal covariate point process model called DMPP, capable of incorporating multimodal contextual data like temperature, humidity, text and image. The intensity of DMPP is designed in a kernel convolution form:
$$
\lambda (\mathbf {s} \mid \mathcal {D}) = \int f _ {w} (\mathbf {u}, \mathbf {Z} (\mathbf {u})) k _ {\phi} (\mathbf {s}, \mathbf {u}) d \mathbf {u}, \tag {5}
$$
where  $\mathbf{u} = (\tau, \mathbf{r})$  is a point in the region  $\mathcal{T} \times \mathcal{X}$ , and  $k_{\phi}(\mathbf{s}, \mathbf{u})$  is a kernel with parameters  $\phi$ .  $f_w$ , a deep neural network, inputs contextual data and outputs nonnegative mixture weights, with  $w$  representing the network parameters.  $\mathbf{Z}(\mathbf{u}) = \{Z_1(\mathbf{u}), \dots, Z_K(\mathbf{u})\}$  are covariate values at point  $\mathbf{u}$ .
Although the intensity formulation integrates multimodal contextual data, in reality, covariate data is only available on a regular grid. DMPP introduces finite representative points  $\{\mathbf{u}_j\}_{j=1}^J$  on the spatio-temporal domain, resulting in a discrete version<sup>2</sup>:
$$
\lambda (\mathbf {s} \mid \mathcal {D}) = \sum_ {j = 1} ^ {J} f _ {w} \left(\mathbf {u} _ {j}, \mathbf {Z} \left(\mathbf {u} _ {j}\right)\right) k _ {\phi} (\mathbf {s}, \mathbf {u} _ {j}). \tag {6}
$$
The discrete version of DMPP offers two significant advantages: (1) the intensity itself is tractable since it requires summation instead of integration; (2) the intensity integral is tractable as long as the kernel integral is tractable, as given by  $\int \lambda (\mathbf{s}\mid \mathcal{D})d\mathbf{s} = \sum_{j = 1}^{J}f_{w}(\mathbf{u}_{j},\mathbf{Z}(\mathbf{u}_{j}))\int k_{\phi}(\mathbf{s},\mathbf{u}_{j})d\mathbf{s}$ . For this reason, DMPP employs simple integrable kernels like RBF kernels, which can be integrated analytically, to simplify parameter estimation. This design choice ensures that the intensity integral is tractable, which is crucial for evaluating log-likelihood. Consequently, parameter estimation can be performed using straightforward back-propagation.
However, this approach has notable drawbacks. Firstly, using limited parametric kernels constrains the intensity's expressive power. Secondly, Euclidean distance in the kernel may not be a suitable measure of similarity, particularly in high-dimensional input spaces. In many applications, the
2The intensities in Eqs. (6) and (7) are approximations of the true intensity in Eq. (5). Since we only use the approximation in the following, we adopt the same notation without explicitly distinguishing between them.
relationship between covariates and event occurrence is complex and often unknown, making it desirable to model this influence using data-driven kernels with non-Euclidean distance metrics.
To address the issues mentioned earlier, we propose DKMPP, which replaces the simple integrable kernel in DMPP with the deep kernel [35]. The deep kernel captures the similarity between representative points and event points in both time and location. The deep kernel has greater expressiveness than the traditional kernel and can flexibly learn the metric's functional form through a neural network-based nonlinear transformation. Specifically, the intensity of DKMPP is designed as:
$$
\lambda (\mathbf {s} \mid \mathcal {D}) = \eta \left(\sum_ {j = 1} ^ {J} f _ {w _ {1}} \left(\mathbf {u} _ {j}, \mathbf {Z} \left(\mathbf {u} _ {j}\right)\right) k _ {\phi , w _ {2}} \left(\mathbf {s}, \mathbf {u} _ {j}\right)\right), \tag {7}
$$
where  $k_{\phi, w_2}(\mathbf{x}, \mathbf{x}') = k_{\phi}(g_{w_2}(\mathbf{x}), g_{w_2}(\mathbf{x}'))$  is the deep kernel, with  $\phi$  as the base kernel parameters and  $w_2$  as the weights of a nonlinear transformation  $g$  implemented by a deep neural network. In subsequent experiments, we found that, even for this three-dimensional problem, the deep kernel's ability to learn metrics from data outperforms the traditional Euclidean distance. Compared to DMPP, DKMPP introduces a link function  $\eta(\cdot)$  to ensure non-negativity of the intensity function<sup>3</sup>. Various functions, such as softplus, exponential, and ReLU, can be utilized as the link function. Unless otherwise specified, we choose softplus as  $\eta(\cdot)$  in this work.
**Representative Points** In this study, we adopt a fixed grid of representative points, which are evenly spaced both along the temporal axis and across the spatial region. The complete set of representative points is derived as the Cartesian product of the temporal and spatial representatives.
Implementation Description Both the kernel mixture weight network  $f$  and the non-linear transformation  $g$  in the deep kernel are implemented using MLPs with ReLU activation functions. To handle multimodal covariates such as numerical values, categorical values, and text, etc., we perform necessary feature extraction. If the covariates are numerical, we normalize them as necessary. If the covariates are categorical, we convert them to numerical type or one-hot encoding. If the covariates are textual, we use a Transformer architecture to extract features. If the covariates are multimodal, we fuse the multimodal information by concatenating the extracted features from multiple modules.
4.2 Parameter Estimation with Score Matching
The traditional approach to estimate point process is based on MLE where the log-likelihood is:
$$
\log p _ {\theta} (S) = \log p \left(\left\{\mathbf {s} _ {n} \right\} _ {n = 1} ^ {N} \mid \lambda_ {\theta} (\mathbf {s})\right) = \sum_ {n = 1} ^ {N} \log \lambda_ {\theta} (\mathbf {s} _ {n}) - \int_ {\mathcal {T} \times \mathcal {X}} \lambda_ {\theta} (\mathbf {s}) d \mathbf {s}. \tag {8}
$$
The intensity integral is a compensator which can be understood as a normalizing constant. The integration is intractable in most cases, and our proposed DKMPP is no exception. To solve this problem, we normally resort to numerical integration, e.g., Monte Carlo or quadrature, to obtain an approximation. However, these numerical methods introduce additional errors; more importantly, they are not scalable to the high-dimensional problem.
Score-DKMPP In this work, we propose an integration-free estimation method based on score matching [12] to estimate the model parameters in our proposed DKMPP. Surprisingly, the score-based estimator for point process model is largely unexplored in recent years. As far as we know, few works attempted to utilize score matching for the estimation of point processes. Sahani et al. [26] proposed a pioneering estimator in this field, linking score matching with temporal point processes. Here, we extend the score matching estimator to our covariate-based spatio-temporal DKMPP. We assume the ground-truth process generating the data has a density  $p(S)$  and design a parameterized model with density  $p_{\theta}(S)$  where  $\theta$  is the model parameter to estimate. A Fisher-divergence objective for our covariate-based spatio-temporal DKMPP is designed as:
$$
F (\theta) = \mathbb {E} _ {p (S)} \frac {1}{2} \sum_ {n = 1} ^ {\tilde {N}} \left(\frac {\partial \log p (S)}{\partial s _ {n}} - \frac {\partial \log p _ {\theta} (S)}{\partial s _ {n}}\right) ^ {2}, \tag {9}
$$
<sup>3</sup>In DMPP (Eq. (6)),  $f_{w}$  only produces non-negative weights. However, in DKMPP (Eq. (7)), we remove the non-negative constraint of  $f_{w}$  and introduce a link function to ensure the non-negativity of the intensity.
where  $s_n$  is any entry in vector  $\mathbf{s}_n$ , i.e.,  $s_n \in \mathbf{s}_n = (t_n, \mathbf{x}_n)$ ,  $\tilde{N}$  is the number of equivalent variables,  $\tilde{N} = N$  for 1-D point process, e.g., temporal point process;  $\tilde{N} = 2N$  for 2-D point process, e.g., spatial point process; and  $\tilde{N} = 3N$  for 3-D point process, e.g., spatio-temporal point process, etc. The optimal estimate of model parameter is given by  $\hat{\theta} = \arg \min_{\theta} F(\theta)$ .
However, the above loss cannot be minimized directly as it depends on the gradient of the ground-truth data distribution which is unknown. Following the derivation in Hyvarinen [12], this dependence can be eliminated by using a trick of integration by parts. Sahani et al. [26] assumed the log density of point process satisfies the proper smoothness and obtained a concise empirical loss. In this work, we prove that without those smoothness assumptions, we can still obtain the same empirical loss (proof provided in Appendix A):
$$
\hat {F} (\theta) = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {\tilde {N} _ {m}} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} \left(S _ {m}\right)}{\partial s _ {m , n}}\right) ^ {2} + \frac {\partial^ {2} \log p _ {\theta} \left(S _ {m}\right)}{\partial s _ {m , n} ^ {2}} + C _ {1}, \tag {10}
$$
where we take  $M$  sequences  $\{S_m\}_{m=1}^M$  from  $p(S)$ ,  $\tilde{N}_m$  is the number of equivalent variables on the  $m$ -th sequence,  $s_{m,n}$  is the  $n$ -th variable on the  $m$ -th sequence, the constant  $C_1$  does not depend on  $\theta$  and can be discarded. It is easy to see that, in Eq. (10), the intensity integral has been removed by the operation of gradient. We refer to our proposed DKMPP estimated by Eq. (10) as score-DKMPP.
Score-DKMPP+ A serious drawback of the objective in Eq. (10) is it requires the second derivative which is computationally expensive. To improve efficiency, we derive the denoising score matching (DSM) method [32] for our proposed DKMPP to avoid second derivatives. We add a small noise to the sequence  $S$  to obtain a noisy sequence  $\tilde{S}$  (we add noise to each variable  $\tilde{s}_n = s_n + \epsilon$ ), which is distributed as  $p(\tilde{S}) = \int p(\tilde{S} \mid S)p(S)dS$ . The DSM method uses the Fisher divergence between the noisy data distribution  $p(\tilde{S})$  and model distribution  $p_{\theta}(\tilde{S})$  as an objective: