SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is worth noting that Sahani et al. [26] assumed the parametric density satisfies the smoothness property:  $\partial_{t_n}\log p_\theta (S)|_{t_n = t_{n + 1}} = \partial_{t_{n + 1}}\log p_\theta (S)|_{t_{n + 1} = t_n}$  to cancel most delta functions. Here, we emphasize that this smoothness assumption is not necessary. In our derivation, we do not utilize this smoothness property, but just take advantage of the non-overlapping of timestamps to eliminate all delta functions and obtain the same objective.
A.2 Spatial Point Processes
Let us consider a planar point process for example. Given a 2-D observation region  $\mathcal{X} \subseteq \mathcal{R}^2$ , a realization from a 2-D spatial point process is composed of a random number of points  $S = \{\mathbf{x}_n\}_{n=1}^N$  where  $\mathbf{x}_n \in \mathcal{X}$  is the  $n$ -th event location (2-D coordinate). It is worth noting that these points are in no order. Similarly, we define a Fisher-divergence objective:
$$
F (\theta) = \mathbb {E} _ {p (S)} \frac {1}{2} \sum_ {n = 1} ^ {2 N} \left(\frac {\partial \log p (S)}{\partial x _ {n}} - \frac {\partial \log p _ {\theta} (S)}{\partial x _ {n}}\right) ^ {2}, \tag {17}
$$
where  $x_{n}$  is any entry in vector  $\mathbf{x}_n$ , i.e.,  $x_{n} \in \mathbf{x}_{n}$ .
Similarly, we use the trick of integration by parts to eliminate the dependence of the loss on the gradient of the unknown ground-truth data distribution. Let us expand Eq. (17), discard  $\left(\frac{\partial\log p(S)}{\partial x_n}\right)^2$  which does not depend on parameter  $\theta$ , and examine the cross-term:
$$
\begin{array}{l} \mathbb {E} _ {p (S)} \left[ \frac {\partial \log p (S)}{\partial x _ {n}} \frac {\partial \log p _ {\theta} (S)}{\partial x _ {n}} \right] \\ = \int p (S) \frac {\partial \log p (S)}{\partial x _ {n}} \frac {\partial \log p _ {\theta} (S)}{\partial x _ {n}} d S \\ = \int_ {S _ {x _ {n} ^ {-}}} \int_ {x _ {n}} \frac {\partial p (S)}{\partial x _ {n}} \frac {\partial \log p _ {\theta} (S)}{\partial x _ {n}} d x _ {n} d S _ {x _ {n} ^ {-}} \tag {18} \\ = \int_ {S _ {x _ {n} ^ {-}}} p (S) \frac {\partial \log p _ {\theta} (S)}{\partial x _ {n}} \Bigg | _ {x _ {n} = - \infty} ^ {x _ {n} = + \infty} - \int_ {x _ {n}} p (S) \frac {\partial^ {2} \log p _ {\theta} (S)}{\partial x _ {n} ^ {2}} d x _ {n} d S _ {x _ {n} ^ {-}} \\ = \mathbb {E} _ {p (S)} \left[ - \frac {\partial^ {2} \log p _ {\theta} (S)}{\partial x _ {n} ^ {2}} \right]. \\ \end{array}
$$
where  $S_{x_n^-}$  represents the realization excluding  $x_n$ , the fourth line uses integration by parts, the fifth line assumes a weak regularity condition:  $p(S)\partial_{x_n}\log p_\theta (S)$  goes to zero for any  $\theta$  when  $|x_{n}|\to \infty$ . It is worth noting that in spatial point processes the limits of  $x_{n}$  are no longer constrained because there is no order for the points. Therefore, the loss in Eq. (17) can be rewritten as:
$$
F (\theta) = \mathbb {E} _ {p (S)} \left[ \sum_ {n = 1} ^ {2 N} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} (S)}{\partial x _ {n}}\right) ^ {2} + \frac {\partial^ {2} \log p _ {\theta} (S)}{\partial x _ {n} ^ {2}} \right] + C _ {1}. \tag {19}
$$
where the constant  $C_1$  does not depend on  $\theta$  and can be discarded. Replacing the expectation by the empirical average, we obtain the final empirical loss:
$$
\hat {F} (\theta) = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {2 N _ {m}} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} (S _ {m})}{\partial x _ {m , n}}\right) ^ {2} + \frac {\partial^ {2} \log p _ {\theta} (S _ {m})}{\partial x _ {m , n} ^ {2}} + C _ {1}, \tag {20}
$$
where we take  $M$  realizations  $\{S_m\}_{m=1}^M$  from  $p(S)$ ,  $x_{m,n}$  is the  $n$ -th variable on the  $m$ -th realization.
A.3 Spatio-temporal Point Processes
It is interesting to see that both score matching estimators for temporal point processes and spatial point processes have the same empirical loss (see Eq. (16) and Eq. (20)) regardless of whether the points are sequential or not. Therefore, it is easy to draw the conclusion that for spatio-temporal point processes the empirical score matching estimator is:
$$
\hat {F} (\theta) = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {\tilde {N} _ {m}} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} \left(S _ {m}\right)}{\partial s _ {m , n}}\right) ^ {2} + \frac {\partial^ {2} \log p _ {\theta} \left(S _ {m}\right)}{\partial s _ {m , n} ^ {2}} + C _ {1}, \tag {21}
$$
where  $s_{m,n}$  is the  $n$ -th variable on the  $m$ -th sequence,  $\tilde{N}_m$  is the number of equivalent variables on the  $m$ -th sequence,  $\tilde{N}_m = N_m$  for 1-D point process, e.g., temporal point process;  $\tilde{N}_m = 2N_m$  for 2-D point process, e.g., spatial point process; and  $\tilde{N}_m = 3N_m$  for 3-D point process, e.g., spatio-temporal point process, etc.
B Denoising Score Matching Estimator
In Appendix A, we provide an estimator trying to match the gradient of the log-density of the point process model to the log-density of the point process data. However, the estimator requires the second derivatives, which is computationally expensive. To avoid this issue, following the derivation of Vincent [32], we derive a denoising score matching estimator.
Differently, the denoising score matching estimator tries to match the gradient of the log-density of the model to the log-density of the noisy point process data. We add a small noise to the sequence  $S$  to obtain a noisy sequence  $\tilde{S}$  (we add noise to each variable  $\tilde{s}_n = s_n + \epsilon$ ), which is distributed as  $p(\tilde{S}) = \int p(\tilde{S} \mid S)p(S)dS$ . Therefore, the Fisher divergence between the noisy data distribution  $p(\tilde{S})$  and model distribution  $p_{\theta}(\tilde{S})$  is:
$$
F _ {\mathrm {D S M}} (\theta) = \mathbb {E} _ {p (\tilde {S})} \frac {1}{2} \sum_ {n = 1} ^ {\tilde {N}} \left(\frac {\partial \log p (\tilde {S})}{\partial \tilde {s} _ {n}} - \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}}\right) ^ {2}, \tag {22}
$$
where  $\tilde{s}_n$  is any entry in the noisy vector  $\tilde{\mathbf{s}}_n$ , i.e.,  $\tilde{s}_n \in \tilde{\mathbf{s}}_n = (\tilde{t}_n, \tilde{\mathbf{x}}_n)$ ,  $\tilde{N}$  is the number of equivalent variables. Let us expand Eq. (22), discard  $(\frac{\partial \log p(\tilde{S})}{\partial \tilde{s}_n})^2$  which does not depend on parameter  $\theta$ , and examine the cross-term:
$$
\begin{array}{l} \mathbb {E} _ {p (\tilde {S})} \left[ \frac {\partial \log p (\tilde {S})}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} \right] \\ = \int p (\tilde {S}) \frac {\partial \log p (\tilde {S})}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} d \tilde {S} \\ = \int \frac {\partial}{\partial \tilde {s} _ {n}} \int p (\tilde {S} \mid S) p (S) d S \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} d \tilde {S} \tag {23} \\ = \iint p (S) \frac {\partial p (\tilde {S} \mid S)}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} d S d \tilde {S} \\ = \iint p (S) p (\tilde {S} \mid S) \frac {\partial \log p (\tilde {S} \mid S)}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} d S d \tilde {S} \\ = \mathbb {E} _ {p (S, \tilde {S})} \left[ \frac {\partial \log p (\tilde {S} \mid S)}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} \right]. \\ \end{array}
$$
Therefore, the loss in Eq. (22) can be rewritten as:
$$
\begin{array}{l} F _ {\mathrm {D S M}} (\theta) \\ = \sum_ {n = 1} ^ {\tilde {N}} \mathbb {E} _ {p (\tilde {S})} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}}\right) ^ {2} - \mathbb {E} _ {p (S, \tilde {S})} \left[ \frac {\partial \log p (\tilde {S} \mid S)}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} \right] + C \\ = \mathbb {E} _ {p (S, \tilde {S})} \sum_ {n = 1} ^ {\tilde {N}} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}}\right) ^ {2} - \frac {\partial \log p (\tilde {S} \mid S)}{\partial \tilde {s} _ {n}} \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}} + C \tag {24} \\ = \mathbb {E} _ {p (S, \tilde {S})} \frac {1}{2} \sum_ {n = 1} ^ {\tilde {N}} \left(\frac {\partial \log p (\tilde {S} \mid S)}{\partial \tilde {s} _ {n}} - \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}}\right) ^ {2} + C _ {2}, \\ \end{array}
$$
where the constants  $C$  and  $C_2$  do not depend on  $\theta$  and can be discarded. Replacing the expectation by the empirical average, we obtain the final empirical loss:
$$
\hat {F} _ {\mathrm {D S M}} (\theta) = \frac {1}{2 M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {\tilde {N} _ {m}} \left(\frac {\partial \log p \left(\tilde {S} _ {m} \mid S _ {m}\right)}{\partial \tilde {s} _ {m , n}} - \frac {\partial \log p _ {\theta} \left(\tilde {S} _ {m}\right)}{\partial \tilde {s} _ {m , n}}\right) ^ {2} + C _ {2}, \tag {25}
$$
(b) MLE-DKMPP
(c) Score-DKMPP
(d) Score-DKMPP+
Figure 2: The intensity function at  $t = 10$  estimated by three estimators (MLE-DKMPP, Score-DKMPP, Score-DKMPP+) with 1,000 Monte Carlo (MC) samples. Three estimators exhibit similar performance. Score-DKMPP and Score-DKMPP+ do not require Monte Carlo integration, and thus their estimation remain consistent regardless of MC samples. In contrast, MLE-DKMPP heavily relies on MC sampling and therefore its performance depends on the number of MC samples.
where we take  $M$  clean and noisy sequences  $\{S_m, \tilde{S}_m\}_{m=1}^M$  from  $p(S, \tilde{S})$ ,  $\tilde{s}_{m,n}$  is the  $n$ -th variable on the  $m$ -th noisy sequence,  $\tilde{N}_m$  is the number of equivalent variables on the  $m$ -th noisy sequence.
C.1 Synthetic Data
Data Simulation We generate a 3-D spatio-temporal point process synthetic dataset. The spatial observation  $\mathbf{x}$  spans the area of  $[0,1] \times [0,1]$ , while the temporal observation window covers the time interval of  $[0,10]$ . We assume a 1-D covariate function  $Z(\mathbf{u} = (\tau, \mathbf{r})) = (\mathcal{N}(r_1 \mid 0.5, 0.5) + \mathcal{N}(r_2 \mid 0.5, 0.5))$  on the domain. We set  $f_{w_1}(\mathbf{u}_j, Z(\mathbf{u}_j)) = 20Z(\mathbf{u}_j) + 0.1$ ,  $k_{\phi, w_2}(\mathbf{s}, \mathbf{u}_j) = k_{\phi}(g_{w_2}(\mathbf{s}), g_{w_2}(\mathbf{u}_j))$  where  $k_{\phi}$  is the RBF kernel  $k_{\phi}(\mathbf{x}, \mathbf{x}') = \exp(-\phi\|\mathbf{x} - \mathbf{x}'\|^2)$  with  $\phi = 100$  and  $g_{w_2}$  is a linear transformation  $g_{w_2}(\mathbf{s}) = \mathbf{s} + 0.1$ . We fix the representative points on a regular grid: 5 representative points evenly spaced on each axis, so there are  $5^3 = 125$  representative points in total. We use the thinning algorithm to generate 5,000 sequences according to the intensity function specified above. The statistics of the synthetic data are shown in Table 3.
Training Details We fit a DKMPP model to the synthetic data with the ground-truth representative points and an RBF base kernel. Both the kernel mixture weight network  $f$  and the non-linear transformation  $g$  in the deep kernel are implemented using MLPs with ReLU activation functions. Therefore, the learnable parameters are  $w_{1}, w_{2}, \phi$ . The intensity functions at  $t = 10$  estimated by three different estimators are shown in Fig. 2.
Table 3: The statistics of synthetic and real-world datasets.
Dataset	Covariate Dimension	# of sequences	average # of events per sequence
Synthetic	1	5,000	17
Crimes in Vancouver	1	1,096	87
NYC Vehicle Collisions	768	61	327
NYC Complaint Data	768	301	63
C.2 Real-world Data
Data Preprocessing The preprocessing details of three real-world datasets are shown below. We listed the statistics of three real-world datasets after preprocessing in Table 3.
Crimes in Vancouver This dataset is composed of more than 530 thousand crime records, including all categories of crimes committed in Vancouver from 2003 to 2017. Each crime record contains the time and location (latitude and longitude) of the crime. We split the data into multiple sequences by year, month and day. Then, we select events from 2013 to 2016, drop the NaN value, and scale the time and space into a volume of  $[0,1] \times [0,1] \times [0,10]$ . We select the categorical feature 'Crime Type' as the descriptive feature for each event. We convert the categorical feature into the corresponding numerical feature as the covariate. Finally, the dimension of covariate  $\mathbf{Z}$  is 1.
NYC Vehicle Collisions The New York City vehicle collision dataset contains about 1.05 million vehicle collision records. Each collision record includes the time and location (latitude and longitude). We split the data into multiple sequences by day. We select the records from 01/01/2019 to 02/03/2019, drop the NaN value and scale the time and space into a volume of  $[0,1] \times [0,1] \times [0,10]$ . We select 'BOROUGH', 'CONTRIBUTING FACTOR VEHICLE 1', 'CONTRIBUTING FACTOR VEHICLE 2' and 'VEHICLE TYPE CODE 1' as the descriptive features for each event. We concatenate several textual features and use a pre-trained DistilBERT [27] to extract the textual features, and concatenate the textual features with other numerical/categorical features as the covariate. Finally, the dimension of covariate Z is 768.
NYC Complaint Data This dataset contains over 228 thousand complaint records in New York City. Each record includes the date, time, and location (latitude and longitude) of the complaint. We split the data into multiple sequences by hour. We select the records from 01/11/2022 to 13/11/2022, drop the NaN value and scale the time and space into a volume of  $[0,1] \times [0,1] \times [0,10]$ . We select 'OFNS_DESC', 'JURIS_DESC', 'LAW Cats CD', 'PD_DESC', 'VIC_RACE', 'VIC_SEX' and 'PREM_TYP_DESC' as the descriptive features for each event. We concatenate several textual features and use a pre-trained DistilBERT [27] to extract the textual features, and concatenate the textual features with other numerical/categorical features as the covariate. Finally, the dimension of covariate Z is 768.
Training Details Each dataset is divided into training, validation and test data using a  $50\% / 40\% / 10\%$  split ratio based on time. For the real-world data, we fix the representative points on a regular grid: 5 representative points evenly spaced on each axis, so there are  $5^3 = 125$  representative points in total. We use three different kernel functions for comparisons: the RBF kernel  $k_{\phi}(\mathbf{x}, \mathbf{x}') = \exp(-\phi\|\mathbf{x} - \mathbf{x}'\|^2)$ , the rational quadratic (RQ) kernel  $k_{\phi}(\mathbf{x}, \mathbf{x}') = (1 + \phi\|\mathbf{x} - \mathbf{x}'\|^2)^{-\frac{1}{2}}$ , and the Ornstein-Uhlenbeck (OU) kernel  $k_{\phi}(\mathbf{x}, \mathbf{x}') = \exp(-\phi\|\mathbf{x} - \mathbf{x}'\|)$ . For the three real-world datasets, the kernel mixture weight network  $f$  and the non-linear transformation  $g$  in the deep kernel are implemented using MLPs with ReLU activation functions and we fixed the number of layers in  $f$  and  $g$  as 2.
Hyperparameters We tested the performance of Score-DKMPP and Score-DKMPP+ with different hyperparameters on three real-world datasets. We tested the number of representation points and network structures. For the representation points, we tested three different values, 64, 125, and 216, respectively. For the network structure, we tested the number of layers of 1, 2, and 4. When we tested the effect of representation points, we fix the network with 2 hidden layers and when we tested the effect of network structure, we fix the number of representation points as 125.
For the representation points, the accuracy of Score-DKMPP and Score-DKMPP+ overall have better performance when the number of representation points is 125. The accuracy tends to increase when
(a) Score-DKMPP
(b) Score-DKMPP+
(d) Score-DKMPP+
Figure 3: (a) The ACC performance of Score-DKMPP with the number of representation points of 64, 125 and 216; (b) the ACC performance of Score-DKMPP+ with the number of representation points of 64, 125 and 216; (c) the ACC performance of Score-DKMPP with the number of layers of 1, 2 and 4; (d) the ACC performance of Score-DKMPP+ with the number of layers of 1, 2 and 4.
the number of representation points increases from 64 to 125, however, except for the performance of Score-DKMPP on the complaint dataset, for other real-world datasets, the accuracy starts to drop.
For the network structure, the accuracy using Score-DKMPP on the Crimes in Vancouver and NYC Complaint data tends to increase as the number of layers increases. However, for Score-DKMPP+, we only capture a similar trend on the NYC Vehicle Collisions data. Other than those mentioned above, we do not observe a significant pattern when increasing the number of layers.