SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
F _ {\mathrm {D S M}} (\theta) = \mathbb {E} _ {p (\tilde {S})} \frac {1}{2} \sum_ {n = 1} ^ {\tilde {N}} \left(\frac {\partial \log p (\tilde {S})}{\partial \tilde {s} _ {n}} - \frac {\partial \log p _ {\theta} (\tilde {S})}{\partial \tilde {s} _ {n}}\right) ^ {2}, \tag {11}
$$
where  $\tilde{s}_n$  is any entry in the noisy vector  $\tilde{\mathbf{s}}_n$ , i.e.,  $\tilde{s}_n \in \tilde{\mathbf{s}}_n = (\tilde{t}_n, \tilde{\mathbf{x}}_n)$ . The optimal parameter estimate is given by  $\hat{\theta} = \arg \min_{\theta} F_{\mathrm{DSM}}(\theta)$ . To facilitate computation, the Fisher divergence in Eq. (11) can be further rewritten as (proof provided in Appendix B):
$$
\hat {F} _ {\mathrm {D S M}} (\theta) = \frac {1}{2 M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {\tilde {N} _ {m}} \left(\frac {\partial \log p \left(\tilde {S} _ {m} \mid S _ {m}\right)}{\partial \tilde {s} _ {m , n}} - \frac {\partial \log p _ {\theta} \left(\tilde {S} _ {m}\right)}{\partial \tilde {s} _ {m , n}}\right) ^ {2} + C _ {2}, \tag {12}
$$
where we take  $M$  clean and noisy sequences  $\{S_m, \tilde{S}_m\}_{m=1}^M$  from  $p(S, \tilde{S})$ ,  $\tilde{s}_{m,n}$  is the  $n$ -th variable on the  $m$ -th noisy sequence, the constant  $C_2$  does not depend on  $\theta$  and can be discarded. With a Gaussian noise  $\epsilon \sim \mathcal{N}(0, \sigma^2)$ , the conditional gradient term can be written in a closed form:  $\partial \log p(\tilde{S}_m \mid S_m) / \partial \tilde{s}_{m,n} = -(\tilde{s}_{m,n} - s_{m,n}) / \sigma^2$ . Equation (12) avoids the unknown ground-truth data distribution, the intractable intensity integral and the tedious second derivatives.
It is worth noting that the denoising score matching is not equivalent to the original score matching, because it is easy to see from Eq. (11) that the model is trained to match the score of the noisy data distribution  $p(\tilde{S})$  instead of the original  $p(S)$ . However, when the noise is small enough, we can consider  $p(\tilde{S}) \approx p(S)$  such that Eq. (12) is approximately equivalent to Eq. (10). We refer to our proposed DKMPP estimated by Eq. (12) as score-DKMPP+.
In the experimental section, we mainly analyze the difference between MLE and score matching for DKMPP, the improvement in performance of DKMPP over baseline models, as well as the impact of various hyperparameters.
5.1 DKMPP: MLE vs. Score Matching
Since our proposed DKMPP model employs a score-based parameter estimation method instead of traditional MLE, a natural question arises regarding the comparison between the two approaches. To
(a) RMSE w.r.t. Data Size
(b) RMSE w.r.t. MC
(c)  $\sigma^2 = 0.01$
Figure 1: (a) The RMSE performance of MLE-DKMPP, Score-DKMPP and Score-DKMPP+ with  $100\%$  and  $10\%$  training data (the number of Monte Carlo samples is fixed to 1,000); (b) the RMSE performance of three estimators with the number of Monte Carlo samples ranging from 10 to 150; (c) the learned intensity function at  $t = 10$  from Score-DKMPP+ with the noise variance  $\sigma^2 = 0.01$ ; (d) the learned intensity function at  $t = 10$  from Score-DKMPP+ with  $\sigma^2 = 10$ .
(d)  $\sigma^2 = 10$
evaluate this question, we generate a 3-D spatio-temporal point process synthetic dataset. The spatial observation  $\mathbf{x}$  spans the area of  $[0,1] \times [0,1]$ , while the temporal observation window covers the time interval of  $[0,10]$ . We design a 1-D covariate function on the domain, a kernel mixture weight function  $f_{w_1}(\mathbf{u}_j,Z(\mathbf{u}_j))$ , and a deep kernel  $k_{\phi ,w_2}(\mathbf{s},\mathbf{u}_j)$  with the RBF base kernel. More details are provided in Appendix C.1. We fix the representative points on a regular grid: 5 representative points evenly spaced on each axis, so there are  $5^3 = 125$  representative points in total. We use the thinning algorithm [19] to generate 5,000 sequences according to the ground truth specified above. The statistics of the synthetic data are provided in Appendix C.1. We try to fit a DKMPP model to the synthetic data with the ground-truth representative points and an RBF base kernel. Both the kernel mixture weight network  $f$  and the non-linear transformation  $g$  in the deep kernel are implemented using MLPs with ReLU activation functions. Therefore, the learnable parameters are  $w_{1},w_{2},\phi$ .
Two aspects of particular interest when comparing MLE and score matching estimators are consistency and computational cost. Consistency refers to the property that as the data size varies, the estimates produced by these methods converge to the true parameter values. The computational costs of these methods refer to the amount of time required to achieve a certain level of accuracy.
For consistency, we use the root mean square error (RMSE) between the estimated intensity and the ground-truth intensity as a metric. During the training of MLE, numerical integration is required to approximate the intractable intensity integral. In our experiments, we use Monte Carlo methods and fix the number of Monte Carlo samples to 1,000, which is sufficient to obtain good results. For Score-DKMPP+, we use a Gaussian noise  $\epsilon \sim \mathcal{N}(0,\sigma^2)$  with  $\sigma^2 = 0.01$ . Refer to Appendix C.1 for the intensity functions learned by different estimators. We test the RMSE of MLE-DKMPP, Score-DKMPP and Score-DKMPP+ under different data sizes (100% and 10%), as shown in Fig. 1a. Three estimators exhibit very similar RMSE performance; however, as the data size decreases, the errors of all three estimators increase slightly. Besides, we examine the effect of Monte Carlo samples on RMSE. Specifically, we use 100% data and increase the number of Monte Carlo samples from 10 to 150. The results are shown in Fig. 1b. Due to the reliance on Monte Carlo integration, MLE suffers from poor RMSE when the number of Monte Carlo samples is insufficient. As the number of Monte Carlo samples increases, the parameter estimation performance of MLE improves and eventually converges. In comparison, score matching estimators (Score-DKMPP and Score-DKMPP+) have an advantage because they only require event location and do not rely on Monte Carlo integration. Therefore, the parameter estimation performance of score matching estimators is independent of the number of Monte Carlo samples.
For computational efficiency, we record the running time required by three estimators to achieve the same level of accuracy. The results are shown in Fig. 1b, indicating that to reach the same RMSE, the Score-DKMPP requires 356.8 seconds, while the MLE is faster, taking 150.8 seconds. Furthermore, the Score-DKMPP+ is extremely fast, completing the task in only 29.4 seconds.
5.2 Real-world Data
In this section, we validate our proposed DKMPP model and its corresponding score-based estimators on real-world data from various domains. We also compare DKMPP against other popular spatial-temporal point process models.
Datasets We analyze three datasets from the fields of transportation and crime, with details of the datasets shown below. Each dataset is divided into training, validation and test data using a  $50\% / 40\% / 10\%$  split ratio based on time. The data preprocessing is provided in Appendix C.2.
Crimes in Vancouver<sup>4</sup> This dataset is composed of more than 530 thousand crime records, including all categories of crimes committed in Vancouver. Each crime record contains the time and location (latitude and longitude) of the crime.
NYC Vehicle Collisions<sup>5</sup> The New York City vehicle collision dataset contains about 1.05 million vehicle collision records. Each collision record includes the time and location (latitude and longitude).
NYC Complaint Data This dataset contains over 228 thousand complaint records in New York City. Each record includes the date, time, and location (latitude and longitude) of the complaint.
Covariates These three datasets contain not only the spatio-temporal information of the events but also multimodal covariate information. For example, the Crime in Vancouver dataset includes information on the cause and type of crime records; the NYC Vehicle Collisions dataset includes textual descriptions of the accident scene, such as borough and contributing factors; the NYC Complaint Data includes numerical and textual covariate information, such as the police department number and offense description. For every representative point, we identify the closest numerical/categorical/textual features in both time and space, which are used as covariates.
Baselines Deep temporal point process models have gained widespread use in recent years, e.g., Mei & Eisner [14]; Omi et al. [23]; Zhang et al. [37]; Zuo et al. [43]. These works primarily focus on modeling time information while neglecting spatial information. As far as we know, there has been limited research on deep learning approaches for spatial-temporal point process modeling. Here, for a fair comparison, we compare DKMPP against deep (covariate) spatial-temporal point process models. Specifically, the following baselines are considered: (1) the homogeneous Poisson process (HomoPoisson) [13]; (2) the neural spatio-temporal point processes (NSTPP) [4]; (3) the deep spatio-temporal point processes (DeepSTPP) [41]; (4) the DMPP [21].
HomoPoisson is a traditional statistical spatial-temporal point process model; NSTPP and DeepSTPP are deep spatial-temporal point process models that do not incorporate covariates, while DMPP and DKMPP are deep covariate spatial-temporal point process models that allow the inclusion of covariate information. For DMPP and DKMPP, we experiment with three different kernels: RBF kernel, rational quadratic (RQ) kernel and Ornstein-Uhlenbeck (OU) kernel (See Appendix C.2).
Metrics We use two metrics to evaluate the performance: test log-likelihood (TLL) and prediction accuracy (ACC). The TLL measures the log-likelihood on the test data, indicating how well the model captures the distribution of the data. The ACC evaluates the absolute difference between the predicted number of events and the actual number of events on the test data,  $1 - \left|\frac{\text{predicted} \# - \text{actual} \#}{\text{actual} \#}\right|$ , assessing how accurately the model predicts the number of events on the test data.
Results We evaluate the performance of all baseline models in terms of TLL and ACC. In our experiments, we use Monte Carlo methods for the baselines which need numerical integration and fix the number of Monte Carlo samples to 1,000 which is sufficient to obtain good results. For Score-DKMPP+, we use a Gaussian noise  $\epsilon \sim \mathcal{N}(0,\sigma^2)$  with  $\sigma^2 = 0.01$ . More training details are provided in Appendix C.2. The performance results are presented in Table 1, which show that Score-DKMPP and Score-DKMPP+ achieve similar performance with the same kernel, and both outperform other baseline models. As expected, HomoPoisson performs the worst due to its lack of flexibility in modeling varying intensity functions over time and space. In comparison to NSTPP and DeepSTPP, DKMPP performs better in two aspects. Firstly, DKMPP incorporates rich multimodal covariate information, which aids in modeling event occurrences. Secondly, DKMPP employs a score-based estimator instead of the MLE estimator, which avoids unnecessary errors caused by numerical integration. The comparison between DKMPP and DMPP with the same kernel constitutes an ablation study, which again verifies that using a deep kernel can provide stronger representational power and achieve better performance in both data fitting and prediction. This is because the deep
<sup>4</sup>https://www.kaggle.com/datasets/wosaku/crime-in-vancouver
<sup>5</sup>https://data.cityofnewyork.us/Public-Safety/NYPD-Motor-Vehicle-Collisions/h9gi-nx95
<sup>6</sup>https://data.cityofnewyork.us/Public-Safety/NYPD-Complaint-Data-Current-YTD/5uac-w243
Table 1: The performance of TLL and ACC (mean±std) for DKMPP and baseline models on three real-world datasets. For DMPP and DKMPP, we experiment with RBF kernel, RQ kernel and OU kernel. Both for TLL and ACC, higher values indicate better performance.
Model	Crimes in Vancouver	NYC Vehicle Collisions	NYC Complaint Data
TLL	ACC (%)	TLL	ACC (%)	TLL	ACC (%)
HomoPoisson	60.468	52.516	394.890	66.811	1.660	26.834
NSTPP	65.45±4.46	74.22±1.36	396.33±19.66	72.08±3.53	3.07±0.38	32.83±1.41
DeepSTPP	62.15±6.16	69.89±4.26	396.60±11.28	69.35±2.07	3.97±1.38	43.72±0.96
DMPP (RBF)	62.58±1.02	56.90±0.98	394.54±2.17	67.91±1.55	2.06±0.56	30.97±1.23
DMPP (RQ)	60.09±0.97	51.52±1.01	399.71±1.99	72.32±1.23	2.11±0.32	31.92±0.68
DMPP (OU)	60.85±1.13	52.59±0.96	402.54±2.49	74.27±1.03	2.02±0.21	29.95±0.57
Score-DKMPP (RBF)	69.06±0.68	78.71±0.37	409.02±0.92	79.67±0.65	2.96±0.11	43.40±0.78
Score-DKMPP (RQ)	69.55±0.39	80.16±1.01	411.03±1.12	79.42±0.79	3.04±0.23	44.57±0.26
Score-DKMPP (OU)	69.51±0.49	78.41±0.83	407.43±1.24	78.84±0.87	3.06±0.16	44.62±1.39
Score-DKMPP+ (RBF)	67.03±0.23	80.20±0.34	402.54±1.06	79.13±0.67	3.13±0.28	46.48±0.43
Score-DKMPP+ (RQ)	69.52±1.14	80.09±0.90	403.93±1.43	79.85±1.89	3.74±0.35	47.34±1.44
Score-DKMPP+ (OU)	68.78±0.90	80.03±0.69	400.32±1.16	79.86±1.08	4.28±0.11	47.36±0.79
Table 2: The RMSE performance (mean±std) of Score-DKMPP and Score-DKMPP+ on the synthetic dataset with various combinations of hyperparameters: for the representative points, we used 1-layer MLPs and a batch size of 100; for the number of layers, we used 125 representative points and a batch size of 100; and for the batch size, we used 125 representative points and 1-layer MLPs.
Model/RMSE	Representative Points	Number of Layers	Batch Size
64	125	216	1	2	4	50	100	200
Score-DKMPP	3.86±0.26	3.66±0.27	3.66±0.26	3.84±0.23	4.00±0.21	4.13±0.16	3.60±0.35	3.47±0.23	4.07±0.17
Score-DKMPP+	3.78±0.23	3.73±0.21	3.72±0.22	3.78±0.17	3.95±0.23	4.08±0.26	3.67±0.31	3.61±0.21	3.65±0.15
kernel can flexibly learn the functional form of the metric by optimizing input space transformation in a data-driven manner, rather than relying on a fixed Euclidean distance-based metric.
5.3 Hyperparameter Analysis
Representative Points The effect of the number of representative points on the performance of DKMPP on synthetic and real-world data is presented in Table 2 and Appendix C.2, respectively. In general, the accuracy of DKMPP tends to improve with an increase in the number of representative points, although the change in performance is not significant. However, having more representative points leads to higher computational costs. Therefore, in the experiment, the number of representative points needs to be chosen as a trade-off between accuracy and efficiency.
Layer and Batch Size We examine how the number of layers in the kernel mixture weight network  $f$  and the non-linear transformation  $g$  of the deep kernel, as well as batch size, affect model performance on synthetic and real data in Table 2 and Appendix C.2, respectively. In general, varying batch sizes do not produce a significant difference in model performance, so we set a batch size of 100 because both overly large and excessively small batch sizes may harm optimization. For the number of layers, we find that the number of layers is correlated with the dimension of covariates. In the case of low-dimensional covariates, a simple hidden layer can provide better performance than multiple layers. Thus, we set the number of layers to 1 for the synthetic data and 2 for the real data.
Noise Variance The noise added by the Score-DKMPP+ is an important hyperparameter. If the noise is too large, the resulting noisy distribution  $p(\tilde{S})$  will deviate significantly from the true data distribution  $p(S)$ , leading to severe biases in the learned results. We conduct experiments with two different noise variance settings,  $\sigma^2 = 0.01$  and  $\sigma^2 = 10$ . When  $\sigma^2 = 0.01$ , the model can learn the intensity function that is close to the ground truth, as shown in Fig. 1c. However, as the noise level increases, when  $\sigma^2 = 10$ , the learned results deviate significantly from the true values, as shown in Fig. 1d. In practice, setting  $\sigma^2$  too small can also lead to numerical instability.
One limitation of the current study is that the proposed multimodal covariate spatio-temporal point process model is constrained to covariates of numerical, categorical, and textual types during the experiments, primarily due to the availability of data. However, future research could greatly benefit from the incorporation of image-type covariates, as this holds immense potential for enhancing the model's ability to capture a wider range of diverse and intricate effects. By including image data, the model could potentially uncover richer patterns and relationships, leading to even more fascinating and insightful results.
In conclusion, we have introduced a novel deep spatio-temporal point process model, the DKMPP, which incorporates multimodal covariate information and captures complex relationships between events and covariate data by utilizing a more flexible deep kernel leading to improved expressiveness. We address the intractable training procedure of DKMPP by using an integration-free method based on score matching and improving efficiency through a scalable denoising score matching method. Through our experiments, we show that DKMPP and its corresponding score-based estimators outperform baseline models, demonstrating the advantages of incorporating covariate information, utilizing a deep kernel, and employing score-based estimators. Our results suggest that DKMPP is a promising approach for modeling complex spatio-temporal events with multimodal covariate data.
This work was supported by NSFC Project (No. 62106121), the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD110001), the fund for building world-class universities (disciplines) of Renmin University of China (No. KYGJC2023012), China-Austria Belt and Road Joint Laboratory on Artificial Intelligence and Advanced Manufacturing of Hangzhou Dianzi University, and the Public Computing Cloud, Renmin University of China.
[1] Adelfio, G. and Chiodi, M. Including covariates in a space-time point process with application to seismicity. Statistical Methods & Applications, 30(3):947-971, 2021.
[2] Bacre, E., Delattre, S., Hoffmann, M., and Muzy, J.-F. Modelling microstructure noise with mutually exciting point processes. Quantitative Finance, 13(1):65-77, 2013.
[3] Baddeley, A., Chang, Y.-M., Song, Y., and Turner, R. Nonparametric estimation of the dependence of a spatial point process on spatial covariates. Statistics and its interface, 5(2): 221-236, 2012.
[4] Chen, R. T. Q., Amos, B., and Nickel, M. Neural spatio-temporal point processes. In International Conference on Learning Representations, 2021.
[5] Cinlar, E. Markov renewal theory. Advances in Applied Probability, 1(2):123-187, 1969.
[6] Daley, D. J. and Vere-Jones, D. An introduction to the theory of point processes. vol. i. probability and its applications, 2003.
[7] Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Krishnapuram, B., Shah, M., Smola, A. J., Aggarwal, C. C., Shen, D., and Rastogi, R. (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 1555-1564. ACM, 2016.
[8] Dufresne, F. and Gerber, H. U. Risk theory for the compound poisson process that is perturbed by diffusion. Insurance: mathematics and economics, 10(1):51-59, 1991.
[42] Zhu, S., Wang, H., Dong, Z., Cheng, X., and Xie, Y. Neural spectral marked point processes. In International Conference on Learning Representations, 2021.
[43] Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer Hawkes process. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 11692-11702. PMLR, 2020.
A Score Matching Estimator
We provide the proof of the score matching estimator for temporal point processes and spatial point processes below, respectively. Generally speaking, the derivation for temporal point processes is more complex than spatial point processes because as we can see later there exists some complications arising from different limits of integration due to the order constraint on timestamps.
A.1 Temporal Point Processes
Given an observation window  $[0, T]$ , a sequence from a temporal point process is composed of a random number of timestamps arranged in a sequential order  $S = \{t_n\}_{n=1}^N$  where  $t_1 < t_2 < \ldots < t_N$  and  $t_n \in [0, T]$  is the  $n$ -th event timestamp. We assume the ground-truth process generating the data has a density  $p(S)$  and design a parameterized model with density  $p_\theta(S)$  where  $\theta$  is the model parameter to estimate. Following Sahani et al. [26], we define a Fisher-divergence objective:
$$
F (\theta) = \mathbb {E} _ {p (S)} \frac {1}{2} \sum_ {n = 1} ^ {N} \left(\frac {\partial \log p (S)}{\partial t _ {n}} - \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}}\right) ^ {2}. \tag {13}
$$
The above loss can be understood as matching the variational derivatives of log-density w.r.t. the counting process  $N(t)$  given that the counting process is non-decreasing and piece-wise constant with unit steps.
However, the above loss cannot be minimized directly as it depends on the gradient of the ground-truth data distribution which is unknown. Following the derivation of Hyvarinen [12], this dependence can be eliminated by using a trick of integration by parts. Let us expand Eq. (13), discard  $\left(\frac{\partial\log p(S)}{\partial t_n}\right)^2$  which does not depend on parameter  $\theta$ , and examine the cross-term:
$$
\begin{array}{l} \mathbb {E} _ {p (S)} \left[ \frac {\partial \log p (S)}{\partial t _ {n}} \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}} \right] \\ = \int p (S) \frac {\partial \log p (S)}{\partial t _ {n}} \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}} d S \\ = \int_ {S _ {t _ {n} ^ {-}}} \int_ {t _ {n}} \frac {\partial p (S)}{\partial t _ {n}} \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}} d t _ {n} d S _ {t _ {n} ^ {-}} \tag {14} \\ = \int_ {S _ {t _ {n} ^ {-}}} p (S) \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}} \Bigg | _ {t _ {n} = t _ {n - 1}} ^ {t _ {n} = t _ {n + 1}} - \int_ {t _ {n}} p (S) \frac {\partial^ {2} \log p _ {\theta} (S)}{\partial t _ {n} ^ {2}} d t _ {n} d S _ {t _ {n} ^ {-}} \\ = \mathbb {E} _ {p (S)} \left[ \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}} \left(\delta \left(t _ {n} - t _ {n + 1}\right) - \delta \left(t _ {n} - t _ {n - 1}\right)\right) - \frac {\partial^ {2} \log p _ {\theta} (S)}{\partial t _ {n} ^ {2}} \right]. \\ \end{array}
$$
where  $S_{t_{\overline{n}}}$  represents the sequence excluding  $t_n$ , the fourth line uses integration by parts, the fifth line uses delta function to evaluate the limits of  $t_n$  given the order constraint of timestamps. Therefore, the loss in Eq. (13) can be rewritten as:
$$
\begin{array}{l} F (\theta) = \mathbb {E} _ {p (S)} \left[ \sum_ {n = 1} ^ {N} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}}\right) ^ {2} \right. \tag {15} \\ \left. - \frac {\partial \log p _ {\theta} (S)}{\partial t _ {n}} (\delta (t _ {n} - t _ {n + 1}) - \delta (t _ {n} - t _ {n - 1})) + \frac {\partial^ {2} \log p _ {\theta} (S)}{\partial t _ {n} ^ {2}} \right] + C _ {1}. \\ \end{array}
$$
where the constant  $C_1$  does not depend on  $\theta$  and can be discarded. To construct the final empirical loss, we replace the expectation by the empirical average, which eliminates the delta functions as any two timestamps cannot overlap with each other, and obtain the final version:
$$
\hat {F} (\theta) = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {N _ {m}} \frac {1}{2} \left(\frac {\partial \log p _ {\theta} \left(S _ {m}\right)}{\partial t _ {m , n}}\right) ^ {2} + \frac {\partial^ {2} \log p _ {\theta} \left(S _ {m}\right)}{\partial t _ {m , n} ^ {2}} + C _ {1}, \tag {16}
$$
where we take  $M$  sequences  $\{S_m\}_{m=1}^M$  from  $p(S)$ ,  $t_{m,n}$  is the  $n$ -th timestamp on the  $m$ -th sequence.