Deep Linear Hawkes Processes

Authors: Yuxin Chang, Alex James Boyd, Cao Xiao, Taha Kass-Hout, Parminder Bhatia, Padhraic Smyth, Andrew Warrington
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Marked temporal point processes, state-space models, Hawkes processes

Abstract:
Marked temporal point processes (MTPPs) are used to model sequences of different types of events with irregular arrival times, with broad applications ranging from healthcare and social networks to finance. We address shortcomings in existing point process models by drawing connections between modern deep state-space models (SSMs) and linear Hawkes processes (LHPs), culminating in an MTPP we call the _deep linear Hawkes process_ (DLHP). The DLHP modifies the linear differential equations in deep SSMs to be stochastic jump differential equations, akin to LHPs. After discretizing, the resulting recurrence can be implemented efficiently using a parallel scan. This brings both linear scaling and parallelism to MTPP models. This contrasts with attention-based MTPPs, which scale quadratically, and RNN-based MTPPs, which do not parallelize across the sequence length. We show empirically that DLHPs match or outperform existing models across a broad range of metrics on eight real-world datasets. Our proposed DLHP model is the first instance of the unique architectural capabilities of SSMs being leveraged to construct a new class of MTPP models.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730170684234

thread_type:
official_review

<thread_object>
creator: reviewer_afmw

<content>
summary:
This paper introduces a novel marked temporal point process (MTPP) model called DLHP by drawing connections between linear Hawkes processes (LHPs) and deep state-space models (SSMs). Based on the SDE form of LHPs, DLHP models the intensity function by utilizing parameterization and parallelization techniques within SSMs. This approach achieves linear scalability and enables parallel processing, addressing the shortcomings of existing MTPP models. Experimental results demonstrate the effectiveness of DLHP.

strengths:
1. Modeling the intensity function with the proposed latent linear Hawkes (LLH) layer is a novel and promising idea. The proposed model has the advantages of linear scalability and parallelism, addressing the shortcomings of existing marked TPPs.
2. The background and the proposed method are presented clearly. The paper is well-written and easy to follow.
3. Experiments are conducted to evaluate the model's predictive performance in terms of log-likelihood and computational efficiency.

weaknesses:
1. While log-likelihood is an important metric for evaluating TPP model performance, this paper does not include experiments on the RMSE of time prediction—a metric commonly used in most baseline studies, such as RMTPP, SAHP, and THP.
2. Performing intensity recovery only on an inhomogeneous Poisson process is insufficient; it is also necessary to include recovery for other traditional TPPs, such as the Hawkes process.

Minor: The per-event log-likelihood in line 359 needs to be divided by the number of events N.

questions:
1. If the background intensity is directly extended to be a function of time t, can the existence of a positive solution for Eq.(9) still be ensured?
2. Is it possible to use other numerical methods, such as the Euler method, to discretize the proposed LLH layer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730255079539

thread_type:
official_review

<thread_object>
creator: reviewer_15be

<content>
summary:
This research paper introduces the Deep Linear Hawkes Process (DLHP), a novel approach to modeling Marked Temporal Point Processes (MTPPs) that bridges the gap between deep state-space models and MTPPs. The benefits of state space models (SSMs) compared to attention-based model were emphasized. The innovation lies in modifying stochastic jump differential equations that describes the intensity function $\lambda$ of MTPP to become linear differential equations in deep state-space models. Its applicability was tested on various benchmark dataset. Notably, this work marks the first successful adaptation of state-space model architectural features to create a new category of MTPP models.

strengths:
The paper introduces a new family of MTPP models that leverages SSM layers. Also, the stochastic jump differential equation was translated in terms of SSM. By stacking multiple layers more flexible intensity function can be generated, while input dependent characteristic enabled more complex behavior.

weaknesses:
Only NLL was used as an evaluation metric, whereas RMSE and accuracy is commonly used in prior works. Also, as a method for modeling a stochastic process, simulation study can help showing the effectiveness of the method.

questions:
- In Fig. 4, THP and IFTPP shows faster runtime when implemented in JAX. Is DLHP still faster when they are all implemented using JAX?
- Demonstration of the simulation study might be helpful to see wether the discretization of DLHP does not hurt the expressivity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685428658

thread_type:
official_review

<thread_object>
creator: reviewer_crnn

<content>
summary:
This paper presents an interesting idea of using deep state-space models (SSMs) to model marked temporal point processes. The authors draw a connection between the SSMs and linear Hawkes processes via their stochastic differential equations, and develop a tailored model that enjoys comparable or superior performance against neural marked TPP baselines. The overall presentation is clear and easy to follow.

strengths:
The idea of combining the deep SSMs with marked temporal point processes is interesting. The methodology/model architecture is well presented, easy for the readers to understand how these models are integrated together, via techniques like diagoalization, discretization, input-dependent parameters, etc. Meanwhile, the new model enjoys a better computational efficiency than other baselines when implemented with parallel scan.

weaknesses:
Though I enjoyed reading the paper, there are still a few aspects of the paper that I think can be improved or answered by the authors:

*Methodology*:

1. I am not very convinced about the use of the Hawkes-process-like mechanism in each layer. I understand the connection between a linear Hawkes process and the SSM via the stochastic differential equation, but what is the need/meaning/interpretation of using this connection in the model? Why use $AB$ instead of one matrix of size $P\times H$ (or use $E\alpha$ instead of one $P*K$ matrix) as the parameters in Equation 10? If the idea is to inherit or imitate the parameter structure in the linear Hawkes process, this structure of parameters is overwritten/ignored since there are multiple layers and non-linear transformations stacked together.

2. On the other hand, if the authors do want to imitate the parameter structure of the linear Hawkes process, then the diagonalization would assume no influence between different marks, which is unrealistic in practice. And there is also no interpretation of the learned $\Lambda$ and $\alpha$ in the experiments.

*Experiments*:

3. I would suggest that the authors conduct more synthetic data experiments and results to validate the model’s ability to recover the ground truth. Only one simple Poisson process example is not enough, and the real data experiments are not straightforward in terms of interpreting the model's effectiveness.

4. No standard deviation was reported for the metrics in the tables/figures. The results of DLHP look like a random win against baselines given no reported standard deviation, especially only on real-world datasets with no ground truth.

5. It would be better if there was a rigorous computational complexity analysis of the model and baselines since the authors claim that it is one of the main strengths of using SSMs in modeling marked TPPs.

questions:
1. The main text says $u_t^{(1)} = 0$ (line 231), but in figure 2 it says $u_t^{(1)}$ is the event embedding. I am confused about this.

2. Do the authors have any insights on how to choose the model architecture, given the performance discrepancy of the model using forward and backward mechanisms?

3. What do the authors mean by 1.4 times improvement from baselines in line 92? If it refers to the average rank of the DLHP, does it mean a lower number is better?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709901381

thread_type:
official_review

<thread_object>
creator: reviewer_pxvb

<content>
summary:
This paper introduces a new marked temporal point process (MTPP) model known as the deep linear Hawkes process (DLHP), which is developed by integrating deep state-space models (SSMs) and linear Hawkes processes (LHPs). Through the use of diagonalization and discretization techniques, the DLHP enables efficient implementation of closed-form update. The model has been shown to surpass current methods in terms of performance on a variety of benchmark tasks, excelling in metrics such as log-likelihood and runtime across different sequence lengths. A key contribution of the research is pioneering the use of SSMs' unique architecture to develop a new class of MTPP models, positioning the DLHP as a competitive model in applications.

strengths:
1. This work stands out for being the first effort in harnessing the distinctive architectural features of deep state-space models (SSMs) to develop marked temporal point process (MTPP) models. 
2. Efficient Implementation: the DLHP has demonstrated robustness, performance, computational efficiency, and extensibility in experiments. These qualities seem to make the DLHP a competitive contender as an out-of-the-box model for a variety of applications. 
3. Clear presentation: This paper presents its idea clearly.

weaknesses:
1. As the authors clarified, DLHP has a higher implementation complexity, and the model architectures for DLHP tends to be more complex compared to other models. Does the enhancement in performance contributes to this increased complexity (since increased model complexity may lead to higher model likelihood value)? It would be better to provide necessary analysis regarding this issue and add some experiments for illustration if possible.
2. The proposed method sacrifices some of the interpretability of latent dynamics and parameters that are characteristic of LHPs, which could impact the model's transparency and understanding, and limit its usability for high-stakes applications where understanding event interactions is crucial. Would you discuss possible solutions or ideas regarding this issue? The motivation stated in line 201-214 of page 4 may provide some clues and could be discussed more deeply.

questions:
The authors may also pay attention to the paper titled with “Mamba Hawkes Process”（https://arxiv.org/abs/2407.05302）and do a detailed comparison with it, also to further confirm the statement "the first instance of the unique architectural capabilities of SSMs being leveraged to construct a new class of MTPP models" in the paper.

See the weaknesses for other questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732001354900

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
# Global Response

Foremost, we thank all four reviewers for taking the time to thoroughly read and provide detailed feedback for our submission. 

We presented the Deep Linear Hawkes Process (DLHP), a novel adaptation of deep state-space models (SSMs) to marked temporal point process (MTPP) applications. Our model draws connections between classical Hawkes processes and deep SSMs, allowing us to use the unique capabilities of deep SSMs to parsimoniously define a point process model with linear work, parallelization, and without the need for additional decoding heads. **We find our model reliably matches or outperforms other methods in terms of: log-likelihood (mark, time, and total), runtime, next mark prediction accuracy, next mark calibration, and inter-arrival time calibration.**

We were very heartened to see all four reviewers praise and score highly the soundness and presentation of our work. The reviewers all then commented on the various positives of our method. We are very happy that all four reviewers clearly engaged deeply with our work, and feel they accurately captured the core contributions of our submission. 

We first comment on any feedback that was shared between reviews below, and directly respond to individual reviewer feedback and questions in separate comments. We are still working on some additional experiments and evaluations, as requested by the reviewers. We will post updates as a further comment when these are complete. These results, as well as promised revisions to the text, will be incorporated in an updated manuscript that we will upload before the end of the rebuttal period.

## 1. Log-Likelihood Results (Table 1) with Repeats

Reviewer CrNn requested repeatability results for our main set of experiments to account for uncertainty. We re-conducted our main log-likelihood experiments (Table 1) across five different random seeds and computed the mean (and standard deviation) for each model and dataset. We present these results here in a global response as we feel that these may be of interest to all reviewers. Higher ↑ log-likelihoods are better; lower ↓ average ranks are better. The best method is bolded, and the second-best method is shown in italics. **The key takeaway is that DLHP is still the best or second-best performing method, evidenced by achieving the best (lowest) average rank.**

| **Model** | **Amazon** ↑ | **Retweet** ↑ | **Taxi** ↑ | **Taobao** ↑ | **StackOverflow** ↑ | **Last.fm** ↑ | **MIMIC-II** ↑ | **EHRShot** ↑| **Avg. Ranking** ↓ |
|-|-|-|-|-|-|-|-|-|-|
| **RMTPP** | -2.136 (0.003) | -7.098 (0.217) | 0.346 (0.002) | 1.003 (0.004) | -2.480 (0.019) | -1.780 (0.005) | -0.472 (0.026) | -8.081 (0.025) |  6.1  |
| **NHP** | 0.129 (0.012) | **-6.348** (0.000) | _0.514_ (0.004) | 1.157 (0.004) | -2.241 (0.002) | -0.574 (0.011) | 0.060 (0.017) | _-3.966_ (0.058) | _2.9_  |
| **SAHP** | -2.074 (0.029) | -6.708 (0.029) | 0.298 (0.057) | -1.64 (0.083) | -2.341 (0.058) | -1.646 (0.083) | -0.677 (0.072) | -6.804 (0.126) |  5.6 |
| **THP** | -2.096 (0.002) | -6.659 (0.007) | 0.372 (0.002) | -1.712 (0.011) | -2.338 (0.014) | -1.712 (0.011) | -0.577 (0.011) | -7.208 (0.096) | 5.5  |
| **AttNHP** | 0.484 (0.077) | -6.499 (0.028) | 0.493 (0.009) | 1.259 (0.022) | -2.194 (0.016) | -0.592 (0.051) | -0.170 (0.077) | OOM | 3.1  |
| **IFTPP** | _0.496_ (0.002) | -10.344 (0.016) | 0.453 (0.002) | **1.318** (0.017) | _-2.233_ (0.009) | **-0.492** (0.017) | _0.317_ (0.052) | -6.596 (0.240) | _2.9_  |
| **DLHP (Ours)** | **0.781** (0.011) | _-6.365_ (0.003) | **0.522** (0.004) | _1.304_ (0.039) | **-2.163** (0.009) | _-0.557_ (0.046) | **1.243** (0.083) | **-2.512** (0.369) | **1.4** |


## 2. Synthetic Experiments

Reviewers CrNn and AFMw raise the good point of applying DLHP to synthetic datasets.  **We are currently in the process of benchmarking the performance of the DLHP against the true model and baselines on synthetic datasets drawn from randomly instantiated parametric temporal point processes.** We will report back on these results before the end of the rebuttal period, and commit to including them in any camera-ready paper. 
 

## 3. RMSE 

Reviewers 15be and AFMw also requested the evaluation of the RMSE metric from our experiments. **We are currently evaluating the RMSE for all the methods, and will report back on the results before the end of the rebuttal period.** We commit to including the RMSE results in any potential camera-ready paper. 


## Summary

We again thank the reviewers for taking time to read and review our paper. We will add another comment shortly when the remaining experimental results come in. We hope these comments and clarification help allay any questions you have. Please do not hesitate to reach out if you have further questions!

-- Submission 3981 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732522650664

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Experimental Results

comment:
# Follow-Up for Global Response


## 1. Log-Likelihood Results (Table 1) with Repeats
We have now updated the full log-likelihood results. **Our DLHP model is consistently comparable with or outperforms other baselines across all datasets.** 


## 2. Synthetic Experiments
As Reviewers CrNn and AFMw suggested, we have conducted additional experiments on multivariate Hawkes processes where the ground truth intensities are available. We have included the **full plots and details at the end of Appendix (page 31-32) in the revision**. After the discussion period, we will move it to the next subsection of synthetic Poisson experiments. **The main takeaway is that our model is able to recover the ground truth almost perfectly with the fewest parameters.** All models are doing a decent job recovering the ground truth intensities, where NHP and DLHP visually better recover the ground truth intensities.


## 3. RMSE 
Reviewers 15be and AFMw suggested including results for RMSE; we have performed the RMSE evaluation on all methods (lower is better, OOM refers to out-of-memory): 

| **Model** | **Amazon**  | **Retweet**  | **Taxi**  | **Taobao**  | **StackOverflow**  | **Last.fm**  | **MIMIC-II**  | **EHRShot** |
|-|-|-|-|-|-|-|-|-|
| **RMTPP** | 0.361 | 16152 | 0.284 | 0.127 | 1.054 | 15.864 | 0.754 | 3445 |
| **NHP** | 0.342 |  15322 | 0.283 | 0.127 | 1.028 | 15.841 | 0.739 | 3430 |
| **SAHP** | 0.345 |  16018 | 0.285 | 0.126 | 1.034 | 15.830 | 0.805 | 3418 |
| **THP** | 0.335 |  15848 | 0.286 | 0.126 | 1.029 | 15.878 | 0.781 | 3504 |
| **AttNHP** | 1.214 |  16220 | 1.273 | 0.130 | 1.311 | 15.889 | 0.852 | OOM |
| **IFTPP** | 0.332 |  2568 | 0.280 | 0.126 | 0.975 | 15.550 | 0.713 | 1899 |
| **DLHP (Ours, sampling)** | 0.395 | 15223 | 0.282 | 0.127 | 1.021 | 15.844 | 0.764 | 3421 |
| **DLHP (Ours, linear probe)** | 0.337 | 14642 | 0.283 | 0.128 | 1.038 | 15.710 | 0.810 | 3350 |

For all baseline methods, results were computed with empirical estimates of expected values via Monte Carlo samples; intensity-based methods were sampled using the thinning algorithm [1]. For our method, we showcase both sampling-based results as well as using a simple linear probe on the output hidden vector to directly estimate the next event time.
In general, we find that our method is comparable to the other intensity-based methods, and do not find substantial differences in RMSE between sampling and the linear probe. That being said, the linear probe was substantially faster (often around 10 times quicker) even when accounting for fine-tuning the linear probe parameters.

We would like to note, for context, that we did not originally include RMSE results for next-event time as it purely describes how well the model captures the expected value of the next-event time [2]. This is not informative when the inter-event times have a multimodal distribution—which we often encounter in practice with real data. Conversely, the time log-likelihood is a proper scoring rule [2,3] and handles multimodal distributions well. As we note in our paper, Daley & Vere-Jones [4] state “testing the model on the basis of its _forecasting performance_ amounts to testing the model on the basis of its _likelihood_” (emphasis added). Following the recommendations from Bosser, T. and Taieb, S.B. [5], we also report the total, time, and mark log-likelihoods, the time and mark calibration, as well as the next-event mark accuracy (see Appendix of original submission) for a comprehensive evaluation of the predictive/forecasting performance. 

[1] Ogata, Yosihiko. “On Lewis' simulation method for point processes.” _IEEE transactions on information theory 27_(1)  (1981): 23-31.

[2] Gneiting, Tilmann. “Making and evaluating point forecasts.” _Journal of the American Statistical Association_ 106.494 (2011): 746-762.

[3] Gneiting, Tilmann, and Matthias Katzfuss. “Probabilistic forecasting.” _Annual Review of Statistics and Its Application 1_(1) (2014): 125-151.

[4] Daley, Daryl J., and David Vere-Jones. _An introduction to the theory of point processes: volume I: elementary theory and methods_ (page 276). Springer New York, 2003.

[5] Bosser, T. and Taieb, S.B., “On the Predictive Accuracy of Neural Temporal Point Process Models for Continuous-time Event Data.” _Transactions on Machine Learning Research_, 2023.


## Summary
We thank again for the detailed feedback by the reviewers. We will incorporate all updates and new experimental results into the revised version of this paper. Please feel free to let us know if these answer your questions or if you have further questions! 


-- Submission 3981 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the Deep Linear Hawkes Process (DLHP), combining deep state-space models (SSMs) with linear Hawkes processes for marked temporal point process modeling. The authors claim DLHP offers linear scaling and parallelism advantages over existing models, reporting competitive or superior performance across various metrics on real-world datasets.

The paper's strengths include the integration of deep SSMs with MTPPs and clear methodology presentation. Reviewers noted the competitive performance on benchmark datasets and potential for improved computational efficiency.

Weaknesses identified include limited interpretability compared to classical Hawkes processes and insufficient synthetic experiments. Initially, there was a lack of comprehensive evaluation metrics beyond log-likelihood, which was later addressed in the rebuttal.

The primary reason for rejection is that the paper, while presenting an interesting combination of existing techniques, does not demonstrate sufficient novelty or practical impact to warrant publication at ICLR. As noted by the Area Chair, the work appears to be a straightforward combination of SSMs and stochastic jump differential equations, which is somewhat incremental. While the model shows competitive performance, the improvements are marginal and do not clearly justify the increased complexity.