Decoupled Marked Temporal Point Process using Neural Ordinary Differential Equations
Authors: Yujee Song
Donghyun LEE
Rui Meng
Won Hwa Kim
Venue: ICLR 2024 poster (2024)
Keywords: Marked Temporal Point Process
Neural ODE
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: A Marked Temporal Point Process (MTPP) is a stochastic process whose realization is a set of event-time data. MTPP is often used to understand complex dynamics of asynchronous temporal events such as money transaction, social media, healthcare, etc. Recent studies have utilized deep neural networks to capture complex temporal dependencies of events and generate embedding that aptly represent the observed events. While most previous studies focus on the inter-event dependencies and their representations, how individual events influence the overall dynamics over time has been under-explored. In this regime, we propose a Decoupled MTPP framework that disentangles characterization of a stochastic process into a set of evolving influences from different events. Our approach employs Neural Ordinary Differential Equations (Neural ODEs) to learn flexible continuous dynamics of these influences while simultaneously addressing multiple inference problems, such as density estimation and survival rate computation. We emphasize the significance of disentangling the influences by comparing our framework with state-of-the-art methods on real-life datasets, and provide analysis on the model behavior for potential applications.

Published as a conference paper at ICLR 2024
DECOUPLED MARKED TEMPORAL POINT PROCESS USING NEURAL ODES
Yujee Song Lee Dong Hyun"
Rui Meng
Won Hwa Kim *Amazon.com com Inc. zmmenge amazon. com
f Pohang University of Science and Technology
{yujees, dongtak, wonhwa]@posstech.oo ac.
ABSTRACT
A Marked Temporal Point Process (MTPP) is a stochastic process whose real- ization is a set of event-time data. MTPP is often used to understand complex dynamics of asynchronous temporal events such as money transaction, social me- dia, healthcare, etc. Recent studies have utilized deep neural networks to capture complex temporal dependencies of events and generate embeddings that aptly represent the observed events. While most previous studies focus on the inter- event dependencies and their representations, how individual events influence the overall dynamics over time has been under-explored.
this regime, we propose Decoupled MTPP framework that disentangles characterization of stochastic process into a set of evolving influences from different events. Our approach em- ploys Neural Ordinary Differential Equations (Neural ODEs) (Chen al. 2018) to learn flexible continuous dynamics of these influences while simultaneously addressing multiple inference problems, such as density estimation and survival rate computation. We emphasize the significance of disentangling the influences by comparing our framework with state-of-the-ee methods on real-life datasets, and provide analysis on the model behavior for potential applications.
INTRODUCTION
Event-time data, which are found across various practical domains such as social media (Zhao et al.
2015 Leskovec & KrevI 20144). healthcare (Du al.] [016]] Meng al. (023) and stock changes [Bacry et al.) 2015 Hawkes] 2018), comprise timelines of events of diverse types. Modeling such temporal data as stochastic process, one seeks to predict time and type of the future events based on the history, i.e., previously observed sequential events. For example, history of purchases of a person may tell when the person will buy new item, and a short message from a famous social media influencer could affect critical bull or bear in stock market. Predicting such future event often realized as a chance of the event, i.e., a likelihood of the event e at a specific time t, which constitutes probability distribution function (pdf) fee
Classically, the event prediction has been handled using Temporal Point Process (TPP), which is a stochastic process whose realization is set of ordered time (Rasmussen) 2018). TPP characterizes at which time point an event will occur given historical time points. Hawkes Process effectively models this stochastic behavior by defining an conditional intensity function which leverages the sum of excitation functions induced by past events called "self-excite" characteristics (Hawkes) 1971 Adamopoulos) 1976). The Hawkes Process independently models the influence of each event and incorporates the entire event cases using a simple linear operation (i.e., summation) of the influences offering interpretabilityy
While various models for TPP are adept at modeling dynamically forthcoming events, they do not incorporate event-related details such as location, node-specific information (particularly when ob- servations pertain to graphs such as multi-sensor or social networks), categorical data, and contextual information (e.g., tokens, images, or textual descriptions). Marked Temporal Point Process (MTPP) addresses it with "marks" which denotes the type of events, which elevates the traditional TPP of univariate event to multiple marks (i.e.. types) of the event. Many recent approaches for the MTPP leverage combinations of the intensity function in the Hawkes Process with neural networks to flexi- bly characterize the probability density function (pdf) of marks (Mei & Eisner 2017 Dia & Benson 2019 Zhang al. 2020 Chen al. 2021 Yang al. 2022 This line of works includes Recurrent

---

Published as a conference paper at ICLR 2024
Marked Temporal Point Process (Du et al. 2016), Neural Hawkes Process (Mei & Eisner 2017 and Transformer Hawkes Process (uuo et al. 2020) that effectively models the behavior of MTPP.
The aformentioned approaches successfully handled the complicated dependencies of intensity func- tions with the neural network, however, they come at the cost of interpretability of the dependencies among events. Existing methods often directly estimate the probability of the event across time and marks as a whole, or the intensity function as a proxy is approximated without considering the dynamics of influences from various marks and time. Moreover, reconstruction of the pdf of marks from the intensity function as well as other characteristic functions such as survival function require several integration operations without closed foosed-form solution, which must be re-evaluated whenever a new event is considered during inference resulting in exponential increase in computation time.
In this regime, we introduce novel decoupled MTPP framework characterized by decoupled hidden state dynamics, utilizing latent continuous dynamics driven by neural ordinary differential equations (ODE), named as Dec-ODE. The hidden state dynamic is modeled separately across each event, and it explains how each event separately contributes to the overall stochastic process with high fidelity. Moreover, such a decoupling approach equipped with the ODE lets our model train on the continuous dynamics of the hidden states in parallel, which results in substantial decrease in computation time compared to previous approaches that sequentially compute the dynamics from a series of events. The formulation of Dec-ODE leads to the contributions summarized as follows:
We propose novel MTPP modeling framework, i.e., Dec-ODE, which decouples indi- vidual events within stochastic process for efficient learning and explainability of each mark,
Dec-ODE models the continuous dynamies of influences from individually decoupled events by leveraging Neural ODEs,
Dec-ODE demonstrates ofte-of-thh-an performance on various benchmarks for MTPP val- idation while effectively capturing inter-time dynamics with interpretability.
BACKGROUND
Temporal Point Process (TPP) is a stochastic process whose realization is a set of ordered times (t)) The observed events until time t; is denoted as history Hr. The time of the next event, given He,, can be characterized using a pdf f())),, == (t), which denotes a probability that a subsequent event occurs during the interval tt, dt) conditioned on He, Its cumulative density function (cdf) is denoted as Ft HG,), and a survival function denoted as S(t) Hr.) = 1-FFtt HG, where both are conditioned on Hig- Throughout this paper, will be used to state that a function is conditioned on Hi.-
It is often difficult to define fixed functional form for (t) due its unknown pattern and the constraint 100 f*(ssds = 1. Therefore, an intensity function (t) = 5*( I-F'(t)
(((()
(((()
is often
introduced for the characterization of the stochastic process (Rasmussen 2018). The (t) > 0
being the only constraint, has been commonly used to model a TPP with flexibility (Hawkes 1971
Mei & Eisner 2017: Omi et al. 2019 Dia & Benson 2019 Zhang et al. 2020) Zuo et al., 2l.20 Chen et al. 2021 Yang et al. 2022). A conditional intensity function, X (*()), utilizes observed events H. calculate the intensity at 1, and (I)dl = p(l. E [1,(+d())7,) d().
Marked Temporal Point Process (MTPP) is a random process whose realization is a set of events S = M. where e; (t,, kz). Here, the k; {1, {I,..., K} denotes class of the event, i.e., mark, and t E IR is the time of its occurrence with t; < f.+1. The entire sequence prior to time is denoted
the history He, {(t,, ky) E S t2 t:). The conditional intensity function
X k) := >(t, k(H()) for MTPP with a mark & is often conveniently written as product of ground
intensity X; and conditional intensity of marker (k((t) given time as (Rasmussen 2018 De
et al. 2019 Daley & Vere-Jones) 2003):
(), = (((1))(((
f(t))))))
f(t, k(ee)
1- F((HL))
1 F((H.))
2

---

Published as a conference paper at ICLR 2024
where (t) is the probability with respect to time, and (k(t) is the probability with respect to
mark. Since (t, le) can be expressed in two separate components x- and o (k(t), the likelihood
of the MTPP can be expressed as follows (Daley & Vere-Jones 2003):
[N,(())
[GGtt II
exp (-/ ( Az(u)du) du
k(t,k)
II
Here, the Ng is counting process characterized by the ground intensity function (t), where the mark of the events is ignored.
Hawkes Process is widely utilized for modeling point processes with self-excitatory behavior. It describes a TPP, where each event triggers subsequent events. Formally, the Hawkes process is defined by an intensity function X*(() as (Adamopoulos) 1976):
X (t) ++
Σ 1lt - 4)
(3)
G
where denotes the time interest, is a non-negan non-negative constant, ju(t) is an excitatory function that
models the impact of past events on future event rates, and I denotes past event time points. The
Hawkes process has been broadly studied in machine learning (Yang et al. 2022 Mei & Eisner
2017 Zuo et al. 2020 Zhang al. 2020) with practical applications (Hawkes 2018
Cai et al.
2018
Du et al. 2015).
The Neural Ordinary Differential Equations (Neural ODEs) blends neural networks with differ-
ential equations, offering an interesting approach to deep learning (Chen et al. 2018) by employing
dontinuous-depth models instead of fixed-laye architecture. Mathematically, they define continuous transformations of network outputs governed by ordinary differential equations (ODEs):
dz(tt dt = (()))
(4)
where 7(-10) is neural network parameterized by to approximate changes of a hidden state z(t) with respect to time In our framework, the neural ODEs is employed to model the continuous dynamics of how each event affects the overall MTPP.
DECOUPLING MARKED TIME POINT PROCESS
Our approach focuses on establishing a general framework that captures the high-fidelity distri- bution of MTPP with decoupled structures. Specifically, given sequence of events denoted as == (e where an event e; = (1s, k;) is composed of time point and a marker, with + observed events, our goal to predict probability distribution of an event 6x+1- Unlike previous works that directly estimate the conditional intensity (1, k), we take an alternative approach of separately estimating the ground intensity and the distribution event, i.c., (t) and (k(t). In our framework, the influences from each events through time are represented by hidden state dynam- described in Fig I (z(t) and f f*(k (k)t) form independent trajectories, and they are combined in the later stage constitute X (t, k).
ies hut;; e,), which undergo a decoding process to yield (t) and f (k(t) separately.
3.1 HIDDEN STATE DYNAMICS
In this section, we introduce decoupled hidden state dynamics, where the hidden state used to characterize the influence each event e; has the overall process. The decoupled hidden states h(t;e;) gd, where = 1, n, are independently propagated through time from the occur- rence of each event t, The dynamics of hitt; ez), which will be used to characterize the complex dynamics of the intensity function within MTPP, are trained using the Neural ODEs (Chen et al, 2018).
Specifically, the initial magnitude hidden state h(t;; e;) depends on the event type k;, and the dynamics are solved via initial value problem (IVP) solvers. Then the hidden state caused by
3

---

Published as a conference paper at ICLR 2024
e(
lih Cln Mph
(C, (2(0)x((((1)))
h(t: (g)
(utt: e:)
Ag (1135))
22 23
Fp
0,
Eg
visualization of the overall framework. Hidden states from each event e; are independently propagated and decoded into trajectories e();es) and fx (k]f,, ez). Each trajectory represents the effect of each event on the MTPP, and the MTPP can be reconstructed by combining all trajectories.
Figure 1:
event e, at time can be obtained as:
dh(t: e,) -h((t kz
(5)
h(t;es)) = M(((ee)) + [ ?(h(s;))) 8, k, O)ds
["
= Wo.kk 71((s; es), 8, k)d0
(6)
where the initial magnitude is obtained using W.(.) f1..... K} Rd, which is trainable, and parameterized with a neural network 6. Hidden states time can be computed parallel multidimensional ODE by taking advantages of the decoupled structure of h(t;; es)
((t:eo))
>(h(t: (h)(t: ko: 0)
d hht) dt
d
: (,h(t;ee),,,kss05
dt
(((tee))
One major advantage of the decoupling is that the influence of events from h(t) can be selectively considered. For example, when computing (t) conditioned on Hey, where t > t;+1, ((t;)))4) should not be included in computation. The details will be discussed in the Sec. 3.2
3.2 GROUND INTENSITY FUNCTION
In our method, the ground intensity (t))) is defined as a mixture of decoupled influence functions u(t: <,,, which can be considered as a generalized form of the exciting functions found in the Hawkes process (Adamopoulos. 1976). Since each influence function is conditioned order to define (1 i.e., the ground intensity conditioned on the history, the influences from historical events must be aggregated. The conditional ground intensity function is defined as,
single event, in
Xg (t (tleo.. ... ...)) (o)t ,.(tte)))
where u(t;e;)) == 9a (h((tee)))
decoded by a neural network 9μ Rd + R. e, H.,., is an observed event, and 4 is positive function to satisfy the non-negativity constraints of (²)).
Notice that the hidden state h htt; ex) is decoded into the influence function μ(t; e,) before aggregated into X3 (t), otherwise, influence from a specific event would not observable.
When learning TPP, integration is pivotal. Not only does the computation of probability distribu- tion (1) (1) (1) exp( Si_, (s)dss requires integration, but also obtaining characteristic functions, survival rate, etc. requires additional integration which does not have a closed-form so- lution in general. Therefore, intensity-based methods often require additional steps for predictions. Methods such as Monte Carlo Inference (Mei Eisner 2017 Zhang et al. 2020 Zuo et al. 2020 Yang et al., 2022) often used for calculating f (t), while sampling methods such as the thinning algorithm Mei & Eisner (2017) are adopted for computing the expected value. these approaches, each integration has to be handled separately with additional sampling. The Neural ODEs, which
4

---

Published as a conference paper at ICLR 2024
extensively utilize to solve for the MTPP, is inherently computationally heavier compared to Monte Carlo Inference due to its sequential solving mechanism. To circumvent the extra computational burden, we leverage the characteristics of the differential equation. By simultaneuusly solving the following multi-dimensional differential equation with respect to h(t) along with numerical integra- tion, we eliminate the need for additional sampling:
htt
(t((t),, 8)
(
2 (g
(9)
Öt F()H))))
(C(()))) exp(An (HH,, (4((HHr))
t. (((H4.))
where Ag (t)H,,) = So Ag (t dt is called the compensator, and k is the marks corresponding to h()). Because the dynamics are learned through differential equations, values at time / can be used for computing others. e.g., (t), which is derivative of (t).). can be computed using h(t).
Therefore, important estimations with inte- gration, such as likelihood, survivor rate (t) == 1 (t),, and y, [t] can be predicted in a single run. This formulation can be applied to any number of integrations. Moreover, as briefly mentioned in Sec. 3.1 the influence functions cording to our interests illustrated in Fig. 2 Therefore, approximations of functions in equation Junder different He, can be ob- tained in parallel, distinguishing Dec-ODE from other intensity-based methods that re- quire separate runs.
n(t;e() nini
p(t;e()
(t
(t)
be selected
63
Figure Visualization equation Different combi- nations of put: e,) can be selected for calculating (²(1)
and f (t) conditioned on different He, in parallel.
3.3 CONDITIONAL PROBABILITY FOR MARKS
Similar to the formulation of the ground intensity in equation g the probability of marks (klt) can be expressed using the following form:
fkilt, Hr.,,) 22 (u(f(k)t, eo),
fkk(t, ((kkit)))))
(10)
where f(klt, e.) == g (e)(es) is an influence from e; on flk(t)) decoded by a neural network gy())
The function T combines fltlt, e;) over events while satisfying Σx +x(+)=1. 1.
The intuition behind such modeling is that the context of an event changing over time carries important information. For instance, getting driver's license would dramatically increase the prob- ability of causing a car accident since there was no chance without the license. However, as time goes on. the person gets better driving and the probability decreases. Using the proposed structure, the changes of implications through time can be inferred by f (k))t,
4 LINEAR DEC-ODE
The combining functions & and 4; can be chosen from a simple summation to a complex neural networks, such Transformer. Nonetheless, in the following section, present linear version of Dec-ODE as offers efficiency and interpretability via parallel modeling of the hidden states.
4.1 COMBINING INFLUENCES FROM PAST EVENTS
In order to demonstrate the competence of the proposed framework, the implementation of Φ and t is defined as linear and simple equations that combines influences of historical events for (t) and (k]t). The ground intensity X²(() and the conditioned probability of marks f* (k:]t) are defined combinations of decoupled dynamics:
(1 .) = @x(((t;ea), ,,,(() = Σ
softplus(p((;ee)))
(11)
eLE
5

---

Published as a conference paper at ICLR 2024
f(k]t, =0x - (f .. [(]], en)) = softmax
Σ
f(k]t, es)
(12)
ELEW
where softplus and softmax functions are used satisfy the constraints from Sec. 3.2 and Sec. 3.3 respectively. This modeling of (7))) with linear summation resembles with the Hawkes process. Yet, our method can model more flexible temporal dynamics, and better suited for modeling complex real-life scenarios.
4.2 TRAINING OBJECTIVE
The objective of our method to maximize the likelihood of the predicted Marked TPP (Daley &
Vere-Jones 2003). Taking a log on the likelihood in equation 2
NN((x)
No(tw) II
exp (- Xg(u)ddu)
In [(L,)) ln
II
(1.)
(kall()))
(yx(x)
In Xg - L.' AN
Ny(tN)
(13)
(g( +
Σ In (kslt))
In Lx
In La
where in is the last observed time point, and the log-likelihood for the ground intensity In Lx and the mark distribution In Ly are independently defined, and used to learn (t and f(ktt), respectively. Intuitively, the first term in LA is the probability of event happening at each t, and the second term the probability of event not happening everywhere else. Therefore, (z should be high at the time of event's occurrence and low everywhere else maximize Lx. Also, for the estimations, f(t is normalized to satisfy J ft)dtt 1.
4.3 TRAINING SCHEME
Many previous works using differential equation based modeling (Jia & Benson 2019 Chen al. 2 sequentially solve the entire time range, which require exhaustive training time. In our case, using the characteristics of 4 with linear summation, such limitation can be alleviated as follows.
First, because each p(t; e;) is independent from other events, we can propagate them simultancously as dimeiiidmmenionnll differential equation as
p(Toi ev)
(u(TT)) 0)
d
dr
(14)
((Tsie)))
where T = [TO, [T] = [to + t, + +t, +T]T.
Second, one of the benefits of formulating the ground intensity as linear equation that the com- pensator Ag (t) can also be calculated in the same manner as in equation I1
(2) = L( xz(s)ds / Σ
softplus(u(s:c:))dd
(15)
E.E..
= Σ softplus(u(s;e:))dd
(16)
C,EH,
where the region of the integration changes since there is no influence of e; before t, The equa- tion 16on]eeoonvees that the integration in equation c3 be computed by integrating softplus(u((; et)) individually first and sum later, instead of sequentially going through the entire sequence. Hence, we can find the number of time-points m, we can solve the entire trajectory within a fixed number of steps. The effectiveness of such modeling is discussed in Sec. 6.4
RELATED WORKS
Intensity modeling. When modeling TPP, tdhe intensity function is widely used due to its flexibility in capturing temporal distributions, and deep learning facilitated the parameterization of complex

---

Published as a conference paper at ICLR 2024
processes using neural networks. Initially. many focused relaxing constraints of traditional TPP
(Hakke]] 1971 Isham & Westcott, 1979. Daley & Vere-Jones 2003) such as Hawkes process. RNN
based neural TPP models (Mei & Eisner 2017 al. 2016) used RNNs to effectively encapsulate
sequences into a history Vector. branseorrer-bbssed methods (Zuo et al. 2020 Zhang et al., 2020)
were used encapsulate He while considering long term and short term dependency of events, and Attentive Neural Hawkes Process (ANHP) (Yang et al. (202)) utilized self-attention for modeling complex continuous temporal dynamics by using a attention network on an unobserved event. Pan et al. (2021) used the Gaussian process to relax the exponential decaying effect of Hawkes process. Omi et al. (2019) adopted a compensator function, which integration of an intensity function, to model TPP to alleviate computing issues in integration with no closed form.
Direct estimation of pdf. An alternative approach, which is to directly predicting the pdf of a TPP, has also been studied. [Shchur et al. (2020) proposed method to model the temporal distribution using a log-normal mixture. Having closed form pdf allowed direct inference of important char- acteristics, while the mixture model provided flexible modeling. There exists various other methods that model the pdf directly using popular deep learning models including GAN, normalizing flow, variational auto encoder and meta learning (Lin et al. 2022 Bae al. 2023).
Learning temporal dynamics using Neural ODEs. In (Chen et al. 2018), Neural ODE was pro- posed where the changes of hidden vector between layers are considered in continuous manner. Differential equation based methods were incorporated into various temporal dynamic modeling Jin et al.] 2022 (Sedat et al. 2022 Park et al. 2023), showing advantages in modeling sparse and irregular time series. Jia & Benson 2019) modeled the jump stochastic differential equation using Neural ODEs, and tested on TPP modeling. Chen et al. (2021) expanded the work to modeling spatio-temporal point process and predicted both temporal and spatial dynamics using Neural ODEs and continuous normalizing flow. Also, an algorithm for handling time varying sequence in batch wise computation was proposed.
6 EXPERIMENT
To evaluate Dec-ODE, we performed experiments under various settings to understand the model behavior. In Sec. 6.2 we compare linear Dec-ODE with the state-of-the-art methods using con- ventional benchmarks. The explainability of Dec-ODE is discussed in Sec. 6.3] and the parallel
computation scheme introduced in Sec. 4.3 is tested in Sec. 6.4
6.1 EXPERIMENT SETUP
Datasets. We used five popular real-world datasets from various domains: Reddit is sequences of social media posts, Stack Overflow (SO) is a sequences of rewards received from the question- answering platform, MIMIC-II is a sequences of diagnosis from Intensive Care Unit (ICU) clinical visits, MOOC is a sequence of user interactions with the online course, and Retweet is also a sequences of social media posts. See Appendix for detailed description of these datasets. To obtain the results in a comparable scale, all time points were scaled down with the standard deviation of time gaps, {4: 4-1 1 of the training set similar to Bae et al. 2023).
Metrics. We used three conventional metrics for the evaluation: 1) negative log-likelihood (NLL)) validates the feasibility of the predicted probability density which is calculated via equation [3] 2) root mean squared error (RMSE) measures the reliability of the predicted time points of events, 3) accuracy (ACC) measures the correctness of the mark distribution f (k]t)). For the prediction, the expected value Ey. [t] was used as the next arrival time of an event, and the mark with the highest probability time t, i.e., arg max (f (lc (t), was used as the predicted mark.
Baseline. We compared our methods with three state-of sate--ofttee of-the-art methods: Transformer Hawkes Process (THP) (Zuo et al., 2020), Intensity-Frre Learning (IFL.) (Shchur et al., 2020), and Attentive Neural Hawkes Process (ANHP) (Yang et al.) 2022 As THP and ANHP are both transformer-based methods, they effectively leverage the entire observed events H,, for predicting 6,+1- IFL directly predicts ft, k) = f (f)f" (kltt using a log-normal mixture predict a flexible distribution. Since it parameterizes the distribution f (t),, E [t] can be obtained a closed form. More details about the implementation are in S.3 B.3
7

---

Published as a conference paper at ICLR 2024
Table 1: Comparison with the state of the art methods on popular real-life datasets. Results with boldface and underline represent the best and the second-best results, respectively. Following Zuo
al. 2020 Yang et al. 2022) the mean and std are gained by bootstrapping 1000 times.
ACC
NLL
РУЕТРР
THP
IFR.
ANDE
Dec-ODE
RMIFP
THP
JFL
ANHEP
Dec Dec-OUE
RNIFP
THP
IFL
ANHP
Dec-ODE
MOOC
(0.015)
0.473
(0.010)
0.470
(0:012)
01.501
(0.010)
0.177
(11112)
11.467
(0.22)
20:35
(0:22)
21.11
(1.30)
37.34
(0.23)
31.53
(8:44)
62.11
(0.031)
-0.315
(0.947)
0.733
(R101)
-2,995
-265
20.15113
-2758
Reddr:
(0016)
11953
(0.136)
6:151
(0.017)
1.030
(0.010)
1.140
(1.177)
01.334
(1.10)
20.07
(0.00)
08177
(1.27)
18.51
(0.16)
63:45
10.113
022-12
(0.070)
3.559
(0.081)
2.335
(0.058)
21158
(9.068)
1,300
0.122
1.007
0.000
1.055
1.012
18883
0.965
51.72
00.68
56.55
50.72
60.17
-2.100
-2.000
-2,672
-3134
2.807
Rerveet
[00010]
(0:015)
(0.015)
(0.01:)
01.116)
10.33
(8.11)
(6: 10.15)
(0.1
(0.33)
(0 (0.025)
(0.015)
(0/03)
(0,019)
10-0391
MIMIC-I
(0.003)
1,851
(0.114)
(0.1211
(0.058)
0.333
(0.173)
18.00
(2.56)
(5.20)
(2:75)
64.30
(3.66)
(0.190)
1.367
(0.304)
5,607
(0.139)
(0.155)
1,222
(0.415)
Stack
1
1.057
1.510
1,052
10015
58.00 95
53.55
13.00
$6,50
55
2.158
2,315
2.518
1,873
2,005
Oventow
(0,11))
(0.0113
(0.01.3)
(0.011)
(1011)
10.
(6.18)
(6.35)
(0.18)
10.20
(0.022)
(0,022)
(0/021)
(0,017)
(0,310)
6.2 COMPARISONS OF RESULTS
We compare our linear Dec-ODE with state-of-the-art TPP methods on five real-life datasets from The summary of the overall results is in Table I In most cases, our approach showed comparable or better results all metrics. It confirms that Dec-ODE successfully models the com- plex dynamics of MTPP with independently propagated influence functions. Specifically. Dec-ODE showed its strength in prediction tasks, represented by high accuracy and low RMSE, while ANHP was marginally better in NLL. Moreover, methods that estimate (I) in order calculate E[(], Dec-ODE and IFL, showed tendency to make more accurate predictions of event time.
Sec. 6.1
Discussion. For fair comparison, we increased the number of sampling used in the thinning algorithm (Chen) 2016 Ogata 1981) implemented by (Ynn et al. 2022). In most cases,the number of samples is multiplied by 20. However, when applied to THP on Reddit dataset the thinning algorithm was not able to correctly sample from ft),,, since it was not able to correctly sample from an intensity function with low magnitude with high fluctuation. Details are Appendix (B.4
6.3
EXPLAINABILITY OF DEC-ODE
In the case of MTPP and TPP, events temporally and complexly influence (t) and (*) (Hawkes) 1971 Isham & Westcott 1979). Therefore, when considering the explainability of neural network modeling an MTPP, it is important to understand how each event affects the prediction, and how they change through time. Dec-ODE naturally yields such information. For example, the temporal dynamics of (t) and (k(t) can be directly observed with the naked eyes in Fig.
Our detailed investigation on the Retweet dataset fur- ther validates the explainability Dec-ODE. For ex- ample, some meaningful patterns from the Retweet data visualized in Fig. The Retweet data is composed of three classes: post from person with small num- ber of followers (~ 50% of the population), a post from person with a medium number of followers(~ 45% of the population), and post from person with a large number of followers (~ 5% of the population), denoted as 0, 1, and 2, respectively. In Fig. (4(a) which visu- alizes μ(t; eq) conditioned on different e;, shows that influence on (f). Such pattern is expected as a post on social media shows immediate reactions, rather than a time- delayed effect. Also, a post from an user with large followers exhibits slower decay. Which is natural since a post from an user with a large followers gets more ex- posure and results in longer lasting influence people.
event type
each event exhibits lemporally-decayinn
64
20
Visualization of propagated f" (klt,e) e;) in StackOverflow experiment. The each axis represent time, event type, and magnitude. The change of influence through time can be easily observed.
Figure
Moreover, the averaged proportion influences on different mark types are illustrated in Fig. 4(b) First, even though the type 2 takes only 5% of the entire population, it shows great influence in the overall occurrence of events. This may be true as person with many followers may have a greater influence broader audience. Second, for each event, events with the same types have the strongest influence on each other. Considering the exponentially decaying patterns of put) and the large initial magnitude of influence of type shown in Fig. 4(a) infer that events with type
8

---

Published as a conference paper at ICLR 2024
7ype 0
Type
Type
20.47
Type
0.8
Normalizon iedd influence
21.08
Type
0.6
10.99
7.08
0.2
3.54
Types
(a) Influence function (t)) conditioned on different event types plotted the same time range.
(b) Averaged proportion of influence from different marks specific marks.
2000
13000
15000
20020
230000
30000
35000
(c) Visualization of events, i.c., marks across time, affecting each other. Bluc(O); users with small follow- Red(1): users with medium followers, Green(2): users with large followers.
Figure 4: Visualization of patterns found in Retweet dataset. (a) p(t; et) with different marks, (b) Influence of marks each other, (c) Actual marks with respect time.
O would show a high clustering effect and actual visualization of the data in Fig. 4(c) validates our reasoning.
6.4 PARALLEL COMPUTING SCHEME
While Neural ODEs effective when learning the continuous dynamics of system, the hidden state propagation requires large computation to solve IVP step by step. In Sec. [43 we proposed propagate the hidden state through time in parallel disentangling individual events, and validate the optimization scheme by comparing the average time for each iteration with traditional IVP. For the sequential propagation, differential equation solved from to through t, step by step. Table summariizs the result, where the computation time significantly decreased in all cases. Parallel computation for Dec-ODE was at least twice (for MIMIC-II) and as much as five times faster (for Reddit) than the traditional sequential computing scheme.
Table 2: Training efficiency comparison in different datasets with the average time taken for an iteration (sec/iter) the metric. Ratio shows that the iteration time at least reduces in half.
StackOverflw
MIMIC-II
Retweet
Reddit Sequential 78.7
parallel
Sequential
Ratio
parallel
Sequential
Ratio
parallel
Sequential
Ratio
parallel
Ratio
15.0
57.7
0.26
2.9
0.47
16.8
67.6
0.25
15.5
CONCLUSION
In this work, we presented novel framework for modeling MTPP, i.e., Dec-ODE, that decouples in- fluence from each event for flexibility and computational efficiency. The Dec-ODE leverages neural ODE to model continuously changing behavior influence from individual events, which govern the overall behavior of MTPP, and we proposed a training scheme for linear Dec-ODE, which let the influences be efficiently computed in parallel. The proposed method was validated on five real-world benchmarks yielding comparable or superior results. Moreover, it provides explainabil- ity to the modeling, which suggests significant potential for applications such as out-of distribbbtnio detection and survival analysis.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by UTT-2019-0-01906 (AI Graduate Program at POSTECH, 10%) funded by the Korea Government (MSIT).
(60%), LITP-2022-0-00290 (30%). and
REFERENCES
Adamopoulos. Cluster models for earthquakes: Regional comparisons. Mathematical Geology 8, 1976.
Emmanuel Bacry, Adrianna luga, Matthieu Lasnier, and Charles-Albert Lehalle. Market impacts and the life cycle of investors orders. Market Microstructure and Liquidity, 1(02), 2015,
Emmanuel Bacry, Martin Bompaire, Stéphane Gaiffas, and Soren Poulsen. Tick: python library for statistical learning, with particular emphasis on iimedeepeneett modelling, 2018.
Wonho Bae, Mohamed Osama Ahmed, Frederick Tung, and Gabriel L. Oliveira. Meta temporal point processes. In International Conference on Learning Representations, 2023.
Renqin Cai, Xueying Bai, Zhenrui Wang, Yuling Shi, Parikshit Sondhi, and Hongning Wang. Mod- cling sequential online interactive behaviors with temporal point process. In International Con- ference on Information and Knowledge Management, 2018.
Ricky T. Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. In Neural Information Processing Systems, 2018.
Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference on Learning Representations, 2021.
Yuanda Chen. Thinning algorithms for simulating point processes, 2016.
D.J. Daley and D. Vere-Jones. An Introduction the Theory Point Processes, volume 1. Springer, 2003.
A. De, U. Upadhyay, and M. GodezzRodriguez. Temporal point processes, 2019.
Nan Du, Mehrdad Farajtabar, Amr Ahmed, Alexander J. Smola, and Le Song. Dirichlet-hawkes processes with applications clustering continuous-time document streams. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigued and Le Song. Recurrent marked temporal point processes: Embedding event history vector. In Special Inter- est Group on Knowledge Discovery and Data Mining, 2016.
Alan G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1), 1971.
Alan G. Hawkes. Hawkes processes and their applications to finance: review. Quantitative Fi- nance, 18(2), 2018.
Valerie Isham and Mark Westcott. self-correcting point process. Stochastic Processes and their Applications, 8(3), 1979.
Junteng Jia and Austin R. Benson. Neural jump stochastic differential equations. In Neural Infor- mation Processing Systems, 2019.
Ming Jin, Yuan-Fang Li, and Shirui Pan. Neural temporal walks: Motif-aware representation learn- ing on continuous-time dynamic graphs. In Neural Information Processing Systems, volume 35, 2022.
Jure Leskovec and Andrej Krevl. SNAP Datasets: Stanford large network dataset collection, 2014.
Haitao Lin, Lirong Wu, Guojiang Zhao, Pai Liu, and Stan Z. Li. Exploring generative neural tem- poral point process. In Transactions Machine Learning Research, 2022.
10

---

Published as a conference paper at ICLR 2024
Hongyuan Mei and Jason Eisner. The neural Hawkes Process: A neurally self-modulating multi- variate point process. In Neural Information Processing Systems, 2017.
Rui Meng, Fan Yang, and Won Hwa Kim. Dynamic covariance estimation via predictive wishart process with application on brain connectivity estimation. Computational Statistics & Data Analysis, 185:107763, 2023.
Y. Ogata. On lewis' simulation method for point processes. IEEE Transactions on Information Theory, 27(1), 1981.
Takahiro Omi, Naonori Ueda, and Kazuyuki Athara. Fully neural network based model for general temporal point processes. In Neural Information Processing Systems, 2019.
Zhimeng Pan, Zheng Wang, Jeff M. Phillips, and Shandian Zhe. Self-adaptable point process with nonparametric time decays. In Neural Information Processing Systems, 2021.
Sungwoo Park, Byoungwoo Park, Moontae Lee, and Changhee Lee. Neural stochastic differential games for seme-series analysis. In International Conference on Machine Learning, 2023.
Jakob Gulddahl Rasmussen. Lecture notes: Temporal point processes and the conditional intensity function, 2018
Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. In In- ternational Conference on Machine Learning, 2022.
Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. In International Conference on Learning Representations, 2020.
Chenghao Yang, Hongyuan Mei, and Jason Eisner. Transformer embeddings of inregularly spaced events and their participants. In International Conference on Learning Representations, 2022.
Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes Process. In International Conference Machine Learning, 2020.
Qingyuan Zhao, Murat Erdogdu, Hera Y. He, Anand Rajaraman, and Jure Leskovec. Seismie: A self-exciting point process model for predicting tweet popularity. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes Process. In International Conference on Machine Learning, 2020.

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTS
Additional experiments are performed to explore the extensibility of our work.
A.1 LINEAR vs. NON-LINEAR INFLUENCE FUNCTION
In the [able[[ the simple Dec-ODE with linear 2 has been tested. However, 4 and 4x can be any functions, even neural networks, that satisfy the conditions mentioned 3.2 and 3.3 In fact, with linear 4 every influence function must satisfy ju(t) >= 0 to keep the intensity positive. Therefore, only an excitatory effect, where an event can only increase the intensity, can be expressed.
In this section, to navigate the extensibility of our framework, a less constrained variant is introduced and tested, denoted as non-linear $ x- The non-linear xx (1) is defined as follows:
Φ (x(ttee)) ... put; e() = softplus
Eu(t:ee)
(17)
Through this modeling approach, an inhibitory effect, where event decreases the overall intensity, can occur.
To see how this relaxation affects in modeling TPP, both are tested in simplified setting, where only the first 40 events of each sequence are used. The results of the experiment are summarized in Jusing NLL as the metric. The overall result regarding the ground intensity improved in most situations showing that the intensity function with higher fidelity can be predicted using more flexible x-
Table 3 using
However, non-linear Dec-ODE shows some limitations in predicting a time-delaying effect, where an event does not have large influence until certain time passes rather than having an instant influence. This happens because the inhibitory effect exceeds the excitatory effect, and the intensity function becomes In such cases, the inhibitory effect rather impairs the model's ability. In con- clusion, this experiment suggests that with a more flexible 2x, prediction can be made with higher fidelity in most cases. However, adding constraints has to be further discussed robustly model different patterns in TPP.
Table 3:
Experiment non-linear 4x The softplus is applied after the summation in order to express inhibitory effects. Details are in A.I
SerckOverllow
MIMIC-II
Retweet
Reddit
Linear Non-linear
Linear
Non-linear
Linear Non-linear
Linear
Non-linear
0.9948
0,8987
0.2451
0.2183
5.0872
5.1649
0.5163
18.2571
A.2 IMPUTATION
In this section, investigate the effect of indepen- dently modeling hidden state dynamics by comparing with methods using contextual information. Events from the StackOverflow dataset are randomly dropped see how the behavior changes as the number of events decreases. For the fair comparison, we ran- domly selected 90%, 80%, 70%, and 60% indexes from the test dataset, and saved the data with the se- lected index as new test sets. The results using the new test sets are visualized in figure 5 The graph illustrates that the decrease in performance of Dec-ODEs shows similar tendency when compared to others.
32
3.0
2.4
22
Drop rati
Figure 5: Imputation experiment done using Stackoverflow dataset, where from 10% to 40% of data are randomly dropped.
The result suggests that other methods cannot retrieve the loss of information. This statement be made since Dec-ODE, which does not consider inter-event relationships, shows similar tendency.
12

---

Published as a conference paper at ICLR 2024
A.3 TRAIN TIME COMPARISON
Table 4: Training time (sec epoch) compared between THP, ANHP, and Dec-ODE.
MOOC
Reddit
Retweet
Stackoverflow
MIMIC-II
THP 12.94
66.95
34.05
13.36
2.27
ANHP Dec-ODE
835.36
541.20
630.64
129.51
3.69
117.35
242.75
484.08
123.28
8.12
Table Required memory (MB) compared between baseline methods with batch size 4.
MOOC
Reddit
Retweet
Stackoverflow
MIMIC-II
RMTPP THP
1110 + 1114
1706 + 1142
1294 + 1112
1306 + 1114
1114 + 1106
3484
15304
1678
3808
1130
ANHP 31310
< 48 GB
11790
39260
1244
Dec-ODE 1422
1472
1314
1422
1470
In order to compare the time required for training, the time required for training one epoch (sec / epoch) is measured. The experiment is conducted on single NVIDIA RTX A6000 GPU (48GB) with the largest batch size possible for the GPU. The result can be found in the table THP required the least amount of time, while methods predicting intensity at each time point showed relatively slower training rate. most cases, Dec-ODE shows shorter training time per epoch.
B IMPLEMENTATION DETAILS
B.1 IVP SOLVING WITH VARYING TIME INTERVALS
Initial Value Problem (IVP) solving with varying time intervals is done following the method intro- duced in Chen et al. (2021). In short, the integration in the region [tatart. is solved within [o. 1] and scaled back to the original region. Using this method, every time interval with different lengths can be computed within the same number steps. We apply this for both batch computations with different lengths and parallel computation of h(t; e,) in Sec. 4.3
B.2 BATCH COMPUTATION
Our method propagates h(t) further than fN. which is the last observed point. Therefore, in order to cover the full range of f (t), two different masking operations required. The first one is the one commonly used when dealing with sequences with different lengths, and we call the sequence mask. With sequences with different lengths, unobserved events should be ignored for both training and inference, and the sequence mask is used to ignore unobserved time points. The second mask is the propagation mask, where the unobserved time points after iN masked. This is used for solving ODEs to propagate h(() until the decoded (t)) converges.
B.3 BASELINE
For THP and ANHP, we used the public GitHub repository https://gihub..om//ang.aaa 123/anhp andtt (Yngg et al.] (2022) with MIT License). The code for the THP has been modified by Yang ctt al. (2022), where time and event prediction is now made by using A(t) with a thinning algorithm instead of the prediction module that is parameterized by neural networks.
Moreover, as mentioned the public GitHub repository THP, there is an error regarding the NL.I computation. In both published versions, NLL is computed conditioned on the observed history information and a current event. i.c., L(e²) = f(e/ee, .., .4). Therefore, the calculation of the intensity has been corrected to [(e)) f(e]eo, ... (4-1), where the modification is made based on the code used for the integration.
For IFL, used the public GitHub repository (2020), no license specified). Some modifications have been made since the original implemen- tation does not support time point prediction. The modification made based the author's
www.sschur/ittmtpppppppp (Shchur al.
13

---

Published as a conference paper at ICLR 2024
comment made in the "issue" page of the repository. However, the computation of E[t] not sta- ble due to the high variance of components in the mixture model. In other words, when a distribution in mixture model has high variance, the exponential term in the calculation causes an overflow, which leads to an overflow of the entire calculation. Therefore, we have tightened the parameters for the gradient clipping in Shchur et al. 22220), done in Bae et al. 2023)
B.4 THINNING ALGORITHM
For the experiment done in [ we tested different parameters for the thinning algorithm (Chen 2016 Ogata 1981p, implemented by Yang et al. (2022), for fair comparison. THP and ANHP sample the next time points using the thinning algorithm then take their average to compute E(4.
The thinning algorithm resembles the rejection sampling, in which the proposal is accepted with the probability \(t)/ Aup where Aup 2 X(t) for all E (f;-1.0) (Mei & Eisner 2017)) Therefore, reliable Aup is required to sample from the correct distribution. dup is too high, all the samples would be rejected, whereas accept incorrect samples if Xup is too low.
In the implementation, Aup is calculated by c * maz(Assu).... is the number of samples used for the calculation, and Sm is a uniformly sampled time point. To find correct Aup. for most of the reported results in Table we increased the number of m by x10. When overall A(l) is too low, the scale goes up to 1000 in the most extreme case.
(sm)). where is a constant, m
B.5 TRAINING DETAILS
When tuning the simple Dec-ODE, three different hyperparameters are considered for the model structure with the addition of an Initial Value Problem (IVP) solving method. Three hyperparameters are the dimension of hidden state D, the dimension of linear layers of neural networks N,, and the number of liner layers L. We haven't 't fully searched for the optimal parameters for each dataset. For testing, we rather applied similar parameters to all datasets.
Throughout the experiment, D was chosen from {32, 64). However, since the dynamics of h(t) is highly dependent on the information in the hidden state, we expect a higher dimension would enable the neural network learn more difficult dynamics.
The width of linear layers is chosen between {128, 256), and the number of layers are tested between {3, 4)5. In most cases, the number of layers did not show much improvement in the results.
The three hyperparameters above that showed the best result chosen. However, in most cases, D = 64, N 256, and I. = 3 were used.
In the case of IVP solver, two different methods used training and testing. For training. Euler's method was used in most cases for efficiency. The purpose of training is to learn the dynamics of hidden state htt), and we believe Euler's method sufficiently satisfies such purpose, and we haven't t thoroughly looked into the benefits of using different solvers.
During testing (L). to make a more precise approximation, we utilize Runge- Kutta method with fixed step size, generally referred to as RK4. The computation of RK4 method is as follows:
yn+1 Un + (ki + 2kz + 2k: + ka)
(18)
tx+ = t, + h
(19)
where,
k1 [(/n yn)) h
(20)
2 U/. + h k1
l2 f(nn
)
(21)
2
Ka f(n
h
5 Un + h k2
K22) )
(22)
Ra f(n h, yn + hkr))
(23)
14

---

Published as a conference paper at ICLR 2024
The RK4 method can precisely model the trajectory compared to Euler's method. Therefore, there is less eπrоr the estimated f (t). Other methods such as dopri5, RK4 with step size control, and other variants of IVP solvers can be applied for more accurate results.
To solve the IVP, we fixed the number of steps required for solving from f, to t- Similar to the IVP solver, used simpler setting for the training and more precise setting for the testing. During training, in most cases, we used 16 steps in between every event. The increase number is expected to improve the result, yet have not thoroughly looked into it since it shows comparable results with state-of-the-at methods. During testing, we increased the number to 64.
When testing f(]tt), the precise approximation of fk]tt, eq) did not show a visible effect on the result. Therefore, for computational efficiency, Euler's method was used with the same number of step sizes used for training.
Datasets
of Seq.
# of Events
Max Seq. Length
# of Marks
MOOC 7,047
389,407
200
97
Reddit
10,000
532,026
100
984
Retweet
24,000
21173,533
264
3
StackOverflow
6,633
480,414
736
22
MIMIC-II
650
2419
33
75
Table 6: Statistics of benchmark datasets used for comparisons.
C DATASET DESCRIPTION
Table shows the statistics of each benchmark dataset used for testing.
C.I BENCIIARK DATASET
MIMIC-II and Retweet were from the public GitHub repository (Zuo et al. 2020
with no license
specified). Others were also from the public GihtHub repository (Lin al. (022 with no license specified).
C.2 DATA PREPROCESS
When dealing with Mooc, Reddit, Retweet, and StackOverflow, two modifications to the dataset have been made. First, when two events have the same time point, one of the events is removed. We chose the latter one in the dataset. This is because when 4; - 4--11 = 0, IFL was not able to properly estimate (t). Also, in many cases, TPP assumes or more events happening at the same time as improbable. Second, as mentioned in Sec. 6. data were scaled by the standard deviation of (14 - f- When the scale of 1, 4--1 too large, overflow happens during the calculations. Therefore, we tried to make the scale similar to the results from Bae et al. (2023). but the standard deviation was chosen since the scale was not specified in the published work.
15

---

Published as a conference paper at ICLR 2024
D VISUALIZATIONS
D.1 VECTOR FIELD
By employing Neural ODEs (Chen et al. 2018) we can visualize the overall dynamics of functions that we estimate. Fig. 0 visualizes the vector fields. Fig. ((a) visualizes the first dimension of the hidden state using the StackOverflow dataset. The arrow represents the dynamics where the value of the tail is given as the input. There are some visible patterns in the dynamic forms. However, a certain part of the field shows very different behavior even when they receive the same input. shows that the hidden state dynamics learned from the data is very complex. On the other hand, ptt e,) visualized in Fig. 6(b) shows a clear tendency in the region.
14.
52.
60
20
100
(b) Visualization of the change of p(t:e) e.) in StackOverflow dataset. Figure 6: Visualization of vector fields. is vector field of the hidden state and b) is the vector field of μ(t; e.). The length of the arrow represents the magnitude.
(a) Visualization of the hidden state dynamics with StackOverflow datasct.
16

---

Published as a conference paper at ICLR 2024
D.2 VISUALIZATION OF CONTINUOUS DYNAMICS
Typo
Type 15
40-
event type
40
60
16.27
14:28
14.25
14:30
14:33
14.32
14:33
14.34
time
C4
80
(a) Visualization of p(t;ee)) trained using MOOC dataset.
(b) f(t e,) changing through time, where es in MOOC datasct.
13.5
15.0
15.5
200
event type
400
0.0
60p
time
800
1000
(c) p((;es) trained using Reddit dataset, each alt; es) is plotted with different colors for visibility. We can infer from the figure that there is delayed effect Reddit dataset.
(d) f(lees) trained on Reddit dataset, changing through time, where 148.
event type
Visualization of e(t;;:) trained using SO dataset. Each pit; es) demonstrates different dynamics.
(f) Visualization of f(::ex) changing through time MIMIC-II dataset, where 10.
Figure 7: Visualization of dynamics of and in benchmark dataset.
17

---

Published as a conference paper at ICLR 2024
D.3 SIMULATION STUDY
20IP
DEC oOe
GRoUTS
ground
(a)
(b)
(c)
ground bruth
Doc ODE
THP
20
75
5.0
1.0
08
0.3
1.0
3.1
1.5
(d)
(c)
Figure 8: Visualization from simulation study. (a), (b), (c), and (d) compare the true intensity function of a Hawkes process and the reconstructed results from THP, ANHP, and Dec-ODEs. Each (t) that composes the result in (d) is illustrated in (e).
RMSE
ACC
NLL
THP ANHP Dec-ODE
0.7734
27.48
3.0121
0.6528
30.31
0.2807
0.6568
30.35
0.2739
Table 7: Estimation accuracy on the simulated Hawkes process.
We have performed simulation study the Hawkes process following similar procedure from the Neural Hawkes Process (NHP) (Mei & Hisner) 2017). In order to obtain the true intensity function we randomly sampled the parameters for the kernel function of the multivariate Hawkes process, and we simulated synthetic data using the tick library (Bacry et al., 2018). The range of the sampling was changed from the NHP since the scale in the library was difterent from the paper. In Figure 8(a) 8(d) THP struggles to simulate the flexible dynamics of the intensity function Xg- whereas ANHP Dec-ODE show similar dynamics to the ground truth intensity. The table 7 also demonstrates that Dec-ODE shows better results than THP and ANHP in all metrics. The result also aligns with the results in Table [ which suggests that Dec-ODE can simulate reliable MTPPs compared other state-of-the-un art methods. Figure 8(e) illustrates μ(t) that compose X (t) from Figure ((d) which shows that the ground intensity (²) can be reconstructed from the individually constructed trajectory, i.e. MTPP can be modeled from decoupled information.
18


Reviews:
[{"review_id": "odbbSEXBP0", "reviewer": "Reviewer_A48s", "summary": "This paper proposes a new modeling framework for marked point processes, where the mark distribution and the conditional intensity function are treated as two separate modeling objectives. These two objectives are assumed to depend on a common latent process that is evolved through a neural ODE model. The proposed algorithm is more computationally efficient than existing methods and provides more interpretable model estimates. The effectiveness of the algorithm is demonstrated by comparisons of five benchmark data sets to state-of-art methods.", "questions": "The paper is quite well written, so I don't have many questions. My major concern is that no simulation studies are conducted to demonstrate the estimation accuracy of the proposed algorithm when the data-generating process is assumed in the paper. Some comparisons with existing methods are desirable as well. That way, one can have better ideas of the advantages and limitations of the proposed method, in terms of estimation accuracy, predictive accuracy, and computing times.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and the presentation is clear. The proposed idea is sound and may have a large potential impact.", "weaknesses": "No simulation studies are conducted to demonstrate the estimation accuracy of the proposed algorithm when the data-generating process is assumed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SXF2OkUWN9", "reviewer": "Reviewer_Ux3F", "summary": "This paper proposes a new neural ODE-based framework for modeling marked Hawkes/mutually-exciting point processes. The framework models the intensity function (of time) as a summation of event-triggered trajectories, each of which solves a neural ODE with an initial state that depends on the time of each event and its mark. The additivity of event-triggered trajectories on the intensity function allows us for the efficient parallel computation for model training and the explainable analysis of the dependence between events, which is the advantage over previous neural network-based Hawkes processes.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of using neural ODEs to model the effect of an event on the subsequent events (essentially, triggering kernel) in temporal point processes is novel and promising.\n- The parallel computing scheme of fitting in the proposed model is beneficial for practical purposes.\n- The validity of the proposed model was evaluated on various real-world data.", "weaknesses": "- The authors insist that an advantage of the proposed method (Dec-ODE) over conventional methods (e.g., Yang et al., 2020) is that Dec-ODE can compute multiple integrations simultaneously [Section 3.2]. But Dec-ODE demands to solve ODEs to compute integrations, which is generally more time-consuming than Monte Carlo integrations needed in conventional methods. In Experiment, there are no comparative analysis about computation time (sec/iter), and the authors’ insistence about computation efficiency remains to be verified. \n- The authors insist that an advantage of Dec-ODE over conventional methods (e.g., Yang et al., 2020) is the explainability due to the decoupled structure. But the decoupled structure has been adopted intensively in the literature, which were not in the benchmark models in Experiment. To verify the insistence, the authors need to compare Dec-ODE with references with the decoupled structure, which would make the pros/cons of the proposed model clearer.\n- RMTPP is a standard benchmark model in the literature, and should be included in comparative experiments. Otherwise, the authors need to mention the reason for not including it. \n- The explanation of the experiment setup seems to be inadequate for reproducibility. The details of neural networks (e.g., activation functions) and the machine spec used in the experiments should be shown.\n- The detailed equation about how to compute the derivative of the likelihood regarding model parameter is not shown.\n- Discussions about the limitation of the proposed model are not in the paper.\n- To the best of my knowledge, there are sentences that seem technically inaccurate as follows, which raises a question about the paper’s quality:\n\t- [3rd paragraph in Section 1] the authors introduce neural Hawkes process (Omi et al., 2019) as a marked temporal point process (MTPP), but the Omi’s paper did not consider the marks of each event. Also, there are two Omi’s papers (2019a and 2019b), but they look identical.\n\t- [2nd paragraph  in Section 2] the authors explains that the modeling of intensity function is preferred over the pdf $f^*(t)$ due to the complex dynamics of $f^*(t)$, but the complex dynamics depending on the past events is not limited to $f^*(t)$.\n\t- [2nd paragraph in Section 2] $\\\\int_0^{\\\\infty} f^*(s) ds = 0$ is correctly $\\\\int_0^{\\\\infty} f^*(s) ds = 1$.\n\t- [3rd paragraph in Section 2] the sentence “$\\\\mathcal{N}_g = \\\\{ t_i \\\\}$ is a temporal point process” seems to fail to explain the definition of $\\\\mathcal{N}_g (t_N)$ in Eq. (2). Do the authors mean that $\\mathcal{N}_g (t)$ is a counting process?\n\t- [Eq. (9)] the definition of $f_{\\theta}(h,t)$ is not found.\n\t- [References] The authors of Transformer Hawkes process are not Yang et al., but Zuo et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYCyBTg3ee", "reviewer": "Reviewer_yduQ", "summary": "This paper proposes a new Dec-ODE model to learn marked temporal point processes (MTPPs). The Dec-ODE model takes into account the individual events' influence on the underlying dynamics of the whole process and models it as a neural-ODE. Since the model is decoupled into ground intensity and conditional mark distribution, the new approach can compute the integrals for different parts in parallel, making the training faster. The experiment shows the effectiveness of the new model.", "questions": "1. I could not find the reference Yang el al., 2020 for the THP baseline. Is the correct one be [1]?\n\n[1] Zuo, S., Jiang, H., Li, Z., Zhao, T., & Zha, H. (2020, November). Transformer hawkes process. In International conference on machine learning (pp. 11692-11702). PMLR.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is novel. The paper takes into account the events' influence on the underlying dynamics of the later process, which is a significant factor but under-explored before.\n2. The decoupled model facilitates parallel computing of the costly integrals that occur in the neural MTPP formulations. \n3. The influence model gives Dec-ODE some extent of explainability as a model of MTPP.", "weaknesses": "1. There is a lack of comparison in computation time across different models. The parallel computing scheme is shown to reduce the computation time of neural ODE, but it is unclear whether Dec-ODE runs faster than baseline models like THP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)