Graph-informed Neural Point Process With Monotonic Nets
Authors: Zhimeng Pan
Zheng Wang
Shandian Zhe
Venue: Submitted to ICLR 2023 (2023)
Keywords: Point Process
Sequential Model
Graph Neural Network

Abstract: Multi-class event data is ubiquitous in real-world applications. The recent neural temporal point processes have used monotonic nets to model the cumulative conditional intensity to avoid an intractable integration in the likelihood. While successful, they are restricted to single-type events and easily sink into poor learning results. To address these limitations and exploit valuable structural information within event participants, we develop a Graph-Informed Neural Point Process (GINPP) that can freely handle multiple event types, greatly improve learning efficiency with the monotonic net, and effectively integrate the graph information to facilitate training. First, we find the bottleneck of the previous model arises from the standard soft-plus transformation over the output of the monotonic net, which greatly enlarges the prediction variations of the monotonic net and increases the training challenge. We propose a shift-scale version that can significantly reduce the variation and promote learning efficiency. Second, we use a conditional mark distribution to model multiple event types, without the need for explicitly estimating the intensity for each type. The latter can be much more challenging. Third, we use random walks to collect the neighborhood of each event participant and use an attention mechanism to update the hidden state of each participant according to the observed events of both the participant itself and its neighborhood. In this way, we can effectively leverage the graph knowledge, and scale up to large graphs. We have shown the advantage of our approach in both ablation studies and real-world applications.

Under review as a conference paper at ICLR 2023
GRAPH-INFORMED NEURAL POINT PROCESSES WITH MONOTONIC NETS
Anonymous authors Paper under double-bling review
ABSTRACT
Multi-class event data is ubiquitous in real-world applications. The recent neural temporal point processes (Omi et al. 2019) have used monotonic nets to model the cumulative conditional intensity to avoid an intractable integration in the likelihood. While successful, they are restricted to single-type events and can easily sink to poor learning results. To address these limitations and to exploit valuable structural information within event participants, we develop a Graph- Traph-Informed Neural Point Process (GINPP) that can freely handle multiple event types, greatly improve learning efficiency, and effectively integrate the graph information to facilitate training. First, find the bottleneck of the previous model arises from the standard softplus transformation over the output of the monotonic net, which enlarges the prediction variations of the monotonic net and increases the training challenge. We propose shift-scale variant that can significantly reduce the variation and promote the learning efficiency. Second, we use a conditional mark distribution model multiple event types, without the need for explicitly estimating the intensity for each type. The latter can be much more challenging. Third, we use random walk to collect the neighborhood of each event participant, and use an attention mechanism to update the hidden state each participant according to the observed events of both the participant itself and its neighborhood. In this way, can effectively leverage the graph knowledge, and scale up to large graphs. We have shown the advantage of our approach in both ablation studies and real-worke applications.
Introduction
Real-world applications often involve multi-class events. For example, 911 calls seek for a variety of helps, traffic records include different types of accidents, and among social network users are various types of interactions (tweeting, following, poking, etc.), Neural temporal point processes (e.g., (Du et al. 2016; Mei and Eisner 2017] Zhang et al. 2020a Zuo et al.) 2020)) are a family of powerful methods for event modeling and prediction, which neural networks (NN) model the intensity of events and can flexibly estimate the complex dependencies among the observed events. However, due the use of NNs. the cumulative (i.e., integral of) conditional intensity in the point process likelihood is often analytically intractable, and demand a complex, expensive approximation. To bypass this issue, the recent work Omi et al. (2019) uses a monotonic net (Sill 1997 Chilinski and Silva. 2020) to model the monotonically increasing cumulative intensity to avoid the integration, and the intensity is obtained by simply taking the derivative. To ensure the positiveness, softplus transformation is applied to the output of the monotonic net. Despite the elegance and success, this method only supports single-type events. More important, it often suffers from inefficient learning and easily falls into poor performance.
In this paper, we propose GINPP, a graph-informed neural point process model to overcome these problems, and to further utilize the valuable structural knowledge within the event participants, which is often available in practice. The major contributions of our work are listed as follows.
First, we investigate the learning challenge of (Omi et al. 2019), and find the bottleneck arises from the softplus transformation over the monotonic net prediction to ensure posi- tiveness. To obtain output slightly above zero, the standard softplus demands the input, i.e., the monotonic net prediction, must be negative and have much greater scales. Hence, a small output range can cause a much wider input (monotonic net prediction) range, which

---

Under review as a conference paper at ICLR 2023
are biased toward the negative domain. The large variation of the prediction scale makes the estimation of the monotonic net much more difficult and inefficient.
Second, we propose shift-scale variant of the softplus function, where the scale controls the shape and the shift controls the position. By setting these two hyperparameters properly, the required input range can be greatly shrunk, and get close to the output range. Accordingly, the variations of the prediction scales is significantly reduced, and the learning of the monotonic net much easier and more efficient.
Third, we construct marked point process for multi-class events. By introducing con- ditional mark distribution, we can freely handle different event types and only need a single-output monotonic net, which models the unified cumulative conditional intensity. This is more efficient and convenient than naive extension that separately estimates the cumulative intensity for each particular event type.
Fourth, to incorporate the graph structure in training, we use random walk to collect the neighborhood for each participant. We use attention mechanism to update the hidden state of each participant, according to the observed events of not only the participant itself, but also its neighborhood. In this way, the estimation of hidden state can be improved with enriched observations, and the event dependencies can be more comprehensively captured. The random walk further enables to scale to large graphs. Accordingly, we develop an efficient, scalable stochastic mini-batch learning algorithm.
For evaluation, we first examined GINPP in ablation studies. We tested the performance of the monotonic net with our shift-scale softplus transformation in learning two benchmark functions: one monotonic and the other is not. Our method converges fast, accurately learns the first function, and finds a close monotonic approximation to the second function. By contrast, with the standard softplus, the learning saturated early at large loss values and the estimation is much worse. Then, we tested on synthetic bi-type event dataset. GINPP accurately recovered the intensity for each event type via the learned overall intensity and the mark distribution. Next, we evaluated GINPP in six real-world benchmark datasets. We examined the accuracy in predicting the time and type of future events. In both tasks, GINPP consistently outperforms all the competing methods. Even without incorporating the graphs, GINPP still achieves better accuracy. When the graph structure is available, GINPP improves the accuracy further.
Background
Temporal Point Process (TPP) is a general mathematical framework for event modeling (Daley and Vere-Jones 2007). A TPP specified by the conditional intensity (or rate) of the events. Suppose we have K types of events, and denote by d (t) the conditional intensity for event type k. Given a sequence of observed events and their types,
((1,81); ..... (LN, sN)], where in is the timestamp and is the type of each event (1 S 8n S K. I lnt1). The likelihood of the TPP is given by
exp (-6, (t(ddtt)
III Aon (tn),
p(T)
(I)
where T the entire span of the observed events. One popular TPP is the homogeneous Poisson process, which assumes each conditional intensity (x(t) is time-invariant constant X and has nothing to do with previous events {(tn, 5n))t, < t]. While simple and convenient, Poisson processes ignore the complex relationships among the events. The Hawkes process (Hawkes) 1971) is more expressive in that models the excitation effect among the events,
(t) X2 Ei Pen Pk. tx)
where Xe 2 0 is the background rate, As., ()) > is the triggering kernel, and quantifies how much contribution the past event at t,, of type Sn, makes to trigger event of type k to occur at t. The most commonly used triggering kernel is an exponential kernel, which assumes an exponential decay of the excitation effect along with the time lag A.
Neural Temporal Point Process. Hawkes processes only account for additive, excitation effects, and inadequate capture various complex event dependencies. To overcome this limitation, recent works (Du et al.) 2016) Mei and Eisner 2017) use neural networks to model the conditional intensity. Typically, recurrent neural network (RNN) is used to capture the complex event dependencies. For
2

---

Under review as a conference paper at ICLR 2023
each event n, we introduce hidden state ha, which is computed according to the previous state h the current time, event type, and other input features. An illustrative example is
h RNN- LENN-Cel(hh,-1:4e 8n).
(3)
Then obtain the conditional intensity through a positive transformation the hidden state, e.g., (t) = [(w h.). We then substitute the intensity into (T) and maximize the likelihood to estimate the model parameters.
Although the NN modeling of (t) greatly increases the model capacitylexrressvitty it makes the cumulative intensity in the likelihood []). namely 5 >x (f)d,, analytically intractable compute. We have to use approximations, such as Monte-Carke sampling and numerical quadrature, which can be expensive and complex. To sidestep this issue, (0i al. (2019) instead modeled the cumulative conditional intensity with the RNN output,
d (hn fep (MNet (hn-1, [" ((T)dT,
(4)
where 1,-1 S In ha-1 1 the RNN state corresponding to the last observed event, MNet is monotonic net (Sill 1997. Chilinski and Silva 2020), which guarantees the output is monotonically increasing along with the input time and hence consistent with the cumulative intensity, and fap is the softplus function, (.) = log(1 + exp(-)). which is to ensure the positiveness. Note fap is also monotonically increasing, and the transformation with Sep does not change the monotonicity on Since Omi et al. (2019) only considered single-type events, omit the subscript ie and denote the single conditional intensity by A(t). Given (4), we can obtain the conditional intensity by taking the
derivative, X(In) = 0e(hn-1.t)
The likelihood of an event sequence [L1,...,AN] [x]] is
3t
t=t,
(-/10 x(t)at)
.exp exp (-f"" x(1)at)
exp N
III N
pll,,...; (N)
A(tm)
N
a (hh hn -1;)
= II (hh--1;4)) WhNN T)
II
(5)
Ot
(=tn
where to = 0 and ho initial state of the RNN. Since there is integration, the computation and optimization is much easier and more convenient, especially with automatic differentiation libraries.
Model
Although the model of (Omi et al. 2019) is smart and successful, only supports single-type events. More important, we found that it often suffers from inefficient learning and easily falls into poor performance. To address these issues and to further take advantage of the structural knowledge within the event participants, we develop GINPP, graph-informed neural point process model based on the monotonic net, presented as follows.
Specifically, we assume our dataset includes K types of events, which were launched by M par- ticipants. Each observed event sequence is a series of mixed-type events launched by particular participant. For example, online social media, tweeter account can be viewed as participant, which can launch a series of events of different types: tweeting. retweeting, replying, like, direct messaging, etc. We denote the event sequence of participant m by I [[tT", s s ),..., ...,, (t7 $ )
where each event type s E [l....,AA K} (1 S n < N,... Among the M participants a graph structure that encodes their correlations, denoted by G (E, V) where V == (1...., M) is the vertex set, and E.- {(i
is the edge set.
First, we consider extending (Omi et al. 2019) to support multiple event types. A straightforward extension is to follow the idea of (Mei and Eisner 2017) and expand the output dimension of the monotonic net to K (see (4) for each participant (vertex) m.. We then apply element- wise softplus transformation to obtain the cumulative conditional intensity for each event type k namely, @% (h,
fap (MNet (h"-,.t) [kl).. where h is the hidden state of the vertex m. However, this method will increase the learning challenge of the monotonic net, because K monotonic constraints have to be satisfied simultaneously. To cireumvent this issue, construct marked point
3

---

Under review as a conference paper at ICLR 2023
process (Daley and Vere-Jones )007). where the event type is considered mark of the event. As in (4) we still use a single-output monotonic net, but to model a global cumulative conditional intensity @b - f). We then introduce mark distribution to sample the event type according the last state h and the time lag A, = 1
(6) where (() is the output of a neural network, and u1 is the embedding of event type k, which will be jointly estimated during training. The conditional intensity is therefore given by Xg (t)
p(s = k(t) exp (u ((
XT (t)ps) k|t) where (t) B0th" 1:() is the global conditional intensity. The likelihood of the
event sequence m = [(TT) .88) ),
(*% $ ) is therefore a minor adjustment of (5)
K
p( exp
x (t)
dt
II n=1
Nm II o (h (h T)
N..
de 1,t) mm p(s = s
II
Öt
t=
This simple modification enables us to freely model multiple types of events, but sidesteps the difficulty of learning a multi-output monotonic net. Empirically, we found our method is much more effective.
Second, we investigate the learning challenge of (Omi et al., 2019). We find the bottleneck arises from the standard softplus trans- formation J'ap in (4). which enlarges the prediction scales of the monotonic net and increases the training difficulty. Specifically, as
a continuous relaxation of the ReLU activation, the output fap (a) approximately equal to the input x only when the output is rel- atively big, e.g., 3.05 = (3). By contrast, when the output is small, e.g. close to zero, the input x negative number with much larger scale, e.g., 0.0009 = f-p(-7). (-7). This can be seen from the inverse, f=p (y) = log(e" 1). As shown in Fig. la when close to zero, and the inverse function
0.
(a) Inverse softplus
candad did own y- datis o- o-.y-
close zero, ed varies violently. As a consequence, a small output range requires much wider input range, and the monotonic net's prediction has to cover this input range. For example, an output range [10- 3] corresponds the input range 19.21, 2.95). Hence, the varia- tion of the monotonic net's prediction is greatly enlarged. This can be further verified from a probabilistic analysis. Suppose the training output follows uniform distribution in 30. This is reasonable, because in practice, we often normalize the data to avoid their scales being too large for better numerical stability and optimization efficiency. We then look into the corresponding input distribution of the standard softplus. As shown in Fig. lb (the blue line), the input distribution is much wider, and includes long, fat tail over the negative domain. That means, the prediction of the monotonic net has to fit many negative values with much larger scales, which can bring back the troubles in optimization efficiency and numerical stability. To alleviate this issue, we propose shift-scale variant of the softplus function.
68
0.992
(b) Density curve
Figure 1: Inverse of the standard softplus and the input distribution for shift-scale softplus.
(xp a. 7) = y log(1 + exp( a
(8) where a and y are two hyperparameters, shift and scale. When a > O, the function body is moved toward the right of the x-axis and hence the negative input range needed to obtain close-to. to-zer outputs is shrunk. Furthermore, when we choose 7 E (0, 1). the absolute value of the input is amplified. That means, achieve the same output, e.g., 0.0009, the scale of the input x i.e. the output of the monotonic net - can be greatly decreased. Therefore, both a and y can shrink the input range, and reduce the fat tail of the distribution over the negative inputs, so as make monotonic net learning easier and more efficient. Fig. shoss the input distribution with
0, 0.5 and = 7 = 0.5. both cases, the distribution over the large negative inputs is greatly reduced. The reduction with a learning with our shift-scale softplus function (see Sec. 6.1
0.5 is more significant. Our experiments have verified the improvement of
4

---

Under review as a conference paper at ICLR 2023
Finally, to incorporate the valuable graph knowledge, we use an attention mechanism to model the RNN states at each vertex (partici- pant), based on the observed events occurred on both the vertex itself and its neighborhood (see Fig. 2). Specifically, we introduce an em- bedding vector Vm represent each vertex m. To compute the RNN states at each vertex we flatten the observed event sequences on
state
bi"" -10
Vertex
Neighborhood
vertexes
Figure 2: State update for vertex.
and its neighborhood Bm to obtain a single sequence 1TM = [[t1, $1, 81, ri)..... ..... (tN, &N, TN)]. where
t S ... S t.. ( ) > are the event types, {rn j the vertex indices, and each In E {m} U For vertex m, we introduce a hidden state h, at each event (In, Sn, Pn)).. We update h by
8n RNN-Cell(h Usao bn tn-1),
(9)
Gn a(T Vr.), h On gn
(10)
an) h
(11)
where is the sigmoid function, On is attention score computed from the inner product of the vertex embeddings, and r > 0 adjusts how much the score is leaning toward 0 or (in our experiment set 2.0). We can see that even if the event did not occur vertex m, L.e,, Tn m, still update the state on m according to its similarity to the neighborhood vertex rns i.e., the attention score On. This score reflects how much the vertex m should pay attention the event happened the neighborhood to update itself. vertex m and nn are very dissimilar, i.e., an RS 0, then there is almost no update. In this way, the correlation between the vertexes (participants), which are reflected by the graph structure, are integrated into our model to bolster the estimation of the cumulative conditional intensity (see (4), better fit the observed data and improve the prediction. When there is no graph structure, we have all ra m and On = 1, and we return to the standard RNN updates.
Algorithm
Given collection of observed event sequences all the vertexes (participants), D (Si,..., . St. } where each S; = {I?,...
(1 < < L) and each 17 is an event sequence on vertex m develop stochastic mini-batch learning algorithm embeddings [v;} 1.4. 11 the event type embeddings {ux 1 the parameters of the RNN and monotoic (see (9) (nd ) and the NN parameters for mark distribution (see (6)).
< m M),, "07....,,"" I'M
estimate the vertex
Specifically, to be scalable to large graph G, at each step, we use random walk to sample small set of vertexes B. We view the vertexes in B are neighborhood each other. Thereby, the neighborhood includes not only the vertexes that are directly connected, but also the ones connected via short path. In this way, we can exploit more abundant local structural information. Next, handle large number sequences L, we randomly select sequence set Si (1 E {1,..., and use the subset of sequences on the sampled vertexes, {1 m B), to compute the likelihood (see 7 and its gradient. We use this gradient stochastic gradient to update the model parameters. We apply any stochastic optimization algorithm. Our stochastic training is summarized in Algorithm
Algorithm GINPP (D, E, ()
Initialize the model parameters repeat
Randomly shuffle the sequence sets in D = (SI,...,S.) St} for each set S do
Random walk over the graph G to sample a small set of vertexes B.
Pick the subset of sequences associated with B: A {T m E 55)
Compute the gradient of the likelihood on A according to 7
Update the model parameters with the gradient (e.g., ADAM)
end for
until E epochs are done
5

---

Under review as a conference paper at ICLR 2023
5
Related Work
Poisson process is a popular tool for event data analysis, e.g., (Loyd et al. 2015 Schein et al. 2015]
2016 2019), but its independent increment assumption ignores the event dependencies interactions.
Many works therefore propose to use Hawkes processes (HPs) (Hawkes] 1971) to capture the mutual
Tan et al. 2016 Linderman and Adams 2014 Du et al. 2015 He et al., [2015) Wang et al. 2017 Yang et al. [011
excitation effects among the events, such as (Blundell et al. 2012
a Xu and Zha 2017: Xu et al, 2018). A series of works improve the learning with HPs. such as nonparametric
kernel estimation (Zhou et al. 2013 Zhang et al., 2020b Zhou et al. 2020), short doubly
7). Granger causality (Xu et al. 2016) and online estimation (Yang et al. 2017b). Another recent line of works (Zhe and Du 2018 Pan et al., 2020 Wang et al., 2020)
event sequences (Xu et (017))
uses the HP framework for high-order structure decomposition and representation learning.
In order to estimate more complex event dependencies, recent research has attempted to use neural networks to construct temporal point processes. Du et al. (2016) used an RNN to construct a marked temporal point process, where the conditional intensity is formulated as linear transformation of the RNN state and the time difference then through an exponential transformation to ensure the positiveness. Mei and Hisner (017) proposed continuous-time LSTT (Hochreiter and Schmidhuber 1997) to model the conditional intensity and used the softplus transformation to ensure the positiveness. To handle the intractable integration in the point process likelihood, Mei and Eisner (2017) used a Monte-Carlo approximation. The recent works (Zhang et al. 2020a; Zuo et al. (020)) use the attention mechanism (Vaswani al. 2017 Bahdanau al. 2014) to replace the RNN framework, but they still have to approximate the integration in the like shood. Omi et al. (2019) bypassed this and Silva, 2020) to
problem by feeding the RNN states into a monotonic net (Sill
directly model the cumulative conditional intensity, so we do not need to explicitly compute the integration. The monotonic net is typically a multi-layer perceptron but imposes the nonnegative constraint over the weights during learning to fulfill the monotonicity. While successful, the model of Omi et al. (0199) only supports single-type events, which can be limited in practice. Our work GINPP extends their model to support multi-type events with the marked point process framework (Daley and Vone-Jones 2007), which is simple and efficient. Note that Du et al. (016) also used the marked point process framework. GINPP uses the RNN framework to model the hidden state due the stable and excellent performance, but an attention mechanism is used to incorporate the graph structure into the state computation. Hence, GINPP can be viewed hybrid approach. It is straightforward to extend GINPP to a full attention model.
Other works include Zhou et al. (202I) that extends HPs with mixture of shifted Beta densities, ODE based models (Rubanova et al., 2019), neural network influence kernels (Zhu et al. 202 deep Fourier kernels (Zhu et al.) 202Tb), the intensity- free point process learning (Shchur et al.. 2019) that models the time difference between successive events, etc. Based on the intensity-free framework (Shchur et al. 2019). Zhang et al. (202T) also proposed neural point process model with prior graph incorporated. But their goal is to infer the labels of nodes in the graph to detect clusters and anomalies, rather than predict the occurrence and type of new events. The recent work of Pan et al. (202I) develops nonparametric decaying model of the temporal inference, and can explicitly recover various excitation and inhibition effects, and their decay patterns among the events. While flexible and interpretable, it still needs approximate the cumulative conditional intensity, which is done via Gauss- Lague-L quadrature.
Experiment
6.1 Ablation Study
We first performed an ablation study to confirm the effectiveness of our shift-scale softplus transfor- mation 8). To this end, we tested with two functions:
(x) = 0.5 a (5(x- - 1)), g2(x)= = 0.3. (0.5 (o.5 2/2 + e - -1) - 0.2 (12)
where 91 (x) > 0 monotonically increases with the input x and g2 does not. We used a monotonic net plus the shift-scale sofiplus transformation to learn the two functions. The monotonic net includes two layers, with 256 neurons per layer, and tanh activation. We uniformly sampled 50 training points from E 2). We implemented the model with TensorFlow, and used Adam for stochastic optimization. The learning rate was set to 10-3 33 We set a = 0, 7 = 0.2 and a = 0.5, y = 0.2. We also tested with the standard softplus We show the learning curves and estimated the functions in

---

Under review as a conference paper at ICLR 2023
standard sotalus
0.5
Ground Truth
B. y
o 0.5.y002 02
0.4-
=-
0.30
-.0.5.--02
0.2
0.05
0.00
0.U
100
200
300
400
503
600
0.00
obs 0.50
0.75
1.00
1:55
1.50 1.75
2.00
apachs
(a) gii learning curve
(b) 9i estimation
0.5
0.4-
standare sotta
a- 0.0. y 0.2
1.5
standard sottlus
Ground Truth
0.5, y 0.2
0-0.0 y 02
0.3-
1.0
0-0.5.y
6.2
0.5
0.1-
0.0-
0.0
350 0.00
0.25
0.50
0.75
1.00 125 1.50 175 2.50
(c) 92 learning curve
(d) ga estimation
Figure 3: The learning curves and estimation results of monotonic net with softplus transformation.
Fig. 3 As we can see, with our shift-scale softplus, the learning converges much faster (see Fig. 3a and Fig. Bc). With the standard softplus, the learning fell into a relative big loss first, and stayed there for quite long time before the new improvement started to happen. By contrast, during the stage that the standard softplus was stuck, our shift-scale softplus had driven the training loss to drop fast and converged (see the red line). This shows that the shift-scale version can greatly boost the learning efficiency. Note that for standard softplus, the long stucking stage can signal wrong message that the training can stop. The estimated functions with our shift-scale softplus are also more accurate. From Fig. 3b and 3dl our learned functions are quite close to the ground-truth. In Fig. 3b our estimation (a = 0.5, 7 0.2) almost overlaps with the ground-truth.. By contrast, the estimation with the standard softplus (the blue line) has clear deviation. Therefore, it shows that our shift-scale softplus not only bolster the learning efficiency but also leads to better learning results.
Next, examined if GINPP can recover the ground- -tutt conditional intensity. To this end, we generated synthetic dataset of bi-type events, where type 0 events ex- cite type events while type inhibits type 0, and events of the same type do not influ- ence on each other,
Ground Tuuhh Estimatec
Po--(()) (A) max (1.0- - 0.05.2 0), (1) = - 0.5 0xp 0.55A.
himet
(a) Total intensity
Est matest Type
Ground Truth Type 0 Grandd Thates Tpoe 1
Tspa 9 Events Type 1 Events
Eht mated Tyme
We then substitute po-+1 and p1-0 into 2 (background rate is zero). To ensure the positiveness, we applied softplus transfor- mation to obtain the conditional intensity. We used Thinning algorithm Lewis and Shedler 1979) to sample 10K sequences for traiiing and 1K for validation. Each sequence consists of 64 events. We evalu-
20
30
(b) Intensity of each type
Figure 4: Estimation of the conditional intensity.
ated the likelihood of the validation set after each training epoch, and halted the training when no improvement was observed (i.e., early stopping). We used the GRU cell for our RNN component. The dimension of the hidden state and event type embeddings was set to 64 and 16, respectively. For the monotonic net component, we used two hidden layers, 256 neurons per layer, and tanh activation. For the NN in the mark distribution (see (6). we used one hidden layer, with 256 neurons and RELU activation function. We set 7 = 0.5 for the shift-scale softplus. We used Adam for stochastic optimization with learning rate 10 and mini-batch size 64. The maximum number of epochs was set to 100. Fig. 4a shows the estimated total conditional intensity, and we can see it matches the ground- tround-truth quite closely. Then Fig. reportss the derived conditional intensity for type 0 and type 1 from the learned mark distribution. As can see, GINPP can also accurately recover the intensity for each event type, although does not explicitly estimate them. The results have confirmed the capability of our multi-type event model.
7

---

Under review as a conference paper at ICLR 2023
6.2 Predictive Performance
Next, we evaluated the performance of GINPP in predicting the time and type of future events. To this end, we used six real-world benchmark datasets, where the first four do not include graph information and the remaining two are associated with graphs. (1) Retweet (Zhao et al., (015), including 24K event sequences and three event types: retweeting by "small", "medium" and "large" retwecters. (2) SO (Du et al. 2016), 6, 633 awarding sequences in the Q/A site Stack Overflow. The event type is the award, such as "Nice Question"" "Guru", and "Great answer". There are 22 event types. (3) MIMIC (Du et al. 2016), clinical visit events from 650 anonymous patients in seven years, and the event type is the diagnosis outcome. There are 75 outcomes. (4) Socia 48.9K student activity events in the campus of a university. The event type corresponds to the campus location of the event and we have 50 event types, and 1,614 event sequences happened among 51 students. (5) 911Cal 251K emergence calls in Montgomery County of Pennsylvania from year 2015 2017. There are three types of calls (events): EMS, FIRE and Traffic. We have collected 6, 187 event sequences from 73 zones. We viewed each zone as vertex, and added an edge if two zones are neighboring each other. (6) SLCCrime 48.7K criminal events happened in 23 zones of Salt Lake City. There are 25 event types (e.g., robbery, murder and arson) and 1,403 sequences. We viewed each zone as a vertex and created the graph according to the geographic neighboring relationship.
We compared with the following popular and/or ofate-of-the-art the--rt approaches: (1) PP. simple ho- mogencous Poisson process. (2) Hawkes process (HP) with an exponential triggering kernel, (3)
Recurrent Marked Temporal Point Process (RMTPP) (Du et al. 2016). (4) Neural Hawkes Processes (NeuralHP) (Mei and Eisner 2017). (5) Self-Attentive Hawkes Process (SAHP) (Zhang et al.) 2020a)
that uses the self-attention mechanism to model the conditional intensity for each event type, (6) Transformer Hawkes process (TRHP) (Zuo et al.) 2020) that also uses the attention mechanism to model the intensity. In addition, we compared with (7) Simple Statistics (SS) that predicts the occurrence time of new events with the average lag between successive events, and predicts the new event type with the most frequent type of the observed events. We used the original imple- mentation of NeuralHP /tttps::/githubb.. wttp:///mmmmmmmmmmameeeemmmm
http:/tttti..ccmmmmM/tttaattttat ITM.EU/ (ww.////nnnurawwessasmoo SAHP (https TRHP (https:/ www.t..tu...... com/ and a popular open-source implementation RMTPP). For GINPP, we employed
7/gi http:///mmm.aaaaaaeeeeeeeeeeeaap
of RMMPPP (https: httpe//PPAttt///atttoooooooooooa the same setting as in the ablation study (see Sec. [. In addition, we set the dimension of the vertex embeddings to 16. To perform random walk, we randomly selected initial vertex, and each step randomly hopped to a vertex that connects to the current vertex. The probability is the inverse of the degree of the current vertex. We maintained the visited vertex set until the number reached to the mini-batch size. Except SS, all the methods used ADAM for stochastic optimization with learning rate 10- 3, and the mini-batch size was chosen from {8, 16, 32). We used the default settings of all the other methods, and early stopping for every method.
We randomly split each dataset into 70% for training, 10% for validation, and 20% for testing. Each method was used to predict the occurrence time and type of the last event in each test sequence. We repeated the experiment for five times, and computed the average root-mean- square-error (RMSE) and classification accuracy (ACC) for time and type predictions, respectively. We computed the standard deviation. For all the datasets, we also ran our method without graphs (i.e., we set an empty 9), denoted by GINPP-1. The results are reported in Table T As we can see, in every dataset, GINPP outperforms all the competing methods, in many cases by a large margin. It shows our method is superior in both event time and type prediction. When the graph knowledge is available, GINPP is better than our method not incorporating the graph structure, i.e., GINPP-1. The results show that the graph structure can further facilitate the training and prediction, and our attention method is effective.
root-mean-ssqure-ereee
Finally, we investigated the attention score between the vertexes after training. Specifically, we examined three zones in 9// Call dataset: Zone 2, 52 and 67. Fig. 5a shows the locations of all the zones. The points represent the observed events on those zones. The color represents the region of each zone. Then Fig 5b] 5c| and 5d]ssw the attention scores between the all the event locations and the events in zone 2, 52 and 67, respectively. The color indicates the magnitude; see the color bar. We can see that in many cases, when the event locations are in the neighboring zones, the attention
html

---

Under review as a conference paper at ICLR 2023
RMSE
Retweet
MIMIC
Social
911Call
SLCCrime
SS
33.241 1: 0.28
1,561 0.02
1,274 1 0.07
11.643 1 3.28
21.754 1 0.52
8.024 0.30
PP
34,294 1: 0.33
1.178 0.02
1.132 0.05
7.630 1 1.08
19.119 0.24
6.340 0.36
HP
32.557 1 0.40
1.200.0.001
1,002 0.05
7,670 1 1.03
18.380 0.244
6.382 1.336
RMTPP
47.704 0.29
1.656 0.04
1,015 0.04
7.966 = 1.06
22.031 1.06
8.873
NeuralHP
34.912 0 0.34
1.73
1.026 0.03
7.740 1.07
20.790 0.23
7,498 0.40
SAHP
34,894 = 0.35
1,565 #0.05
1,035 0.04
7,941 1.07
20.398 0.30
7.970 + 0,40
TRHP
34,055
+ 0.36
1.127 + 0.02
1,071 - 0.03
7.912 + 1.03
20.919
0.26
7.337 + 0.48
GINPP-1
32.258 0.31
1.112 = 0.01
0.874 - 0.03
7.616 + 1.08
16.155 0.27
6.072 + 0.31
GINPP
0.28 15.446
5,993 0.29
ACC
SS
0.549 + 0.002
0.366 0.010
0.305
0.021
0.122 - 0.006
0.526
0.008
0.222 0.006
PP
0.549
0.366
0.186 0.050
0.110 0.005
0,544 0.008
0.181 0.011
HP
0.540
0.357
1.041
0.123 0.009
0
0.216 0.013
RMTPP
0.006
0.006
0.848 0.848
0.509 0.008
0.531
0.203 = 0.009
NeuralHP
0.574
0.383 1.83+000 0.007
0 0.041
0.556 0.009
4.111
0.218 0.013
SAHP
0.497 + 0.027
0.305 + 0.025
0.337 = 0.036
0.053 0.015
0.293
0.041 0,016
TRHP
0.541 + 0.004
0.375 + 0.008
0.768 0.015
0.485 0.007
+
0.220 = 0.011
GINPP-1
0.607 0.004
0.401 0.009
0.863 0.015
0.616 1 0.011
0.550 0.005
0.228 0.011
GINPP
0.553 0.005
0.233 0.009
Table 1: Predictive performance of the time (RMSF) and type (ACC) of the future events.
X
40.4
40.4
0.5
40.3
40.2
402
0.4
40.2
6.2
40.0
15.
75-4
-25
-75.
75.5
75.4
75.2
(a) Zone map
(b) Zone
0.5
40.3
0.5
403
0.5
60.2
602
40.1
43.1
0.2
60.0
63.0
Lanqeuaee
LongRuude
(c) Zone 52
(d) Zone 67
Figure 5; The attention scores of 52, and 67. The cross the center of the zone.
score with them is high. This implies those events take significant impacts on updating the hidden state of the zone. However, there are also cases that the attention score with neighboring zones is small or close to zero, for example, the purple points surrounding Zone 2 (see Fig. (5b) and on right side of Zone 67 (see Fig. 5dh. In such cases, the neighborhood events have little effect on the state update. This has shown the selection effect of the attention mechanism. On the other hand, the attentions score in general smaller or closer to zero for distant zones. This might be attributed to the incorporation of the graph structure, because the distant vertexes unlikely to be collected by the random walk, and hence their embeddings can be more dissimilar after training.
7 Conclusion
We have developed GINPP, a graph-inform neural point process that can avoid intractable integra- tion in the likelihood, support multiple event types, and incorporate valuable graph knowledge into training and prediction. The experiments in ablation studies and seven real-worlh applications have shown the encouraging results.

---

Under review as a conference paper at ICLR 2023
References
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint MKKVv:140900433
Blundell, C., Beck, J., and Heller, K. A. (2012). Modelling reciprocating relationships with hawkes processes. In Advances in Neural Information Processing Systems, pages 2600-2608.
Chilinski, P. and Silva, R. (2020). Neural likelihoods via cumulative distribution functions. In Conference on Uncertainty Artificial Intelligence, pages 420-429.. PMLR.
Daley, D. J. and Vere-Jones, D. (2007). An introduction the theory of point processes: volume II: general theory and structure. Springer Science & Business Media.
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, L.. (2016). Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1555-1564.
Du, N., Farajtabar, M., Ahmed, A., Smola, A. J., and Song, L. (2015). Dirichlet-hawkes processes with applications to clustering continuous-time document streams. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 219-221 228. ACM.
Hawkes, A. G. (1971). Spectra some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90.
He, X., Rekatsinas, T., Foulds, J., Getoor, L., and Liu, (2015), Hawkestopic: A joint model for network inference and topic modeling from text-based cascades. In International conference machine learning, pages 871 970-
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 988):1735 1780.
Lewis, P. W. and Shedler, G. S. (1979). Simulation of nonhomogeneous poisson processes by thinning. Naval research logistics quarterly, 2633):403-413.
Linderman, S. and Adams, R. (2014). Discovering latent network structure in point process data. In International Conference on Machine Learning, pages 1413-1421.
Lloyd, C., Gunter, T., Osborne, M., and Roberts, S. (2015). Variational inference for gaussian process modulated poisson processes. In International Conference on Machine Learning, pages 1514-1822. PMLR.
Mei, H. and Eisner, J. M. (2017). The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, pages 6754-6764.
Omi, T., Aihara, K., et al. (2019). Fully neural network based model for general temporal point processes. Advances in Neural Information Processing Systems, pages 2122-2132.
Pan, Z., Wang, Z., Phillips, J. M., and Zhe, S. (2021). Self-adaptable point processes with nonpara- metric time decays. Advances Neural Information Processing Systems, 34:4594
Pan, Z., Wang, Z., and Zhe, S. (2020). Scalable nonparametric factorization for high-order interaction events. In International Conference on Artificial Intelligence and Statistics, pages 4325 4335.. PMLR.
Rubanova, Y., Chen, R. T., and Duvenaud, D. K. (2019). Latent ordinary differential equations for tregularly-samplee time series. Advances in neural information processing systems, 32.
Schein, A., Linderman, S., Zhou, M., Blei, D., and Wallach, H. (2019). Poisson-randomized gamma dynamical systems. In Advances in Neural Information Processing Systems, pages 782-793.
10

---

Under review as a conference paper at ICLR 2023
Schein, A., Paisley, J., Blei, D. M., and Wallach, H. (2015). Bayesian poisson tensor factorization for inferring multilateral relations from sparse dyadic event counts. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1045-1054. ACM.
Schein, A., Zhou, M., Blei, D. M., and Wallach, H. (2016). Bayesian poisson tucker decomposition for learning the structure of international relations. In Proceedings of the 33rd International Conference International Conference Machine Learning Volume 48, ICML' 16, pages 2810-2819. JMLR.org.
Shehur, O., Bilovs, M., and Günnemann, S. (2019). Intensity- free learning of temporal point processes. In International Conference on Learning Representations.
Sill, J. (1997). Monotonic networks. Advances in neural information processing systems, 10.
Tan, X., Naqvi, S. A., Qi, A. Y., Heller, K. A., and Rao, V. (2016). Content-based modeling of reciprocal relationships using hawkes and gaussian processes. In UAL
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u. and Polosukhin, (2017). Attention is all you need. In Guyon, I., Luxburg. U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Wang, Y., Ye, X., Zha, H., and Song, L. (2017). Predicting user activity level in point processes with mass transport equation. In Advances in Neural Information Processing Systems, pages 1644-1654.
Wang. Z., Chu, X., and Zhe, S. (2020). Self medf-modul nonparametric event-tensor factorization. In International Conference on Machine Learning, pages 9857-9867. PMLR.
Xu, H., Farajtabar, M., and Zha, H. (2016). Learning granger causality for hawkes processes. In International Conference on Machine Learning, pages 1717-1726.
Xu, H., Luo, D., Chen, X., and Carin, L. (2018). Benefits from superposed hawkes processes. In International Conference on Artificial Intelligence and Statistics, pages 623-633 631.. PMLR.
Xu, H., Luo, D., and Zha, H. (2017). Learning hawkes processes from short doubly-censored event sequences. In International Conference Machine Learning, pages 3831-3840.
Xu, H. and Zha, H. (2017). dirichlet mixture model of hawkes processes for event sequence clustering. In Advances Neural Information Processing Systems, pages 1354-1363.
Yang, J., Rao, V. A., and Neville, J. (2017a). Decoupling homophily and reciprocity with latent space network models. In UAI.
Yang. Y., Etesami. J., He, N., and Kiyavash, N. (2017b). Online learning for multivariate hawkes processes. In Advances in Neural Information Processing Systems, pages 4937-4946.
Zhang, Q. Lipani, A., Kirnap, O., and Yilmaz, E. (2020a). Self-attentive hawkes process. In International Conference on Machine Learning, pages 11183-11193. PMLR.
Zhang, R., Walder, C., and Rizoiu, M.-A. (2020b). Variational inference for sparse gaussian process modulated hawkes process. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 6803-6810.
Zhang, Y., Sharma, K., and Liu, Y. (2021). Vigdet: Knowledge informed neural temporal point process for coordination detection on social media. Advances in Neural Information Processing Systems, 34:3218-3231.
Zhao, Q., Erdogdu, M. A., He, H. Y., Rajaraman, A., and Leskovec, J. (2015). Seismic: A self- exciting point process model for predicting tweet popularity. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1513-1522. ACM.
11

---

Under review as a conference paper at ICLR 2023
Zhe, S. and Du, Y. (2018). Stochastic nonparametric event-tensor decomposition. In Advances in Neural Information Processing Systems, pages 6856-6866
Zhou, F., Li, Z., Fan, X., Wang, Y., Sowmya, A., and Chen, F. (2020). Efficient inference for nonparametric hawkes processes using auxiliary latent variables. Journal of Machine Learning Research, 21(241):1-31.
Zhou, F., Zhang, Y., and Zhu, J. (2021). Efficient inference of flexible interaction in spiking- neuron networks. In Proceedings of the International Conference on Learning Representations (ICLR).
Zhou, K., Zha, H., and Song, L. (2013). Learning triggering kernels for multi-dimensione. hawkes processes. In International Conference on Machine Learning, pages 1301-1309.
Zhu, S., Wang, H., Dong, Z., Cheng, X., and Xie, Y. (2021a). Neural spectral marked point processes. In International Conference on Learning Representations.
Zhu, S., Zhang, M., Ding, R., and Xie, Y. (2021b). Deep fourier kemnel for self-attentive point processes, In International Conference on Artificial Intelligence and Statistics, pages 856-864. PMLR.
Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. (2020). Transformer hawkes process. In International Conference on Machine Learning, pages -17221 PMLR.
12


Reviews:
[{"review_id": "sfY8hWnObF6", "reviewer": "Reviewer_ou7C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces several modifications to the neural point process model from [Omi et al. 2019](https://arxiv.org/abs/1905.09690) that aim to improve the expresiveness and training speed of the model.\nThese improvements include:\n- Passing the output of the monotonic neural network (modeling integrated intensity) through a shift-scale softplus activation function. This change leads to faster training of the model.\n- Extending the model to marked TPPs with categorical marks by modeling the conditional distribution of the mark type given time $p(s = k | t)$ with a neural network.\n- Incorporating the known relation structure between event types into the RNN update equation.\n", "summary_of_the_review": "A more thorough empirical evaluation would be necessary to justify that all the proposed modifications to the model architecture indeed have a positive effect over the base model of Omi et al.", "strength_and_weaknesses": "Strengths:\n- The paper is clearly written and is easy to follow.\n- The proposed parametrization of the event type distribution given the time $p(s = k | t)$ provides an elegant and parameter-efficient way to model the intensity of marked TPPs. Conceptually, it offers clear advantages over models that model marks independently of the inter-event times.\n- I really liked the inclusion of the \"simple statistic\" baseline in the experiments and wish more papers did the same.\n\n\nWeaknesses:\n1. Lack of ablations that convincingly demonstrate the effectiveness of each proposed modification to the model. It would be necessary to conduct experiments studying the effect of each individual modification. These would include comparisons to the original model from Omi et al.:\n    - Compare the vanilla model from Omi et al. vs. model with the proposed shift-scale softplus activation on real-world dataset (showing both the convergence plots and the final RMSE/Accuracy results). The current experiment with recovering simple functions may not be representative of behavior on real-world event datasets.\n    - Compare the proposed parametrization of $p(s_n = k | t)$ vs. the vanilla model from Omi et al. with conditionally independent marks $p(s_n = k)$ (as done, e.g., in [Du et al. 2016](https://www.kdd.org/kdd2016/papers/files/rpp1081-duA.pdf)).\n    - The difference in performance between the model that incorporates the relational structure between event types (GINPP) and the model that ignores the graph (GINPP-1) in Table 1 is within the standard deviation of the scores in 3/4 cases. This brings into question whether the proposed attention mechanism improves the predictive performance.\n\nThe lack of these ablations makes it impossible to determine whether the overall good performance of the model in Table 1 is related to the proposed modifications, or if the original model by Omi et al. would achieve comparable results.\n\n2. The novelty claim for the shift-scale softplus function (\"we propose a shift-scale variant of the softplus function\") could be adjusted. The scaled softplus function is quite standard (e.g., available in [PyTorch](https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html)) and was even used in the context of TPPs in the original Neural Hawkes Process paper. It's also unclear whether the shift component of the softplus function is needed for the reported improvements due to the missing ablation. It could also be that simply normalizing the inter-event times before fitting the model would achieve similar improvements.\n\n\nMinor comments:\n\n3. Equations 5, 7: The first two terms in the second line should be $\\exp(-\\phi(...))$ instead of $\\phi(...)$.\n4. The statement \"this method will increase the learning challenge for the monotonic net because K monotonic constraints have to be satisfied simultaneously\" in Section 3 is not accurate. The monotonic constraints are satisfied by design of the neural network, so satisfying them simultaneously poses no challenge. Moreover, a recent work (https://dl.acm.org/doi/abs/10.1145/3511808.3557399) suggests that modeling a separate intensity for each mark can often improve the predictive performance.\n5. In Section 6, clarify if we predict the type of the next event assuming the time is known (i.e., using $p(s_n = k | t = t_n)$) or without knowing the time (i.e., using $p(s_n = k)$).\n6. Results for Poisson process (PP) in Table 1: The accuracy results should be identical for PP and SS, since the estimated intensity of the PP for the majority class should be higher than for other classes. Could it be that the PP model hasn't converged during training? Could other models be affected by the same issue?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sMT5_aKY31", "reviewer": "Reviewer_aMtU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new neural point process model with three improvements: (1) shift-scale variant for the output of monotonic net; (2) use marked process with conditional distribution; (3) incorporate graph information with random walk-based neighbor sampling. The authors compared the new method GINPP to several state-or-art methods on 6 real-world datasets. ", "summary_of_the_review": "The authors propose three practical improvement to NPP models with monotonic net. However, the technical contributions are only marginally significant or novel.", "strength_and_weaknesses": "Strength\n1. The authors carry out empirical evaluation on a diverse set of real-world point processes with comparison to several state-of-art methods with superior performance.\n2. The three improvements proposed in the new methods are well-justified with good motivation and dedicated analysis e.g. the scale shift-scale variant for the monotonic network output.\n3. The authors carry out experiments on synthetic dataset to individually verify the effect of different components.\n\nWeakness\n1. The improvements to the previous method is a bit incremental. The marked process with conditional distribution is already widely used in point process modeling like Hawkes process. Similar to the incorporation of the neighbor information. The shift-scale variant output is also a standard trick for NN.\n2. From the results in Table 1, it seems that the incorporation of graph information does not lead to better accuracy. Among two metrics on 2 datasets, GINPP is significantly better with only one setting (911Call RMSE). Moreover, it would be better for the authors to mark improvement with statistical significance.\n3. It would be better if the authors could carry out single variable ablation experiments for the improvement (1) and (2). Currently, (1) is tested only partially on a monotonic net. It would be better to see the performance in the full NPP model. For (2), only the performance of the GINPP is presented. It would be interesting to see a comparison without the usage of conditional mark distribution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_vw8Lx97Al", "reviewer": "Reviewer_cb2m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers a scenario in which there are M sequences of events, and each sequence consists of K event types.\nThe FullyNN based neural point process model (Omi 2019) demonstrates useful but needs to model each event type with a separate cumulative conditional intensity, which could be computationally demanding!\n- To address this issue, this paper uses a separate function to explicitly model event type.\n- The paper also analyzes the training difficulty of FullyNN induced by the SoftPlus function, and proposes a shift-scale variant of the softplus function, which demonstrates its effectiveness in the experiments.\n- To leverage external graph information among sequences, they use an attention structure to modify how the recurrent neural nets' update its hidden states.", "summary_of_the_review": "The paper mainly addresses a defect of FullyNN in modeling mixed-event types behind multiple related event sequences.\nThe novelty lies in modeling the event types with a separate neural net.\nThe analysis of the training issues of softplus function looks interesting, and could be very useful for latter research.\nThe paper uses an attention design to leverage external graph information among sequences, to better update RNN states, which further improve the model prediction accuracy.\n\nOverall, the methods look interesting but appears to be incremental! The final experiments, only compared some known neural TPPs. I also suggest a few alternatives to make a fair comparison.", "strength_and_weaknesses": "Strength\n- The analysis of the training issue of the softplus function, looks interesting, and could be useful for later research.\n \nWeaknesses\n- The new model in Eq.(6)(7) seems to be incremental. \n- It is not clear how to make time and event type prediction in Algo.1 of Sec.4.\nThe concrete equations for the time and type prediction, should be given !\n- Why do not compare the predictive loglikelihood of the methods ?\n- For the final experiments, why FullyNN (Omi 2019) is not included since you are circumventing its limitations? \n- As I remember, the authors of transformer Hawkes processes also provide some solutions to incorporate graph among event sequences.\nCan you compare your method to it?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U5T1MN4HIe", "reviewer": "Reviewer_jeUB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigated an important research problem in point process modeling and aimed to model multi-class event sequence in a network. As the paper pointed out, one of the recent work Omi et al. (2019) is known to be computationally efficient and flexible in representing complex triggering effects between events, however fails to model multi-class events and sometimes suffers from inefficient learning and easily falls into poor performance. The proposed method hinges on Omi et al. (2019) and extends the idea of modeling cumulative hazard function using monotonic net to support multiple event types. One of the most important assumption the proposed methods made is that the process has independent marks, i.e., the marks are conditionally independent of each other, so that the mark can be modeled separately. This simple modification enables efficient computation for both learning and inference. Another finding of this paper is that they investigate the learning challenge of Omi et al. 2019 and found that the bottleneck comes from the standard soft plus transformation at the output layer of the neural network, which causes large variance in the prediction. To alleviate this issue, the paper also proposes a shift-scale variant of the softplus function. ", "summary_of_the_review": "See above.", "strength_and_weaknesses": "In general, the paper has several strength. First, the paper is clearly written and the presentation is easy to follow. Second, the finding of the learning challenge in Omi et al. 2019 is significant. Modeling cumulative hazard function and calculating its derivative may result in unstable training process and poor predictive performance due to the softplus function, and the proposed shift-scale variant of the softplus function can indeed alleviate this problem. \n\nWhile the extension of the Omi et al. 2019 is meaningful, it does not strike me as a fundamental modeling contribution to the point process literature, in that it does not offer a novel way to address a challenging modeling issue. The reason is two fold: 1. While being computationally efficient, decoupling lambda and mark term simplifies the model and dramatically limits the representative power; 2. The contribution of the variant of the softplus function is relatively incremental regarding the problem that this paper aims to address. \n\nThere are also a number of other previous studies attempting to address the same problem in different approaches without decoupling the mark and the conditional intensity, to name a few:\n- https://proceedings.mlr.press/v157/hong21a.html\n- https://openreview.net/forum?id=0rcbOaoBXbg. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject