EasyTPP: Towards Open Benchmarking Temporal Point Processes
Authors: Siqiao Xue
Xiaoming Shi
Zhixuan Chu
Yan Wang
Hongyan Hao
Fan Zhou
Caigao JIANG
Chen Pan
James Y. Zhang
Qingsong Wen
JUN ZHOU
Hongyuan Mei
Venue: ICLR 2024 poster (2024)
Keywords: Event sequence
Temporal point process
Open benchmarking
Primary area: datasets and benchmarks
TL;DR: We present a code base along with datasets for benchmarking temporal point process.

Abstract: Continuous-time event sequences play a vital role in real-world domains such as healthcare, finance, online shopping, social networks, and so on. To model such data, temporal point processes (TPPs) have emerged as the most natural and competitive models, making a significant impact in both academic and application communities. Despite the emergence of many powerful models in recent years, there hasn't been a central benchmark for these models and future research endeavors. This lack of standardization impedes researchers and practitioners from comparing methods and reproducing results, potentially slowing down progress in this field. 
In this paper, we present EasyTPP, the first central repository of research assets (e.g., data, models, evaluation programs, documentations) in the area of event sequence modeling. Our EasyTPP makes several unique contributions to this area: a unified interface of using existing datasets and adding new datasets; a wide range of evaluation programs that are easy to use and extend as well as facilitate reproducible research; implementations of popular neural TPPs, together with a rich library of modules by composing which one could quickly build complex models. We will actively maintain this benchmark and welcome contributions from other researchers and practitioners. 
Our benchmark will help promote reproducible research in this field, thus accelerating research progress as well as making more significant real-world impacts. The code and data are available at \url{https://github.com/ant-research/EasyTemporalPointProcess}.

Published as a conference paper at ICLR 2024
EASYTPP: TOWARDS OPEN BENCHMARKING TEMPORAL POINT PROCESSES
Siqiao Xue Xiaoming Shi Zhixuan Chu Yan Wang Hongyan Hao Fan Zhou Caigao Jiang Chen Pan James Y. Zhang Qingsong Wen Jun Zhou Ant Group, Alibaba Group
sigiao. xsq0 a www.ibaaabb..... com
Hongyuan uan Mei TTIC hongyuan@ttic.edd
ABSTRACT
Continuous-time event sequences play vital role in real-world domains such healthcare, finance, online shopping, social networks, and so on. To model such data, temporal point processes (TPPs) have emerged as the most natural and com- petitive models, making a significant impact both academic and application communities. Despite the emergence of many powerful models in recent years, there hasn't t been a central benchmark for these models and future research en- deavors. This lack of standardization impedes researchers and practitioners from comparing methods and reproducing results, potentially slowing down progress this field. In this paper, we present Easy the first central repository of research assets (e.g., data, models, evaluation programs, and documentation) in the area of event sequence modeling. Our EasyTPP makes several unique contributions this area: unified interface of using existing datasets and adding new datasets; a wide range of evaluation programs that are easy to use and extend well as facili- tate reproducible research; implementations of popular neural TPPs, together with a rich library modules by composing which one could quickly build complex models. We will actively maintain this benchmark and welcome contributions from other researchers and practitioners. Our benchmark will help promote repro- ducible research in this field, thus accelerating research progress well as making more significant real-world impacts. The code and data are available at ht ttss //github. cwm/ ant conttresean research. wewecheeaaayeeeeeeeeeeeee PointP rocess...
INTRODUCTION
Continuous-time event sequences ubiquitous various real-world do- mains, such neural spike trains in neuroscience (Williams et al., 2020), orders in financial transactions (Jin
Hawkes process
Temporal point process
Article Count
Temporal event sequence
et al., 2020), and user page view- ing behavior in the e-commerce plat-
200
form (Hernandez et al., 2017). To model these event sequences, tempo- ral point processes (TPPs) are com- monly used, which specify the prob- ability of each event type's instanta- neous occurrence, also known as the intensity function, conditioned on the past event history. Classical TPPs, such Poisson processes (Daley &
100
2014 2015 2016
2017
2018
2019 2020 2021 2022 Year
Figure 1: ArXiv submissions over time on TPPs. See Ap- pendix E.1 for details.

---

Published as a conference paper at ICLR 2024
Vere-Jones, 2007) and Hawkes processes (Hawkes, 1971), have well-established mathematical foundation and have been widely used to model traffic (Cramér, 1969), finance (Hasbrouck, 1991) and seismology (Ogata, 1988) for several decades. However, the strong parametric assumptions in these models constrain their ability to capture the complexity of real-world phenomena.
To overcome the limitations of classical TPPs, many researchers have been developing neural ver- sions of TPPs, which leverage the expressiveness of neural networks to learn complex dependencies; see section 7 for a comprehensive discussion. Since then, numerous advancements have been made in this field, as evidenced by the rapidly growing literature on neural TPPs since 2016. Recent re- views have documented the extensive methodological developments in TPPs, which have expanded their applicability to various real-world scenarios. As shown in Figure 1 (see Appendix E. for details of how we count the articles.), the number of research papers on TPPs has been steadily increasing, indicating the growing interest and potential impact of this research area. These ad- vancements have enabled more accurate and flexible modeling of event sequences in diverse fields.
In this work, inspired by Hugging Face (Wolf et al., 2020) for computer vision and natural language processing, we take the initiative to build a central library, namely EasyTPP, of popular research assets (e.g., data, models, evaluation methods, documentations) with the following distinct merits:
1. Standardization. We establish a standardized benchmark to enable transparent comparison of models. Our benchmark currently hosts 5 popularly-used real-world datasets that cover diverse real-world domains (e.g., commercial, social), and will include datasets in other domains (e.g.. earthquake and volcano eruptions). One of our contributions is develop unified format for these datasets and provide source code (with thorough documentation) for data processing. This effort will free future researchers from large amounts of data- mata-processing work, and facilitate exploration in new research topics such as transfer learning and adaptation (see Section 6).
2. Comprehensiveness. Our second contribution is to provide a wide range of easy-to-use eval- uation programs, covering popular evaluation metries (e.g., log-likelihood, kinds of next-event prediction accuracies and sequence similarities) and significance tests (e.g., permutation tests). By using this shared set of evaluation programs, researchers in this area will not only achieve higher pace of development, but also ensure a better reproducibility of their results.
3. Convenience. Another contribution of EasyTPP is rich suite of modules (functions and classes) which will significantly facilitate future method development. We reproduced previ- ous most-cited and competitive models by composing these modules like building LEGOs; other researchers can reuse the modules to build their new models, significantly accelerating their im- plementation and improving their development experience. Examples of modules are presented in section 3.
Flexibility. Our library is compatible with both PyTorch (Paszke et al., 2019) and Tensor- Flow (Abadi et al., 2016), the top-2 popular deep learning frameworks, and thus offers great flexibility for future research in method development.
5. Extensibility. Following documentation and protocols, could easily extend the EasyTPP library by adding new datasets, new modules, new models, and new evaluation programs. This high extensibility will contribute building healthy open-source community, eventually bene- fiting the research area of event sequence modeling.
2 BACKGROUND
Definition. Suppose we are given fixed time interval [0, T] over which event sequence is observed. Suppose there are 7 events in the sequence at times 0 < 4 < <... < 4, S T. We denote the sequence as ToooT] (f1, kı)..... (fr, ky) where each k; (1,..., ......,K) K} is a discrete event type. Note that representations in terms of time t; and the corresponding inter-event time = t; 4 are isomorphic, we use them interchangeably. TPPs are probabilistic models for such event sequences. If we use (t (jo,)) to denote the probability that an event of type k occurs over the infinitesimal interval [t. E- dt), then the probability that nothing occurs will be -- Pk (t ()).))) Formally, the distribution of a TPP can be characterized by the intensity (x ().))) 2 for each event type k at each time L > such that pt ())oo)) = >x (io,))dl. dd.
2

---

Published as a conference paper at ICLR 2024
Event type
Intensity 1
Event type
Intensity
time
Event Sequence
(41,k)
(12, k2)
(t3, kz)
(f k4)
Figure 2: Drawing event stream from neural TPP. The model reads the sequence of past events (pelygons) arrive hidden (blue). That state determines the future "intensities" of the types of events-that is, their time-varying instantaneous probabilities. The intensity functions are continuous parametric curves (solid lines) determined by the most recent model state. Events will update the future intensity curves as they occur.
Neural TPPs. A neural TPP model autoregressively generates events after another via neural networks. A schematic example is shown in Figure and a detailed description on data samples can be found at the documentation of the repository. For the i-th event (t,,;;), it computes the embedding of the event e; R via an embedding layer and the hidden state h; gets updated conditioned on e, and the previous state h Then one can draw the next event conditioned on the hidden state h;:
4+1, k4t1 ~ ks (h:), h, = Supdate (h;-1, ei);
(1)
where fupdate denotes a recurrent encoder, which could be either RNN (Du et al., 2016; Mei & Eisner, 2017) more expressive attention-based recursion layer (Zhang et al., 2020; Zuo et al., 2020; Yang et al., 2022). A new line research models the evolution of states completely in continuous time:
h;- = h fupdate (h,-, e.) at event time t
between event times
(2)
The state evolution in Equation (2) is generally governed by ordinary differential equation (ODE) (Rubanova et al., 2019). For a broad and fair comparison, in Easy we implement not only recurrent TPPs but also ODE-based continuous-ting state model.
Learning TPPs. Negative log-likelin log-likelihood (NLL) is the default training objective for both classical and neural TPPs. The NLL of TPP given the entire event sequence T[o,
log As. (4 moo,))
de M(o.))
(4)
Derivations of this formula can be found in previous work (Hawkes, 1971; Mei & Eisner, 2017).
3 THE BENCHMARKING PIPELINE
Figure presents the open benchmarking pipeline implemented in EasyTPP. This pipeline will facil- itate future research in this area and help promote reproducible work. In this section, introduce and discuss each of the key components.
Data Preprocessing. Following common practices, we split the set of sequences into disjoint train, validation, and test set. Normally, the input is fed batch-wise into the model; there may exist quences of events that have unequal-length lengths into the model, pad all sequences to the same length, then the "sequence _mask" tensor to identify which event tokens are padding. As we implemented several variants of attention- based TPPs, also generated the "attention mask" to mask all the future positions each event
the same batch. To feed the sequences of varying
3

---

Published as a conference paper at ICLR 2024
Data Preprocess
Evaluation
Open Datasets
Prepare the data, such as packing and masking
Models
Trainer
Thinning Sampler
Measure the performance of the trained model
Hyper Tuning
Search the parameter space
Figure 3: An open benchmarking pipeline using Easy
to avoid "pecking into the future"". The padding and masking mechanism the same as that used in NLP filed. See Appendix C.I for detailed explanation on sequence padding and masking in EPP
Model Implementation. Our Easy library provides a suite of modules, and one could eas- ily build complex models by composing these modules. Specifically, implemented the models (see section 5.1) evaluated in this paper with our suite of modules (e.g., continuous-time LSTM, continuous-tmee attention). Moreover, some modules model-agnostic methods for training and inference, which will further speed up the development speed of future methodology research. Be- low are two signature examples:
compute_loge Joglikliogll (function), which calculates log-likelln leeliikl of model given data, It is non-trivial to correctly implement it due to the integral term of log-likelihood in Equation (4), and we have found errors in popular implementations.
EventSampler (class), which draws events from a given point process via the thinning algorithm. The thimning algorithm is commonly used in inference but it is non-trivial to implement (and rare to see) an efficient and batched version. Our efficient and batched version (which we took great efforts implement) will be useful for nearly all intensity- based event sequence models.
Training. We estimate the model parameters locally maximizing the NLL in Equation (4) with any stochastic gradient method. Note that computing the NLL can be challenging due to the presence of the integral the second term in Equation (4). In EasyTPP, by default, we approximate the integral by Monte-Carlo estimation to compute the overall NLL (see Appendix B. Nonethe- less, EasyTPP also incorporates intensity-free models (Shchur al., 2020), whose objectives are easier to compute.
Sampling. Given the learned parameters, we apply the minimum Bayes risk (MBR) principle to predict the time and type with the lowest expected loss. recipe can be found in Appendix B.2. Note that other methods exist for predicting with TPP, such adding MLP layer to directly output the time and type prediction (Zuo et al., 2020; Zhang et al., 2020). However, the primary focus of this benchmark is generative models of event sequences, for which the principal approach the thinning algorithm (Ogata, 1988). In EasyTPP, we implemented a batched version of thinning algorithm, which is then used evaluate the TPPs our experiments.
Hyperparameter Tuning. Our EasyTPP benchmark provides programs for automatic hyperpa- rameter tuning. In addition to classical grid search, we also integrate Optuna (Akiba et al., 2019) in our framework adaptively prune the search grid.
4 SOFTWARE INTERFACE
The purpose of building EasyTPP is to provide simple and standardized framework to allow users to apply different state-of-thh-ar (SOTA) TPPs to any datasets as they would like to. For researchers, TPs provides implementation interface to integrate new recourse methods in easy-to-use way, which allows them to compare their method to already existing methods. See the pseudo implementation in listing 1. For industrial practitioners, the availability of benchmarking code helps them easily assess the applicability of TPP models for their own problems. See an example of running an existed model in EasyTPP in listing 2. The full documentation of software interfaces
4

---

Published as a conference paper at ICLR 2024
be found repo. For more details of software architecture, please introduction part of online documentation and Appendix A as well.
5
EXPERIMENTAL EVALUATION
5.1
EXPERIMENTAL. SETUP
We comprehensively evaluate 9 models in our benchmark, which include the classical Multivari- ate Hawkes Process (MHP) with exponential kernel, (seeAppendix B for more details), and widely-cited state-of-the-ar neural models:
Two RNN-based models: Recurrent marked temporal point process (RMTPP) (Du et al., 2016) and neural Hawkes Process (NHP) (Mei & Eisner, 2017).
Three attention based models: self-attentive Hawkes pocess (SAHP) (Zhang et al., 2020), trans- former Hawkes process (THP) (Zuo et al., 2020), attentive neural Hawkes process (AttNHP) (Yang et al., 2022).
One TPP with the fully neural network based intensity: FullyNN (Omi al., 2019).
One intensity-free model IFTPP (Shchur al., 2020).
One TPP with the hidden state evolution governed by neural ODE: ODETPP. is simplified version of the TPP proposed by Chen et al. () by removing the spatial component...
from easy-tpp. model. torcttrrroooooo model. torch basemodel import TorchBaseModel.
import argparse
from easy-tpp. fasy.tpp.config config factory contig Config
import
Custon TPP implementations need to inherit from the BaseModel interface class NewModel (Toocccc((Trr))BBeeeodee)) def _init_ model.config): super (NewModel, solff...nitt.f model.config)
from easy runner import Runner
def main();
parser argparse argparse. Arguuentrrrr
parser ausser.addare ("-cooogdr" type=str, required-True, help='Dir of
Forward along the sequence, output the states intensities event times
config to train evaluate model .²)
def forward batch);
parser argumentttt (x-exxxeimeent..
type-str, required-True, help='
return states
Compute the loglikelihood loss def loss, batch);
Experiment id in the config file.*)
args parser parser.parse parse parse.aggs((
return loglike
# Build up the configuation config Config. www.dd.rrm.mymmm.ff....... congg.confi,,, args. exgeeexperimet).
Compute the intensities given sampling times, used by Thinning sampler def
# Intialize the runner of pipelime model Runner. build fuild_from.conff www..coooiigcoonoiig
compute intensities at sample times (self batch, sample times, **kaargs):
# Start running model_runnee run ()
intemsities
Listing 1: Pseudo implementation of customizing a TPP model in PyTorch using EasyTPP.
main_..
main
Listing 2: Example implementation of running TPP model using EasyTPP.
5

---

Published as a conference paper at ICLR 2024
We conduct experiments on 1 synthetic and 5 real-world datasets from popular works that contain diverse characteristics in terms of their application domains and temporal statistics (see Table 2):
Synthetie. This dataset contains synthetic event sequences from univariate Hawkes process sampled using Tick (Bacry et al., 2017) whose conditional intensity function is defined by X(t) = f + Li,c ag exp (t tz)) with μز 0.2, a 0.8, 3 = 1.0. We randomly sampled disjoint train, dev, and test sets with 1200, 200 and 400 sequences.
Amazon(Ni, 2018). This dataset includes time-stamped user product reviews behavior from Jan- uary. 2008 to October, 2018. Each user has a sequence of produce review events with each event containing the timestamp and category of the reviewed product, with each category corresponding to an event type. We work on subset 5200 most active users with an average sequence length of 70 and then end up with K = 16 event types.
Retweet (Ke Zhou & Song, 2013). This dataset contains time-stamped user retweet event quences. The events are categorized into K = 3 types: retweets by "small," "medium" and "large" users. Small users have fewer than 120 followers, medium users have fewer than 1363, and the rest are large users. We work on subset of 5200 active users with an average sequence length of 70.
Taxi (Whong, 2014). This dataset tracks the time-stamped taxi pick-up and drop-off events across the five boroughs of the New York City; each (borough, pick-up drop-off) combination defines an event type, so there are = 10 event types in total. We work on a randomly sampled subset of 2000 drivers with average sequence length of 39.
Taobao (Xue et al., 2022). This dataset contains time-stampe stme-stamped user click behaviors on Taobao shopping pages from November 25 to December 03, 2017. Each user has sequence of item click events with each event containing the timestamp and the category of the item. The categories of all items are first ranked by frequencies and the top 19 are kept while the rest are merged into one category, with each category corresponding to an event type. We work on a subset of 4800 most active users with an average sequence length of 150 and then end up with K 20 event types.
StefkOverflow (Leskovec & Krevi, 2014). This dataset has two years of user awards on website: each user received sequence of badges and there are K = 22 different kinds of badges in total. We work on a subset of 2200 active users with an average sequence length of 65.
Evaluation Protocol. We keep the model architectures as the original implementations in their papers. For a fair comparison, we use the same training procedure for all the models: we used Adam (Kingma & Ba, 2015) with the default parameters, biases initialized with zeros, no learning rate decay, the same maximum number of training epochs, and early stopping criterion (based on log-like ligeiike on the held-out dev set) for all models.
We mainly examine the models in two standard scenarios.
Goodness-of-fit we fit the models on the train set and measure the log bog-probability they assign to the held-out data.
Next-event prediction: we use the minimum Bayes risk (MBR) principle to predict the next event time given only the preceding events, as well as its type given both its true time and the preceding events. We evaluate the time and type prediction by RMSE and error rate, respectively.
In addition, we propose a new evaluation task: the long-horizon prediction. Given the prefix of each held-out sequence T[o,T), autoregressively predict the next events in future horizon (rrr"" It is evaluated by measuring the optimal transport distance (OTD), type of edit distance for event sequences (Mei et al., 2019), between the prediction &(T,T") and ground truth F(r,T1 As pointed out by Xue et al. (2022), long-horizon prediction of event sequences is essential various real-world domains, and this task provides new insight into the predictive performance of the models.
is worth noting that the original version of the FullyNN model does not support multi-type event sequences. Therefore it is excluded from the type prediction task.

---

Published as a conference paper at ICLR 2024
01
4.4
Logiitee
Logite
2.5
4.
-12
Mec
accose
crerry
Fupett
clegp
Per
auter
oscrr
rubeta
gropp
astver cserre
(a) Synthetic I-D Hawkes
(b) Retweet
(c) Taxi
Figure 4: Performance of all the methods the goodness-of-fe task synthetic Hawkes, Retweet, and Taxi
data. A higher score better. All methods are implemented PyTorch.
MODEL
METRICS (TIME RMSE / TYPE ERROR RATE)
AMAZON
RETWEET
TAXI
TAOBAO
STACKOVERFLOW
MHP
0.635/75.9%%
22.92/55.779 75%
0.382/9.53%
0.539/68. 1%
1.388/65.00
0.005/0.005
0.212/0.004
0.002/0.0004
0.004/0.004
0.011/0.005
RMTPP
0.620/68.11 1% 22.31/44.
0.371 19.51%
0.531 0.531/55.89 /55.8%
1.376/37.33
0.005/0.006
0.209/0.003
0.003/0.0003
0.005/0.004
0.018/0.005
NHP
0.621/67.1%
21.9040.0%%
0.369/8.500
0.531/54.29%
1.372/55.0%
0.005/0.006
/.184/0.002
0.003/0.0005
0.005/0.006
0.011/00006
SAHP
0.619/67.7%
22.40/41.66
0.372/9.75%.
0.532/54.66
1.375/56.19
0.005/0.006
0.301/0.000
0.003/0.0008
0.004/0.002
0.013/0.005
THP
0.621/66.1% 1%
22.01/4 5415%
0.370/8.68%
0.531/53.6%
1.374/55.0%
0.003/0.007
0.188/00003
0.003/0.0006
0.003/0.004
0.021/0.006
ATTNHP
0.621/65.39)
22.19/40 /0.10%.% 1%
0.371/8.71%
0.529/53.7 7%
1.372/55.29
0.005/0.006
0.180/0.003
0.003/0.0004
0.005/0.001
0.019/0.003
ODETРP
0.620/65.899 22.48//3.229
0.371/10.54' 54.%
0.533/55.449
L.374/56.89
0.006/0.008 (..75/00.004
0.003/0.0008
0.005/0.007
0.022/0.004
FULLYNN
0.615/N.A.
2L.92/N.A.
0.373/N.A.
0.529/N.A
L.375/N.A.
0.005/N.A.
0.159/N.A.
0.003/N.A.
0.005N.A.
0.015/N.A.
IFTPP
0.618/67.5%
22.18/39.79
0.377/8.56% 0.003/0.006
0.531/55.4%% 0.005/0.004
1.373/55.19
0.005/0.007
/0.00
0.010/0.005
Table I: Performance of all the methods on next-event's time prediction and next-event's type prediction on
five real datasets (for each model, first row corresponds to the metrics value while second row corresponds to the standard deviation). Lower not applicable for the type prediction tasks.
better. All methods implemented in PyTorch. As clarified, FullyNN
5.2 RESULTS AND ANALYSIS
Figure 4 (see Table for exact numbers in the figure) reports the log-likelihood three held-out datasets for all the methods. We find IFTPP outperforms all the competitors because it evaluates the log-likenn ligeliiod in close form while the others (RMTPP, NHP, THP, AttHHP, ODETPP) com- pute the intensity function via Monte Carlo integration, causing numerical approximation errors. FullyNN method, which also exactly computes the log- likelihood, has worse fitness than other neu- ral competitors. As Shehur et al. (2020) points out, the PDF of FullyNN does not integrate to due to a suboptimal choice of the network architecture, therefore causing a negative impact on the performance.
Table reports the time and type prediction results on five real datasets. We find there no single winner against all the other methods. Attention-baut methods (SAHP, THP, AttHHP) generally perform better than or close to non-attention methods (RMTPP, NHP, ODETPP.FullyNN and IFTPP) on Amazon, Taobao, and Stackoverflow, while NHP is the winner both Retweet and Taxi. We

---

Published as a conference paper at ICLR 2024
11.5
w40
aumre
tre
ny
orerme
Anteo
imm
Figure 5: Long horizon prediction on Retwert data: Figure Long horizon prediction on Taxi data: left left (avg prediction horizon 5 events) vs. right (ave (avg prediction horizon 5 events) vs. right (avg pre- prediction horizon 10 events).
diction horizon 10 events).
see that NHP a comparably strong baseline with iitention-based TPPs. This not too surprising because similar results have been reported in previous studies (Yang et al., 2022).
Not surprisingly, the performance of the classical model MHP worse than the neural models across most of the evaluation tasks, consistent with the previous findings that neural TPPs have demonstrated to be more effective than classical models at fitting data and making predictions.
Please Appendix D.3 for more results (e.g., actual numbers of figures) on all the datasets. With growing number of TPP methods proposed, we will continuously expand the catalog models and datasets and actively update the benchmark in repository.
Analysis-I: Long Horizon Prediction. We evaluate the long horizon prediction task on Retweet and Taxi datasets. On both datasets, we set the prediction horizon to be the one that approximately has 5 and 10 events, respectively. Shown in Figure 5 and Figure 6, we find that AuNHP and THP are two co- co-winners on Retweet and THP is a single winner on Taxi. Nonetheless, the margin of the winner over the competitors is small. The exact numbers shown in these two figures could be found in Table in Appendix D.3. Because these models are autoregressive and locally normalized, they all exposed to cascading errors. To fix this issue, one could resort other kinds of models such as a hybridly normalized model (Xue et al., 2022), which is out of the scope of the paper.
Analysis-II: Models with Different Frameworks PaanrrkssPTooohhee vs. TensorFlow. Researchers nor- mally implement their experiments and models for specific MI. frameworks. For example, recently proposed methods are mostly restricted to PyTorc and are not applicable to TensorFlow models. As explained in Section 4, facilitate the use of TPPs, we implement two equivalent sets of methods in Py PyTorch and TensorFlow. Table 6 in Appendix D.3 shows the relative difference between the results of Torch and TensorFlow implementations are all within [.55%, 1.5%]. To conclude, the two sets of models produce similar performance in terms of predictive ability.
6 FUTURE RESEARCH OPPORTUNITIES
We summarize thoughts on future research opportunities inspired by benchmarking results.
Most importantly, the results seem to be signaling that we should think beyond architectural design. For the past decade, this area has been focusing on developing new architectures, but the perfor- mance of new models on the standard datasets seem to be saturating. Notably, all the best to-date models make poor predictions on time of future events. Moreover, on type prediction, attention- based model (Zuo et al., 2020; Zhang et al., 2020; Yang et al., 2022) only outperform other archi- tectures by a small margin. Looking into the future, we advocate for few new research directions that may bring significant contributions to the field.
The first to build foundation models for event sequence modeling. The previous model-building work all learns data-specific weights, and does not test the transferring capabilities of the learned models. Inspired by the emergence of foundation models in other research areas, we think it will be beneficial to explore the possibility to build foundation models for event sequences. Conceptually, learning from a large corpus of diverse datasets like how GPTs (Nakano et al., 2021) learn by reading open web text-has great potential to improve the model performance and generalization beyond what could be achieved in the current in-domain in-data learning paradigm. Our library can facilitate exploration in this direction since we unify the data formats and provide easy-to-use
8

---

Published as a conference paper at ICLR 2024
interface that users seamlessly plug and play any set of datasets. Challenges in this direction arise as different datasets tend to have disjoint sets event types and different scales of time units.
The second is to go beyond event data itself and utilize external sources to enhance event sequence modeling. Sceing the performance saturation of the models, we are inspired to think whether the performance has been bounded by the intrinsic signal-to-noise ratio of the event sequence data. Therefore, it seems natural and beneficial to explore the utilization of other information sources, which include but are not limited to: (i) sensor data such satellite images and radiosondes signals; (ii) structured and unstructured knowledge bases (e.g., databases, Wikipedia); (iii) large pretrained models such as ChatGPT (Brown et al., 2020), may assist event sequence models in improving their prediction accuracies. Concurrent with this work, Shi et al. (2023) has made an early step in this direction.
The third is to go beyond observational data and embed event sequence models into real-world interventions (Qu et al., 2023). With interventional feedback from the real world, an event sequence model would have the potential to learn real causal dynamics of the world, which may significantly improve prediction accuracy.
All the aforementioned directions open up research opportunities for technical innovations.
RELATED WORK
Temporal Point Processes. Over recent years, a large variety of RNN-based TPPs have been proposed (Du al., 2016; Mei & Eisner, 2017; Xiao al., 2017: Omi al., 2019; Shchur al., 2020; Mei et al., 2020; Boyd et al., 2020). Models of this kind enjoy continuous state spaces and flexible transition functions, thus achieving superior performance on many real datasets, compared to the classical Hawkes process (Hawkes, 1971). To properly capture the long-range dependency in the sequence, the attention and transformer techniques (Vaswani et al., 2017) have been adapted to TPPs (Zuo et al., 2020; Zhang et al., 2020; Yang al., 2022: Wen et al., 2023) and makes further improvements on predictive performance. There has also been research in creative ways of training temporal point processes, such as in meta learning framework (Bae al., 2023). Despite significant progress made in academia, the existing studies usually perform model evaluations and comparisons in an ad-hoc manner, e.g., by using different experimental settings or different ML frameworks. Such conventions not only increase the difficulty in reproducing these methods but also may lead to inconsistent experimental results among them.
The significant attention attracted by TPPs in recent years nat- urally leads to a high demand for an open benchmark to fairly compare against baseline models. While many efforts have been made in the domains of recommender systems (Zhu et al., 2021) and natural language processing (Wang et al., 2019), benchmarking TPPs is an under-explored topic. Tick (Bacry et al., 2017) and pyhawkes are two well-known libraries that focus on statistical learn- ing for classical TPPs, which are not suitable for the SOTA neural models. Poppy (Xu, 2018) is a PyTorch-basse based toolbox for neural TPPs, but it has not been actively maintained since 2021 and has not implemented any recent SOTA methods. To the best of our knowledge, EasyTPP is the first package that provides open benchmarking for popular neural TPPs.
Open Benchmarking on TPPs.
CONCLUSION
In this work, we presented EasyTPP, an open and comprehensive benchmark for standardized and transparent comparison of TPP models. The benchmark hosts diversity of datasets and models. In addition, it provides a user-friendly interface and a rich library, with which one could easily integrate new datasets and implement new models. With these features, EasyTPP has the potential to significantly facilitate future research in the area of event sequence modeling.
hhttps://g htm. http://g..hhh...ooommmmemee/e/ae inderman/yyyhaweeees

---

Published as a conference paper at ICLR 2024
REFERENCES
Abadi, M.. Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., al. Tensorflow: system for large-scale machine learning. In OSDI' volume 16, pp. 265-283, 2016.
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparame- ter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
Bacry, E., Bompaire, M., Gaiffas, S., and Poulsen, S. tick: Python library for statistical learning, with a particular emphasis on time-dependent modeling. ArXiv e-prints, 2017.
Bae, W., Ahmed, M. O., Tung, F., and Oliveira, G. L. Meta temporal point processes. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Bai, T., Zou, L.,, Zhao, W. X., Du, P., Liu, W., Nie, J.-Y., and Wen, J.-R. Ctrec: A long-short demands evolution model for continuous-time recommendation. In Proceedings of the 42nd In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval, SI- GIR'19, pp. 675-6644 New York, NY. USA, 2019. Association for Computing Machinery. ISBN 9781450361729.
Bao, J. and Zhang, Y. Time-aware recommender system via continuous-time modeling. In Pro- ceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, pp. 2872-2876, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384469.
Boyd, A., Bamler, R., Mandt, S., and Smyth, User-dependent neural sequence models for continuous-time event data. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chen, R. T., Amos, B., and Nickel, M. Neural spatio-temporal point processes. ICLR, 2021.
Chu, 7., Hao, H., Ouyang, X., Wang, S., Wang, Y., Shen, Y., Gu, J., Cui, Q. Li, L.,, Xue, S., et Leveraging large language models for pre-trained recommender systems. arXiv preprint arXiv:2308. 2023.
Cramér, H. Historical review of Filip Lundberg's works on risk theory. Scandinavian Actuarial Journal, 1969(uu33) 1969.
Daley, D. J. and Vere-Jones, D. An Introduction to the Theory of Point Processes, Volume II: General Theory and Structure. Springer. 2007.
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguezz M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Fan, Z., Liu, Z., Zhang, J., Xiong, Y., Zheng, L., and Yu, S. Continuous-time sequential rec- ommendation with temporal graph collaborative transformer. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, pp. 433-442, New York, NY, USA. 2021. Association for Computing Machinery. ISBN 9781450384469,
Hasbrouck, J. Measuring the information content of stock trades. The Journal of Finance, 46(1): 179-207, 1991.
Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 1971.
Hemanneez, S., Álvarez, P.. Fabra, J., and Ezpeleta, J. Analysis of users' behavior in structured e-commerce websites. IEEE Access, 5:11941-11958, 2017.
10

---

Published as a conference paper at ICLR 2024
Jin, Z., Guo, S., Chen, N., Weiskopf, D., Gotz, D., and Cao, N. Visual causality analysis of event sequence data. IEEE transactions on visualization and computer graphics, 27(2):1343-1352, 2020.
Ke Zhou, H. Z. and Song. L. Learning triggering kernels for multi limensonaa hawkes processes. In Proceedings of the International Conference on Machine Learning (ICML), 2013.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2015.
Leskovec, J. and Krevi, A. SNAP Datasets: Stanford large network dataset collection, 2014.
Lewis, P. A. and Shedler, G. S. Simulation of nonhomogeneoos Poisson processes by thinning. Naval Research Logistics Quarterly, 1979.
Liniger, T. J. Multivariate Hawkes processes. Diss., Eidgenössische Technische Hochschule ETH Zürich, Nr. 18403, 2009.
Mei, H. and Eisner, J. The neural Hawkes process: neurally self-mooulating multivariate point process. In Advances in Neural Information Processing Systems (NeurlPS), 2017.
Mei, H., Qin, G., and Eisner, J. Imputing missing events in continuous-timg event streams. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
Mei, H., Qin, G., Xu, M., and Eisner, J. Neural Datalog through time: Informed temporal modeling via logical specification. In Proceedings of the International Conference Machine Learning (ICML), 2020.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C.,, Hesse, C., Jain, S., Kosaraju, V., Saunders, W. et al. Webgpt: Browser-assisted question-answering answering with human feedback. arXiv preprint arXiv:2112.093322 2021.
Ni, J. Amazon review data, 2018.
Ogata, Y. Statistical models for earthquake occurrences and residual analysis for point processes. J. Am. Stat. Assoc, 83(401):9 27, 1988.
Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L.., al. Pytorch: An imperative style, high-perffrmance deep learning library. arXiv preprint arXiv:1912.01700 2019.
Qu, C., Tan, X., Xue, S., Shi, X., Zhang, J., and Mei, H. Bellman meets hawkes: Model-based reinforcement learning via temporal point processes. Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Rubanova, Y., Chen, R. T., and Duvenaud, D. K. Latent ordinary differential equations for arregularyysamplee time series, Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Shchur, O., Biloš, M., and Günnemann, S. Intensity-free learning of temporal point processes. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Shi, X., Xue, S., Wang, K., Zhou, F., Zhang, Y., Zhou, J., Tan, C., and Mei, Language models can improve event prediction by few-shot abductive reasoning, arXiv preprint arXiv:2305.16646 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polo- sukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR, 2019.
11

---

Published as a conference paper at ICLR 2024
Wen, Q.. Zhou, T., Zhang, C., Chen, W., Ma, Z.,, Yan, J., and Sun, L. Transformers in time series; A survey. In International Joint Conference on Artificial Intelligence((ICAA), 2023.
Whong, C. FOILing NYC's taxi trip data, 2014.
Williams, A.. Degleris, A., Wang, Y., and Linderman, S. Point process models for sequence detec- tion in high-dimensional neural spike trains. Advances in neural information processing systems, 33:14350 13:14350-14361, 2020.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer. S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics.
Xiao, S., Yan, J., Yang, X., Zha, H., and Chu, S. Modeling the intensity function of point process via recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
Xu, H. Poppy: A point process toolbox based pytorch. arXiv preprint arXiv:1810.10122 2018
Xue, S., Shi, X., Zhang, Y. J., and Mei, H. Hypro: A hybridly normalized probabilistic model for long-horizon prediction event sequences. In Advances in Neural Information Processing Systems (NeurIPS). 2022.
Yang, C.. Mei, H., and Eisner, J. Transformer embeddings of irregularly spaced events and their participants. In Proceedings the the International Conference on Learning Representations (ICLR), 2022.
Zhang, Q., Lipani, A., Kirnap, O., and Yilmaz, Self-attentive Hawkes process. In Proceedings of the International Conference Machine Learning (ICML), 2020.
Zhou, T., Niu, P., Wang, X., Sun, L., and Jin, R. One fits all: Power general time series analysis by pretrained Im. arXiv preprint arXiv:2302. 1eee119939 1939, 2023.
Zhu, J., Liu, J., Yang, S., Zhang, Q., and He, X. Open benchmarking for click-througe rate predic- tion. In Demartini, G., Zuccon, G., Culpepper, J. S., Huang, Z.,, and Tong, H. (eds.), CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 5, 2021, pp. 2759-2769. ACM, 2021.
Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer Hawkes process. In International Conference Machine Learning, pp. 11692-11177 11702. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2024
HighLavel Config
RunnerContig
HPOConlg
Pipeline Runner
HPORunner
Confgg
DataConfig
ModelConfigg
EvaiConfig
HPOConfig
Wrapper
Tensortlow/irapper
PyTorohWrrapper
Module
DetaLoader
Models
Matrics
HPO
Preprocess
Hyper Tuning
Open Datasets
Model
Trainer
Thinning Sampler
portormance
Swarch
padkingg
Custom Implementation
8 SOTA
TPPs
Figure 7: Architecture of the EasyTPP library. The dashed arrows show the different implementation possi- bilities, either to use pre-defined SOTA TPP models or provide a custom implementation. All dependencies between the configurations and modules are visualized by solid arrows with additional descriptions.
Appendices
A EASYTPP'S SOFTWARE INTERFACE DETAILS
In this section, we describe the architecture of our open-source benchmarking software Easy in more detail and provide examples of different use cases and their implementation.
A.1 HIGH LEVEL SOFTWARE ARCHITECTURE
The purpose of building EPsyTTP is to provide a simple and standardized framework to allow users to apply different state-of-the-arn (SOTA) TPPs to arbitrary data For researchers, Easy pro- vides an implementation interface to integrate new recourse methods in an easy-to-use way, which allows them compare their method already existing methods. For industrial practitioners, the availability of benchmarking code helps them easily assess the applicability of TPP models for their own problems.
A high level visualization of the Easy TPs'SPPPs software architecture is depicted in Figure 7. Data Preprocess component provides a common way to access the event data across the software and maintains information about the features. For the Model component, the library provides the possi- bility to use existing methods or extend the users' custom methods and implementations. wrapper encapsulates the black-box models along with the trainer and sampler. The primary purpose of the wrapper is to provide a common interface to easily fit in the training and evaluation pipeline, inde- pendently of their framework (e.g., PyTorch TensorFlow). See Appendix A.2 and Appendix A.3 for details. The rumning of the pipeline is parameterized by the configuration class RunnerConfig (without hyper-paramete tuning) and HPOConfig (with hyper-parameter tuning).
A.2 WHY DOES EASYTPP SUPPORT BOTH TENSORFLOW AND PYTORCH
FeoworFlow and PyTorch are the two most popular Deep Learning (DL) frameworks today. PyTorch has a reputation for being a ressarch--ocuseed framework, and indeed, most of the authors have imple- mented TPPs in Py PoTrrh, which are used as references by EasyTPP. On the other hand, TensorFlow has been widely used in real world applications. For example, Microsoft recommender, NVIDIA
https::/ wwwp:htwww.occcccceeecoeeceo http://thtttt..ooooooooooooe/eee
13

---

Published as a conference paper at ICLR 2024
TeosorFlow Wrapper.
Data Preprocess
Thinning
Evaluation
Prepare the data, such packing masking
Messure the performance model
DataLoader
Model
Trainer
Thinning
PyTorch Wräpper
Figure Illustration of TensorFlow and PyTorch Wrappers in the EasyTPP library.
Merlin and Alibaba ResyRec* are well-known industrial user modeling systems with TensorFlow as the backend. In recent works, TPPs have been introduced better capture the evolution of the user preference in continuous-tiig time (Bao & Zhang, 2021; Fan et al., 2021; Bai et al.. 2019). To support the use of TPPs by industrial practitioners, we implement an equivalent As result, EasyTPP not only helps researchers analyze the strengths and bottlenecks of existing models, but also facilitates the deployment of TPPs industrial applications.
of TPPs in TensorFlow.
A.3 HOw DOES EASYTPP SUPPORT BOTH PYTORCH AND TENSORFLOW
We implement two two equivalent sets of data loaders, models, trainers, thinning samplers in TensorFlow and PyTorch, respectively, then use wrappers to encapsulate them so that they have the same API exposed in the whole training and evaluation pipeline. See Figure 8.
A.4 EASYTPP FOR RESEARCHERS
The research groups inherit from the BaseModel implement their own method in EasyTPP. This opens up way of standardized and consistent comparisons between different TPPs when exploring new models.
Specifically, if we want to customize a TPP in Py PyTorh, we need to initialize the model by inheriting the class Boseel:seModel:
from easy.tpp ttp.ppppmolll...... model. torch model. torch basemodel import TorchBaseModel
Custom Torch TPP implementations need to inherit from the TorchBaseModel interface class NeuModel (TorchBaseeeddde)) def (seit_ model_config): super (NewModel, self) inft (mode_loomod.
# Forward along the sequence, output the states / intensities at the event times
def forward(self,, batch);
return states
Compute the loglikelihood loss def loglike (oss,, batch):
return loglike
Compute the intensities given sampling times
# Used in the Thinning sampler def
(seef, batch, cample_times, **kwargs):
return intensities
'https://developer www.dia..comnvviiiiaeiiiec
com com/al alib wwwbabaaaaaaaa babaakasyRe..
14

---

Published as a conference paper at ICLR 2024
Listing 3: Pseudo implementation of customizing TPP model in PyTorch using Easy
Equivalent, if we want to customize a TPP in TensorFlow, we need to initialize the model by inher- iting the class TfBaseModel:
from wtpppppppe......mmmmmmmmmmmm.. torch basemodel import TfBaseModel
Custom Torch TPP implementations need to
# inherit from the TorchBaseModel interface
class NeuModel (TYBaseModel):
def _init_ :odel_config):
super (NexModee, self) init.. (mmdttt_modde..ooffff.
# Forward along the sequence, output the states / intensities at the event times
def forward(self, batch);
return states
Compute the loglikelihood loss def losa batch):
return loglike
Compute the intensities at given sampling times
# Used in the Thinning sampler
def compute intensities sample times, batch, sample times, **kwarge)
return intensities
Listing 4: Pseudo implementation of customizing TPP model in TensorFlow using Easy TPP.
A.5 EASYTPP AS A MODELING LIBRARY
common usage of the package is to train and evaluate some standard TPPs. This can be done by loading Back-box-moddl and data sets from our provided datasets, by user-defined models and datasets via integration with the defined interfaces. Listing 5 shows an implementation example of a simple use-case, fitting TPP model method a preprocessed dataset from our library.
import argparse
from easy-tpp cantor import Config
from easy_tpp runner import Runner
def main():
parser argparse. (rgumentParser()
parser addser.ad add.urgmmnntt (o-pieeetttccoo--teee ('-config.dirr,
type=str,
required-False,
help='Dir configuration yaml to train and evaluate the
model.") 2)
parser add argument
(t ("-exerrmmntt.idl
type*str,
required=False,
default=' Intneesrennnssttfre.eraaan'
help='Experines id in the config file.")
15

---

Published as a conference paper at ICLR 2024
args parser weree.parrseee parse-arga
Build up the configuation for the runner config = Contig.build experiment.id)
file (aggeaaag.cooniig...ll dirr experiment.id-ary
Intialize the runner for the pipeline model runner Runner. build from config(config)
Start running model rwnner.uunnrrruu run
if
main main ()
name
=
Listing 5: Example implementation of running TPP model using EasyTPP.
B MODEL IMPLEMENTATION DETAILS
We have implemented the following TPPs
Multivariate Hawkes Process (MHP). We implemented it using Tick (Bacry al., 2017) with an exponential kernel with fixed decays: the base intensity is set to 0.2 for all dimensions while the decay matrix is be a unit matrix plus small noise. See online document for more details.
Recurrent marked temporal point process (RMTPP) (Du et al., 2016). We implemented both the Tensorflow and PyTorch version of RMTPP by our own.
Neural Hawkes process (NHP) (Mei & Eisner, 2017) and Attentive neural Hawkes process (AUNHP) (Yang et al., 2022). The Pytorch implementation mostly comes from the code from the public GitHub repository at https: //tp::///ittt htttbb......tg.g//g con/ yangalanl an122/ aaaaannppaaaaaa andtt ( (Yang et al., 2022) with MIT License. We developed the Tensorflow version of NHP and ttNHP by our own.
Self-attentive Hawkes process (SAHP) (Zhang et al., 2020) and transformer Hawkes process (THP) (Zuo et al., 2020). We rewrote the PyTorch versions SAHP and THP based on the public Github repository at https: http::://githaa com/ yangalan12 aangaaannn21 anng-andtt (Yang et al., 2022) with MIT License. We developed the Tensorflow versions of the two models by our own.
Intensity-free free TPP (IFTPP) (Shchur et al., 2020). The Pytorch implementation mostly comes from the code from the public GitHub repository at tps: / //gtthub.b... , com/ shchur/ifl-tee (Shchur et al.. 2020) with MIT License. We implemented Tensorflow version by our own.
Fully network based TPP (FullyNNN (Omi al.. 2019). We rewrote both the Tensorflow and Py PoTch versions of the model faithfully based on the author's code at nttps: / github. con/ omi takahi i ro/ /Nuura
Please not that the model only considers the number of the types to be one, i.e., the sequence's K 1.
ODE-based TPP (ODETPP) (Chen et al., 2021). We implement TPP model, in both Tensorflow and PyTorch with a continuous time state evolution governed by neural ODE. It is basically the spatial-temporaa point process (Chen al., 2021) without the spatial component.
B. LIKELIHOOD COMPUTATION DETAILS
In this section, we discuss the implementation details of NLL computation in Equation (4).
The integral term in Equation (4) computed using the Monte Carlo approximation given by Mei & Eisner (2017, Algorithm 1), which samples times t. This yields unbiased stochastic gradient. For the number of Monte Carlo samples, we follow the practice of Mei & Eisner (2017): namely, at training time, match the number of samples the number of observed events at training time, reasonable and fast choice, but to estimate log-likelln likelihoddood when tuning hyperparameters or reporting final results, take 10 times many samples.
16

---

Published as a conference paper at ICLR 2024
At each sampled time t. the Monte Carlo method still requires summation over all events obtain A(t). This summation can be expensive when there are many event types. This is not serious problem for our Eas implementation since it can leverage GPU parallelism.
B.2 NEXT EVENT PREDICTION
is possible to sample event sequences exactly from any intensity-base based model in EasyTPP, us- ing the thimming algorithm that is traditionally used for autoregressive point processes (Lewis & Shedler, 1979; Liniger, 2009). In general, to apply the thinning algorithm to sample the next event at time 2 lo. necessary have an upper bound on (Qe(t) 1 (oo 00)) for each event type explicit construction for the NHP (or AuNHP) model was given by Mei & Eisner (2017, Appendix B.3).
Section 3 includes task-based evaluation where we try predict the time and type of just the next event. More precisely, for each event in each held-out sequence, we attempt to predict time given only the preceding events, as well as its type given both its true time and the preceding events.
We evaluate the time prediction with average 1-2 loss (yielding a pont-mean-squrree error, or RMSE) and evaluate the argument prediction with average 0-1 loss (yielding error rate).
Following Mei & Eisner (2017), we use the minimum Bayes risk (MBR) principle to predict the time and type with the lowest expected loss. For completeness, we repeat the general recipe in this section.
For the i-th event, its time t, has density (t) X(t) exp( Sú.,, X(t')dt). We choose Ji_, (t)dt as the time prediction because it has the lowest expected L2 loss. The integral can be estimated using i.i.d. samples of t; drawn from (t( by the thinning algorithm.
Given the next event time t, we choose the most probable type arg max, (tit)) as the type predic- tion because it minimizes expected 0-1 loss.
B.3 LONG HORIZON PREDICTION
The TPP models are typically autoregressive: predicting each future event is conditioned on all the previously predicted events. Following the approach in (Xue et al., 2022), we set up prediction horizon and OTD measure the divergence between the ground truth sequence and the predicted sequence within the horizon. For more details about the setup and evaluation protocol, please see Section 5 in Xue et al. (2022).
C DATASET DETAILS
To comprchensively evaluate the models, preprocessed one synthetic and five real-world datasets from widely-cited works that contain diverse characteristics in terms of their application domains and temporal statistics. All preprocessed datasets available at Google Drive.
Synthetic. This dataset contains synthetic event sequences from univariate Hawkes process sampled using Tick (Bacry et al., 2017) whose conditional intensity function defined by
A(t)= = + Σ a. exp(-((pp tist
with = 0.2, 0.8, 1.0. We randomly sampled disjoint train, dev, and test sets with 1200, 200 and 400 sequences.
Amazon (Ni, 2018). This dataset includes time-stamped user product reviews behavior from Jan- uary, 2008 to October, 2018. Each user has sequence of produce review events with each event containing the timestamp and category of the reviewed product, with each category corresponding an event type. We work on subset of 5200 most active users with an average sequence length of 70 and then end up with K = 16 event types.
Retweet (Ke Zhou Song, 2013). This dataset contains time-stamped user retweet event se- quences. The events categorized into K types: retweets by "small," "medium" and
17

---

Published as a conference paper at ICLR 2024
DATASET
K
# OF EVENT TOKENS
SEQUENCE LENGTH
TRAIN
DEV
TEST
MIN
MEAN
MAX
RETWEET
3
369000
62000
61000
10
41
97
TAOBAO
17
350000
53000
101000
3
51
94
AMAZON
288000
12000
30000
94
TAXI
10
51000
7000
14000
36
37
38
STACKOVERFLOW
22
90000
25000
26000
41
65
101
HAWKES- ID
1
55000
7000
15000
62
79
95
Table 2: Statistics of each dataset.
"large" users. Small users have fewer than 120 followers, medium users have fewer than 1363, and the rest are large users. We work on subset of 5200 most active users with an average sequence length of 70.
Taxi (Whong, 2014). This dataset tracks the time-stamped taxi pick-up and drop-off events across the five boroughs of the New York City: each (borough, pick-up or drop-off) combination defines an event type, there are K = 10 event types in total. We work on a randomly sampled subset 2000 drivers and each driver has a sequence. We randomly sampled disjoint train, dev and test sets with 1400, 200 and 400 sequences.
Taobao (Xue et al., 2022). This dataset contains time-stamped user click behaviors on Taobao shopping pages from November to December 03, 2017. Each user has sequence of item click events with each event containing the timestamp and the category of the item. The categories of all items are first ranked by frequencies and the top 19 are kept while the rest are merged into one category, with each category corresponding to event type. We work
subset of 4800 most active users with an average sequence length of 150 and then end up with K 20 event types.
Stack Overflow (Leskovee & Krevl, 2014). This dataset has two years user awards on a question-ansserring website: each user received sequence of badges and there are K 22 different kinds of badges total. We randomly sampled disjoint train, dev and test sets with 1400, 400 and 400 sequences from the dataset.
Table 2 shows statistics about each dataset mentioned above.
PADDING AND MASKING
Given input sequence 2|0,7] (t), (tr, for each event, firstly embedding layer to map the event type k; each event to a dense vector in higher space; then pass it to the following modules construct the state embedding h;.
Normally, the input fed batch-wise into the model; there may exist sequences of events that have unequal-length in the same batch. We pad them the same length, where the padding mechanism is the same as that in NLP when we pad text sentences.
1. Users can choose to decide whether padding the beginning end, padding to max length of batch or fixed length across the dataset. The function interface and usage are almost the same as those in huggingfacettrasfformer package.
2. The sequence mask is used to denote whether the event is a padded one or real one, which will be used in computation likeliiid and type/rmse evaluation as well.
A more detailed explanation of dataset preprocessing operation can be found at our online document.
D EXPERIMENT DETAILS
D.1 SETUP
Training Details. For TPPs, the main hyperparameters to tune are the hidden dimension of the neural network and the number of layers I. of the attention structure (if applicable). In practice
18

---

Published as a conference paper at ICLR 2024
MODEL
DESCRIPTION
VALUE USED
hädden_size
32
time_embusiee
RMTPP
num layers
2
hädden
64
Sime.emmb_sie
16
NHP
nummlagers hädden_size
2
32
téme_.nme .em............................. size num.layen nea_hheads
16
SAHP
2
2
hädden size time emb size layers reum_heads hadden size
61
16
THP
2
2
32
time emub size
16
ATTNHP
Jayers
heads
fadden size
32
ODETPP
Lime erub size
16
Jayers
2
hidden
32
FULLYNN
time_eeei emab_sze
16
wumn layers hadden_sīze
2
32
INTENSITYFREE
time_emmbuse .embusize
namJayyer
2
Table 3: Descriptions and values of hyperparameters used for models.
the optimal D for a model was usually 16, 32, 64; the optimal was usually 1, To train the parameters for a given generator, we performed early stopping based on log- likelioe on the held-out dev set. The chosen parameters for the main experiments are given in Table 3.
Computation Cost. All the experiments were conducted on server with 256G RAM, a 64 logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz) and one NVIDIA Tesla P100 GPU for acceleration. For training, the batch size is 256 by default. On all the dataset, the training of AuNHP takes most the time (i.e., around 4 hours) while other models take less than hours.
D.2 SANITY CHECK
For each model we reproduced in our library, we ran experiments to ensure that our implementation could match the results in the original paper. We used the same hyperparameters as in original papers; we reran each experiment 5 times and took the average.
In Table 4, we show the relative differences between the implementations on Retweet and Taxi datasets. As we can see, all the relative differences are within (-5%, 5%), indicating that our implementation close to the original.
D.3 MORE RESULTS.
For better visual comparisons, we present the results in Figure 4, Figure 5 and Figure also in the form of tables, see Table 7 and Table 5.
The relative difference between the results Torch and TensorFlow implementations can found in Table 6.
19

---

Published as a conference paper at ICLR 2024
MODEL
METRICS (TIME RMSE
TYPE ERROR RATE)
RETWEET
TAXI
RMTPP
-4.1% 3.5%
2.9% 3.7%
NHP
+3.4%/ / 3.1%
+2.6%
+ 3.5%
SAHP
+1.3% 1.7%
+1.1% 1.2%
THP
+1.3% 1.8%
-1.6%/ + 1.5%
ATTNHP
+1.2% 1.0%
-1.2% - -1.2%
ODETPP
-4.0% 3.9%
-4.3% 4.5%
FULLYNN
55.0% N.A.
4.1%%/..A. /N.A.
IFTPP
+3.4% 3.1%
+3.9% + 3.00%
Table 4: The relative difference between the results of Easy TPP and original implementations.
MODEL
OTD
RETWEET AVG EVENTS
RETWEET AVG 10 EVENTS
TAXI AVG EVENTS
AVG EVENTS
MHP
5.128 (0.040)
11,270 (0.091)
4.633
12.784
(0.037)
(0.111)
RMTPP
5.107
10.255
4.401
12.045
(0.041)
(0.099)
(0.030)
(0.114)
NHP
5.080
10.470
4.412
12.110
(0.012)
(0.085)
(0.032)
(0.125)
SAHP
5.092
10.475
1.422
12.051
(0.039)
(0.079)
(0.039)
(0.139)
THP
5.091
10.450
4.398
11.875
(0.052)
(0.090)
(0.041)
(0.108)
ATTNHP
5.077
10.447
4.420
12.102
(0.039)
(0.090)
(0.044)
(0.109)
ODETPP
5.115
10.483
4.408
12.095
(0.041)
(0.088)
(0.039)
(0.100)
FULLYNN
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
IFTPP
5.079
10.513
4.501
12.052
(0.061)
(0.115)
(0.032)
(0.121)
Table 5:
Long horizon prediction on Retweet and Taxi data (the first row prediction value while second row is the standard deviation).
E ADDITIONAL NOTE
E. CITATION COUNT IN ARXIV
We search the TPP-related articles in ArXiv https::/arxtv..rrr using their own search engine in three folds:
Temporal point process: we search through the abstract of articles which contains the term 'tem- poral point process".
Hawkes process: we search through the abstract of articles with the term 'hawkes process" but without the term 'temporal point process'.
Temporal event sequence: search through the abstract of articles which include the term "tem- poral event sequence" but exclude the term 'hawkes process' and 'temporal point process'.
We group the articles found out by the search engine by years and report in Figure 1.
20

---

Published as a conference paper at ICLR 2024
MODEL
REL DIFF
TIME RMSE (1ST Row)
TYPE ERROR RATE (2ND ROW)
AMAZON
RETWEET
TAXI
TAOBAO
STACKOVERFLOW
RMTPP
0..2%
+1.0% +1.3%
+0.1%
+0.1%
+0.4%
+0.5%
+0.6%
+0.2%
-0.7%
NHP
+0.7%
+0.5%
-0.2%
+0.1%%
-0.1%
+0.6%
+1.4%
+0.4%
-0.3%
-0.1%
SAHP
-0.8%
+0.7%
0.8%
+0.4%
0.3%
+0.6%
+0.6%
0.6%
+0.4%
0.3%
THP
+0.6%
+0.6%
-0.2%
-0.5%
0.6%
+1.2%
+0.9%
-0.6%
+0.7%
0.4%
ATINHP
+0.4%
+0.4%
+0.3%
-0.1%
-0.2%
+0.2%
-0.7%
-0.6%
+0.4%
+0.2%
ODETPP
0.5%
+1.1%
+0.9%
+0.6%
0.4%
+0.8%
+1.3%
+1.1%
-0.5%
-0.5%
FULLYNN
+0.5%
-0.7%
0.3%
0..3%
+0.2%
NA
NA
NA
NA
NA
IFTPP
0..9%
+1.0%
+0.4%
+0.6% 4-0.2%
+0.3%
+0.4%
00.7%
0.3%
+...2%
Table 6: Relative difference between Torch and TensorFlow implementations of methods in Table
MODEL
METRICS (LOGLiKE)
SYNTHETIC
RETWEET
TAXI
MIIP
-3.150 (0.028)
5.949 (0.033)
-1,466 (0.011)
RMTPP
00.998
1.237
-0.227 (0-001)
(0.009)
(0.033)
NHP
-0.443
137
-0,208
(0.001)
(0.050)
(0.001)
SAHP
-1.337
-5.009
-0.478
(0.013)
(0.044)
(0-003)
THP
-1.238
-4.66
-0.442
(0.011)
(0.041)
(0.004)
ATTNHP
-1.001
4.756
-0.491
(0.008)
(0.052)
(0.004)
ODETPP
-1.007
-4.527
-0.217
(0.007)
(0.044)
(0.002)
FULLYNN
-2.318
-5. -5.889
-1.317
(0.014)
(0.032)
(0.009)
IFTPP
0.186
-0.212
0.019
(0.002)
(0.002)
(0.001)
Table 7: Performance in numbers all methods
Figure 4 (the first row is the prediction value while the second row is the standard deviation).
E.2 FUTURE DEVELOPMENT WORK
As the emergence of LLM-bbsed temporal models (Zhou et al., 2023; Chu al., 2023). making EasyTPP compatible with LLM models: such as hosting dataset models in HuggingFace, building L.LM-compatible training and evaluation pipelines.
21

---

Published as a conference paper at ICLR 2024
We are committed to actively maintaining this benchmark and welcome contributions from other researchers and practitioners.
22


Reviews:
[{"review_id": "oe8zLA3DC6", "reviewer": "Reviewer_Xrns", "summary": "The authors contribute the EasyTPP package for temporal point processes (TPPs). TPPs have been of increasing interest to the machine learning community in recent years, with many deep learning-based TPP models being able to outperform classic parametric TPP models in prediction tasks. There have not been many software packages implementing many different TPP models, which has made it difficult to compare and evaluate different models across standardized benchmark data sets and evaluation metrics. The EasyTPP package implements many recent TPP models and can be used with either PyTorch or TensorFlow. The authors utilize the package to benchmark these TPP models across a variety of data sets and prediction tasks.\n\n*After author rebuttal:* I have read through the other reviews and author rebuttal and still strongly support the paper. I further expand on my reasoning in a reply to the author comment titled \"Technical Contribution and Broader Impact.\"", "questions": "1. Is there an easy way to incorporate marks into your package? From the description in the paper, it seems like you can easily handle different event types through a multivariate TPP, but I don't see any discussion of marks. In many application settings, we can have a feature vector for each event, which can be modeled as a mark.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Addresses a major need in the machine learning community interested in TPPs--the lack of a standardized benchmarking tool. Most researchers are piecing together implementations from other papers in order to make comparisons. Existing software packages, such as PoPPy are out of date and not maintained. The proposed EasyTPP package could fill a major need for the community.\n- Implements a comprehensive list of models and evaluation metrics, together with several representative data sets.\n- The authors perform a thorough comparison of existing models using their EasyTPP package, which should also be useful to researchers in the area.", "weaknesses": "- The authors compare against the classical Multivariate Hawkes Process (MHP) but don't specify what type of kernel they use (I assume exponential) or structure of the excitation matrix.\n\nMinor concerns:\n- Some typos, e.g. Multivariate Hakwes Process (MHP)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0amopdnMH2", "reviewer": "Reviewer_MAje", "summary": "This paper proposes a software package that abstracts the implementation of several temporal point process models.", "questions": "1. This paper lacks technical innovation. I don't think a software-level abstraction is a good fit for ICLR.\n2. The description of the system design and interface lacks details.\n3. It's unclear what points the authors intend to demonstrate in the experiment section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "1. Good motivation", "weaknesses": "1. Lack of technique contribution\n2. Unclear presentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LS2iHhtZu8", "reviewer": "Reviewer_btMZ", "summary": "To promote reproducible research in this field, this article proposes an open and central benchmark for point process research. The benchmark provides a complete pipeline and software interface. The effectiveness of this benchmark is demonstrated by the unified comparison of various model results.", "questions": "Would you mind elaborating further on the broader impact this benchmark will have on the community?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The most important contribution lies in providing a simple and standardized framework to allow users\nto apply different TPP models to any datasets, which promotes reproducible research in this field.", "weaknesses": "The main concern is contribution. Considering the quality of the ICLR community, although it is particularly urgent to propose a unified comparison and processing framework for point processes, the contribution is still limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1AWIufOFCf", "reviewer": "Reviewer_2AV3", "summary": "The authors propose an end-to-end benchmarking package for neural temporal point process models. The package contains 6 diverse real-world datasets, 1 synthetic dataset, 8 recent models, and support for both Tensorflow and PyTorch. The paper itself serves as a useful exposition of the field, tasks, types of models, proposed package, and detailed empirical results.", "questions": "My questions align with the three specific shortcomings I identified:\n\n1) For reproducibility purposes, it would be useful to the community to know the exact accuracy values (and confidence bounds) from the various methods. Therefore in my opinion all plots showing empirical results should be replaced with tables. Is this possible in the camera-ready version?\n\n2) I may have missed this in my reading, but in Fig 1, I don't quite know what is meant by \"Temporal event sequence\" as opposed to \"temporal point process\". Furthermore, isn't a Hawkes process a version of a temporal point process? It seems that these publication sets may overlap somewhat. I'm wondering if this is clarified in the text, or if the authors can clarify.\n\n3) In Section 4's \"Data Preprocess\" section, the authors write \"to feed the sequences of varying length into the model, we pad all sequences to the same length, then use the \"sequence_mask\" tensor to identify which even tokens are padding.\" These masks are not defined, and are referenced no where else in the manuscript, so in my opinion this passage carries almost no information. To my understanding (which may be incorrect), some TPP datasets consist of a single sequence, and some models may process the sequence recurrently, i.e. each state embedding h_t depends only on the previous embedding h_{t-1}. For these models, why are variable-length sequences needed? On the other hand, datasets with events of multiple types (and models that can handle multi-type events) will need variable-length sequences. It would be very useful to the reader if the authors could describe these multiple scenarios, and specifically how EasyTPP handles each one at the implementation level.\n\nIf given satisfactory answers to the above, I am willing to raise my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper is a welcome contribution to TPP modeling at a timely moment when significant neural point process approaches have been published yet many open questions still remain. The proposed package has all the hallmarks of usefulness:\n\n1) Reproducible workflows via modular API.\n\n2) Diverse models and datasets.\n\n3) Supports top-2 ML languages equally (this is rare among benchmarking frameworks).\n\n4) Open-source repository and datasets.\n\nThe paper is overall well-written.", "weaknesses": "There are some presentation/exposition gaps/weaknesses:\n\n1) The accuracy plots would be more useful as tables.\n\n2) The \"temporal event sequence\" topic is not well-defined.\n\n3) The processing of sequence data is not adequately described.\n\nI elaborate on these weaknesses in questions to the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)