Detecting Anomalous Event Sequences with Temporal Point Processes
Authors: Oleksandr Shchur
Ali Caner Turkmen
Tim Januschowski
Jan Gasthaus
Stephan Günnemann
Venue: NeurIPS 2021 Poster (2021)
Keywords: temporal point processes
out-of-distribution detection
anomaly detection
event data
generative models

Abstract: Automatically detecting anomalies in event data can provide substantial value in domains such as healthcare, DevOps, and information security. In this paper, we frame the problem of detecting anomalous continuous-time event sequences as out-of-distribution (OOD) detection for temporal point processes (TPPs). First, we show how this problem can be approached using goodness-of-fit (GoF) tests. We then demonstrate the limitations of popular GoF statistics for TPPs and propose a new test that addresses these shortcomings. The proposed method can be combined with various TPP models, such as neural TPPs, and is easy to implement. In our experiments, we show that the proposed statistic excels at both traditional GoF testing, as well as at detecting anomalies in simulated and real-world data. 

Detecting Anomalous Event Sequences with Temporal Point Processes
Oleksandr Shehur' Technical University of Munich shchur@in. tum.
Ali Caner Türkmen Amazon Research atturkm@amazon, com
Tim Januschowski
Amazon Research tjjnscheaazzoncom com
Jan Gasthaus
Stephan Günnemann
Amazon Research
Technical University of Munich guennemann@in. tum de
gasthaus@amazon.cmm com
Abstract
Automatically detecting anomalies event data can provide substantial value in domains such as healthcare, DevOps, and information security. In this paper, we frame the problem of detecting anomalous continuous- event sequences as out-of-distribution (OoD) detection for temporal point processes (TPPs). First, we show how this problem can be approached using goodness-offfit (GoF) tests. We then demonstrate the limitations of popular GoF statistics for TPPs and propose a new test that addresses these shortcomings. The proposed method can be combined with various TPP models, such as neural TPPs, and is easy to implement. In our experiments, we show that the proposed statistic excels both traditional GoF testing, well at detecting anomalies in simulated and real-world data.
Introduction
Event data is abundant in the real world and is encountered in various important applications. For example, transactions in financial systems, server logs, and user activity traces can all naturally be represented as discrete events in continuous time. Detecting anomalies in such data can provide immense industrial value. For example, abnormal entries in system logs may correspond to unnoticed server failures, atypical user activity in computer networks may correspond to intrusions, and irregular patterns financial systems may correspond fraud or shifts the market structure.
Manual inspection of such event data is usually infeasible due to its sheer volume. At the same time, hand-crafted rules quickly become obsolete due software updates or changing trends (He al., 2016). Ideally, we would like to have an adaptive system that can learn the normal behavior from the data, and automatically detect abnormal event sequences. Importantly, such a system should detect anomalies in completely unsupervised way, as high-quality labels are usually hard to obtain.
Assuming "normal" data is available, sequences as an instance of out-of-distribution (OoD) detection. Multiple recent works consider OoD detection for image data based deep generative models (Ren et al., 2019; Nalisnick et al., 2019; Wang et al., 2020). However, none of these papers consider continuous-time event data. Deep generative models for such variable-length event sequences known neural temporal point processes (TPPs) (Du et al., 2016). Still, the literature on neural TPPs mostly focuses on prediction tasks, and the problem of anomaly detection has been adequately addressed by existing works (Shchur et al., 2021), We aim to fill this gap in our paper.
formulate the problem of detecting anomalous event
Work done during an internship at Amazon Research. Code datasets: https:/ //gtthu.. com htap//hhhur/ttee taaachurrppp-eeee -ndeaaayeyelyettiiiio
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Our main contributions are the following:
1. Approach for anomaly detection for TPPs. We draw connections between OoD detection and GoF testing for TPPs (Section 2). By combining this insight with neural TPPs, we propose an approach for anomaly detection that shows high accuracy synthetic and real-world event data.
2. A new test statistic for TPPs. We highlight the limitations of popular GoF statistics for TPPs and propose the sum-of-squared -paccings statistic that addresses these shortcomings (Section 4). The proposed statistic can be applied to both unmarked and marked TPPs.
2 Anomaly detection and goodness-of-fit testing
Background. A temporal point process (TPP) (Daley & Vere-Jones, 2003), denoted as P, defines a probability distribution over variable-length event sequences in an interval [0. 2]. TPP realization X consists of strictly increasing arrival times (f1....,fx) fx), where N, the number of events, is itself a
random variable. A TPP is characterized by its conditional intensity function (t) := {(()H,) that is equal to the rate of arrival of new events given the history Hr = (t, it < t). Equivalently, TPP can be specified with the integrated intensity function (a.k.a. the compensator) A (+) = fo X- X(uudu
Out-of-distribution (OoD) detection. We formulate the problem of detecting anomalous event sequences an instance of OoD detection (Liang al., 2018). Namely, we assume that are given a large set of training sequences Drain = (X1,... NM) } that were sampled i.i.d. from some unknown distribution P data over domain X' At test time, we need determine whether new sequence X was also drawn from Paarn (i.e.. X is in-distribution or "normal") or from another distribution Q & I' dnna (i.e., X is out-of-distribution anomalous). We can phrase this problem as a null hypothesis test:
Ho: : X datta
H1 X Q for some 전 P datta.
To reiterate, here we consider the case where X is variable-length event sequence and datn is some unknown TPP. However, the rest of the discussion in Section also applies to distributions over other data types, such as images.
Goodness-of-fi (GoF) testing. First, we observe that the prob- lem of OoD detection is closely related to the problem of GoF testing (D' Agostino, 1986). We now outline the setup and ap- proaches for GoF testing, and then describe how these can be applied to OoD detection. The goal of GoF test determine whether a random element X follows a known distribution P Fmodel 2
Demity xX) Ho
1.5
1.0
pevalue (a
2xcreaa of the shaded region
0.5
0.0
Hy: X ~ P model
H1: X ~ Q for some Q o Pmoddl- (2)
2
Value of the test statistic of
We can perform such a test by defining a test statistic sXX), where s: X + IR (Fisher, 1936). For this, we compute the (two-sided) p-value for an observed realization x of X as²
Figure 1: p-value is computed as the tail probability under the sam- pling distribution s s())
(x) = 2 x (P(((())(X)) (()))Ho), 1 PPr(spPrs)))))))))) S Ho))
The factor 2 accounts for the fact that the test is two-sided. We reject the null hypothesis (i.e., conclude that X doesn't follow P Piwdee) if the p-value is below some predefined confidence level a. Note that computing the p-value requires evaluating the cumulative distribution function (CDF) of the sampling distribution, i.e., the distribution test statistic s(X) under the null hypothesis Ho-
GoF testing vs. OoD detection. determine whether X follows a certain distribution P and no assumptions are made about the alternative Q This means that we can perform OoD detection using the procedure described above, that is, by defining a test statistic s(X) and computing the respective p-value (Equation 3). However, in case of GoF testing (Equation 2), the distribution Pmmmiell is known. Therefore, we can analytically compute or approximate the CDF of x(X)X IP FFmudees and thus the p- valuee In contrast, in an OoD detection hypothesis test (Equation 1), we make no assumptions about Paat and only
The two hypothesis tests (Equations and 2) appear similar-in both
cases the goal
We test single realization X. is common in TPP literature (Brown et al., 2002). Note that this differs from works on univariate GoF testing that consider multiple realizations. i.e., Ho: Xi..., . XM P "I the rest of the paper, the difference between the random element X and its realization is unimportant, denote both X, as usually done in the literature.

---

have access to samples Drain that were drawn from this distribution. For this reason, we cannot compute the CDF of s(X) X ~ P Fatn analytically. Instead, we can approximate the p-value using the empirical distribution function (EDF) of the test statistic s ((X) on Duaii.
The ahove procedure can be seen as a generalization of many existing methods for unsupervised Ool detection. These approaches usually define the test statistic based on the log-likelihoo (LL) of generative model fitted Drain (Choi al., 201 8; Ren al., 2019; Ruff' al., 2021). However, as follows from our discussion above, there is no need to limit ourselves to LL-based statistics. For instance, we can define test statistic for event sequences based on the rich literature on GoF testing for TPPs. We show in Section 6 that this often leads to more accurate anomaly detection compared to LL. Moreover, the difference between OoD detection and GoF testing is often overlooked. By drawing clear distinction between the two, we can avoid some of the pitfalls encountered by other works (Nalisnick et al., 2019), as we elaborate in Appendix A.
The anomaly detection framework we outlined above can be applied to any type of data-such as images or time series- but in this work we mostly focus on continuous-time event data. This means that our main goal is to find an appropriate test statistic for variable-length continuous-tmme event sequences. In Section 3, we take look at existing GoF statistics for TPPs and analyze their limitations. Then in Section 4, we propose a new test statistic that addresses these shortcomings and describe in more detail how it can be used for OoD detection.
3 Review of existing GoF test statistics for TPPs
Here, we consider a GoF test (Equation 2), where the goal is to determine whether an event sequence X = (f1...., .ty) was generated by known TPP IP model with compensator A We will return to the problem of OoD detection, where the data- aata-generating distribution I' is unknown, in Section 4.2.
Many popular GoF tests for TPPs are based on the following result (Ogata, 1988; Brown et al., 2002).
Theorem 1 (Random time change theorem (Brown et al., 2002)). A sequence X = (t,...., .N) is distributed according a TPP with compensator A on the interval [0, V] if and only if the sequence Z = (f2)......... A (fx)) is distributed according to the standard Poisson process on (V))..
Intuitively, Theorem 1 can be viewed as a TPP analogue of how the CDF of an arbitrary random vari- able over R transforms its realizations into samples from Uniform[[0, 1]). Similarly, the compensator A converts a random event sequence X into a realization Z of the standard Poisson process (SPP). Therefore, the problem of GoF testing for an arbitrary TPP reduces testing whether the transformed sequence Z follows the SPP on [0, A (T)]. In other words, we can define a GoF statistic for a TPP with compensator A by (1) applying the compensator to X to obtain Z and (2) computing one of the existing GoF statistics for the SPP on the transformed sequence. This can also be generalized to marked TPPs (where events can belong to one of K classes) by simply concatentating the transformed sequences (k) for each event type k E f1....,K K} (see Appendix D for details).
SPP, i.e., the Poisson process with constant intensity (t) conceive. However, as we will shortly see, existing GoF statistics even for this simple model have considerable shortcomings and can only detect limited class of deviations from the SPP. More importantly, test statistics for general TPPs defined using the above recipe (Theorem 1) inherit the limitations of the SPP statisties.
is the most basic TPP can
For brevity, we denote the transformed arrival times as Z (V1, UN) = (42), (Ix)) and the length of the transformed interval as V A (T). One way to describe the generative process of an SPP is as follows (Pasupathy, 2010)
NIV ~ Poisson(V) (V)
44 N, V ~ Uniform([O, V1) for i = I.....N.
(4)
An SPP realization Z = (f1,... UN) is obtained by sorting the u, in increasing order. This is cquivalent to defining the arrival time i; as the i-th order statistic ((z). We can also represent Z by the inter-event times (wi,..., (NN,1).,,N+.1) where w; U;- 0;-1, assuming to 0 and UN+1 = V.
Barnard (1953) proposed a GoF test for the SPP based on the above description (Equation 4) and the Kolmogorov-Smimon arrival times 01, UN are distributed uniformly in the [o, V] interval. For this, we compare Fur. the empirical CDF of the arrival times, with (u) u/V the CDF of the Uniform((). (0, VJ) distribution.
(KS) statistic. The main idea of this approach is to check whether the
3

---

This can be done using the KS statistic on the arrival times (KS arrival), defined as
N Fur = N Σ 11
Karr (2) VN-
[Far(u) Fan Far w
sup
where
(5)
Another popular GoF test for the SPP is based on the fact that the inter-event times w, distributed according to the Exponential(1) distribution (Cox, 1966). The test compares Fini, the empirical CDF of the inter-event times, and (u) 1 exp(-u), the CDF of the Exponential(1) distribution. This leads to the KS statistic for the inter-event times (KS inter-event)
N+1 Σ
Fim (2) VN-
sup [Fi((u) (in( wE(0,00)
where
Pin(u) = N
1( S u).
(6)
KS arrival and KS inter-event statistics are often presented as the go-to approach for testing the goodness-of-fit of the standard Poisson process (Daley & Vere-Jones, 2003). Combining them with Theorem leads to simple GoF tests for arbitrary TPPs that are widely used to this day (Gerhard al., 2011; Alizadeh et al., 2013; Kim & Whitt, 2014; Tao al., 2018; 8; Li et al., 2018).
Limitations of the KS statistics. The KS statistics Kar (Z) and Kint (2) are only able to differentiate the SPP from a narrow class of alternative processes. For example, KS arrival only checks if the arrival times 22 are distributed uniformly, conditioned on the event count N. But what if the observed N is itself extremely unlikely under the SPP (Equation 4)? KS inter-event can be similarly insensitive to the event count- removing all events 2 < U2 S V from an SPP realization will only result in just single atypically large inter-event time W,, which changes the value of Nim (Z) at most by WIT' We demonstrate these limitations of Kam ()) and Kim in our experiments in Section 6.1. Other failure modes of the KS statistics were described by Pillow (2009). Note that ad-hoc fixes the KS statistics do not address these problems. For example, combining multiple tests performed separately for the event count and arrival times using Fisher's 's method (Fisher, 1948; Cox, 1966) consistently decreases the accuracy, as we show in Appendix G. In the next section, we introduce a different test statistic that aims to address these shortcomings.
4 Sum-off-qqurre-spaciin (3S) statistic for TPPs
4.1 Goodness-of-fi testing with the 38 statistic
good test statistic should capture multiple properties of the SPP once: it should detect deviations w.r.t. both the event count N and the distribution of the arrival or inter-event times. Here, we propose approach GoF testing with wewooo--uuuddddssseeaeeeee (3S) statistic that satisfies these desiderata,
(2)
Σ
Σ
This statistic extends the www.oo---uuurdsppcppaaeeeee statistic proposed as test uniformity for fixed- length samples by Greenwood (1946). The important difference between definition (Equation 7) and prior works (D' Agostino, 1986) is that we, for the first time, consider the TPP setting, where the number of events N is random well. For this reason, we use the normalizing constant 1/V instead of N/V2 (see Appendix B for details). As we will see, this helps capture abnormalities in the event count and results in more favorable asymptotic properties for the case of SPP.
Intuitively, for a fixed N, the statistic & is maximized if the spacings are extremely imbalanced, i.e., one inter-event time w; is close to V/ and the rest are close to zero. Conversely, vé attains its minimum when the spacings are all equal, that is w; N+T
V for all i.
In Figure 2a we visualize the distribution of N, V for two different values of N. We see that the distribution of sb depends strongly on N, therefore a GoF test involving t will detect if the event count N is atypical for the given SPP. This is in contrast to Kar and Kim, the distributions of which, by design, are (asymptotically) invariant under N (Figure 2b). Even if one accounts for this effect, e.g.. by removing the correction factor VN in Equations 5 and 6, their distributions change only slightly compared to the sum of squared spacings (see Figures 2c and 2d). To analyze other properties of the statistic, consider its moments under the null hypothesis.

---

1.5
Nar 1 - 86,
Ray 18 86,
Rox |N 85,V
1.5
1N
Frequency
1.0
10
1.0
0.3
38 statisttic
KS arrival (b)
KS arrival V
correction
inticr-ovent
v/N correction
(2)
(8)
(d)
Figure 2: Distribution of different test statistics for the standard Poisson process on [0, 100), con- ditioned on different event counts N. The 3S statistic allows us to differentiate between different
values of N, while the KS statistics are not sensitive to the changes in N.
Proposition 1. Suppose the sequence Z is distributed according to the standard Poisson process on the interval [0, Vl. Then the first two moments of the statistic Vp := 22() are
2 (V - 1) and Varly V]
4 12 (2V 7 e V (2V2 + 4V - V)).
E V-
The proof of Proposition 1 can be found in Appendix C. From Proposition 1 it follows that
lim V =
V-+00 Var[e]V] = 0.
lim
This leads to a natural notion of rypicality in the sense of Nalisnick al. (2019) and Wang et al. (2020) for the standard Poisson process. We can define the typical set of the SPP as the set of variable-length sequences Z on the interval [o, V] that satisfy |v (z) 2| S e for some small e O. It follows from Equation 8 and Chebyshev's 's inequality that for large enough V, the SPP realizations will fall into the typical set with high probability. Therefore, at least for large V, we should be able to detect sequences that are not distributed according the SPP based on the statistic vb.
Summary. To test the GoF of TPP with a known compensator A for an event sequence X (4,..., (,,), we first obtain the transformed sequence Z = (4)),),..., A A(x))) and compute the statistic (Z) according Equation 7. Since the CDF of the statistic under Hu cannot be computed analytically, we approximate it using samples drawn from P model- That is, we draw realizations Dmodel = (Xi....,Xp } from the TPP (c.g., using the inversion method (Rasmussen, 2018)) and compute the p-value for X (Equation 3) using the EDF of the statistic on Dmodel (North et al., 2002).
4.2 Out-of-distribution detection with the 3S statistic
We now return to the original problem of OoD detection in TPPs, where we have access to a set of in-distribution sequences Dusin and do not know the data-ges gett--ennratiig process Fdata-
s(X)JX Peam
3
s(X)XX P P
Our idea is to perform the OoD detection hypothesis test (Equa- tion 1) using the sum-of-sque www.oouuardd--paiinnaeeeeeeeeeee test statistic that we introduced in the previous section. However, since the gatergener TPP datt is unknown, we do not know the corresponding compensator that is necessary to compute the statistic. Instead, we can fit neural TPP model Pmodel (Du et al., 2016) to the sequences in Drrain and use the compensator A of the learned model to compute the statistic sX). . High flexibility of neural TPPs allows these models to more accu- rately approximate the true compensator. Having defined the statistic, we can approximate its distribution under Hn (i.c., assuming X PPaaa) by the EDF of the statistic on D We use this EDF to compute the p-values for our OoD detection hypothesis test and thus detect anomalous sequences. We provide the pseudocode description of our OoD detection method in Appendix D.
2
1
0
2
Figure 3: Sampling distribution for the OoD test (blue) and the GoF test (orange). While the same statistic s(X) is used in both cases, the p- values are computed differently de- pending on which test we perform.
We highlight that an OoD detection procedure like the one above is not equivalent to a GoF test for the learned generative model IP Pootell& as suggested by earlier works (Nalisnick et al., 2019). While we use
"We can replace the 3S statistic on the transformed sequence Z with any other statistic for the SPP, such as KS arrival. In Sections 6.2 and 6.3, compare different statistics constructed this way.
5

---

the compensator of the learned model to define the test statistic s(X), we compute the p-value for the OoD detection test based on s sX)] ~ P dama. This is different from the distribution { mmodel used in a GoF test, since in general PP model # dann Therefore, even if the distribution of a test statistic under the GoF test can be approximated analytically (as, e.g. for the KS statistic (Marsaglia et al., 2003)), we have to use the EDF of the statistic on Duain for the OoD detection test. Figure 3 visualizes this difference. Here, we fit a TPP model on the in-distribution sequences from the STEAD dataset (Section 6.3) and plot the empirical distribution of the respective statistic s(X) on Dinin (corresponds to [X ~ daama) and on model samples D meodel (corresponds to s [X ~ Fmodde).
5 Related work
Unsupervised OoD detection. OoD detection approaches based on deep generative models (similar to our approach in Section 4.2) have received lot of attention in the literature. However, there several important differences between our method and prior works. First, most existing approaches perform OoD detection based the log- likelihoodd (LL) of the model or some derived statistic (Choi et al., 2018; Ren et al., 2019; Nalisnick et al., 2019; Morningstar et al., 2021; Ruff et al., 2021). We observe that LL can be replaced by any other test statistic, e.g., taken from the GoF testing literature, which often leads to more accurate anomaly detection (Section 6). Second, unlike prior works, we draw clear distinction between OoD detection and GoF testing. While this difference may seem obvious in hindsight, it is not acknowledged by the existing works, which may lead to complications (see Appendix A). Also, our formulation of the OoD detection problem in Section 2 provides an intuitive explanation to the phenomenon of "typicality" (Nalismick et al., 2019; Wang et al., 2020). The (c, --typicall set of distribution IP simply corresponds to the acceptance region of the respective hypothesis test with confidence level (Equation 1). Finally, most existing papers study OoD detection for image data and none consider variable-length event sequences, which is the focus of our work.
Our OoD detection procedure is also related to the rarity anomaly score (Ferragut et al., 2012; Janzing al., 2019). The rarity score can be interpreted as the negative logarithm of one-sided p-value (Equation 3) of a GoF test that uses the log-likelihood of some known model as the test statistic. In contrast, we consider a broader class of statistics and learn the model from the data.
Anomaly detection for TPPs. OoD detection, as described in Section 2, is not the only way to formalize anomaly detection for TPPs. For example, Ojeda et al. (2019) developed a eestance-based approach for Poisson processes. Recently, Zhu et al. (2020) proposed to detect anomalous event sequences with an adversarially-traines model. Unlike these two methods, our approach can be combined with any TPP model without altering the training procedure. Liu & Hauskrecht (2019) studied anomalous event detection with TPPs, while we are concerned with entire event sequences.
GoF tests for TPPs. Existing GoF tests for the SPP usually check if the arrival times are distributed uniformly, using, c.g., the KS (Lewis, 1965) chi chi-squared statistic (Cox, 1955). Our 3S statistic favor- ably compares to these approaches thanks to its dependence on the event count N, as we cxplain in Sec- tion 4 and show experimentally in Section 6. Methods combining the random time change theorem with a GoF test for the SPP (usually, the KS test) have been used at least since Ogata (1988), and are es- pecially popular in neuroscience (Brown al., 2002; Gerhard et al., 2011: 1: Tao et al., 2018). However, these approaches inherit the limitations of the underlying KS statistic. Replacing the KS score with the 3S statistic consistently leads to a better separation between different TPP distributions (Section 6).
Gerhard & Gerstner (2010) discussed several GoF tests for discrete-time TPPs, while we deal with continuous time. Yang et al. (2019) proposed GoF test for point processes based on Stein's 's identity, which is related to a more general class of kernel-based based GoF tests (Chwialkowski et al., 2016; Liu et al. 2016). Their approach isn't isn't suitable for neural TPPs, where the Papangelou intensity cannot be computed analytically. A recent work by Wei et al. (2021) designed a GoF test for self-exciting processes under model misspecification. In contrast to these approaches, our proposed GoF test from Section 4.1 can be applied to any TPP with a known compensator.
statistic. A similar statistic was first used by Greenwood (1946) for testing whether a fixed number of points are distributed uniformly in an interval. Several follow-up works studied the limiting distribution of the statistic (conditioned on N) as N + do (Hill, 1979; Stephens, 1981; Rao & Kuo, 1984). Our proposed statistic (Equation 7) is not invariant w.r.t. N and, therefore, is better suited for testing TPPs. We discuss other related statistics in Appendix B.
luun-of--quared--paciig

---

interevent
DecreasingRate
Stopping
Renewal
Hawkes
Intomogeneouroolszon
SelfCorrecting
0.9
0.8
0.5
6.6
0.8
0.8
ROC ACCAUC
0.7
0.7
0.7
0.7
0.7
0.7
0.6
0.4
Detectability
Detectabilty
Detectability
Detectability
Detectability
Datectability
Figure 4: GoF testing for the standard Poisson process using different test statistics, measured with ROC AUC (higher better). See Section 6.1 for the description of the experimental setup.
Experiments
Our experimental evaluation covers two main topics. In Section 6.1, we compare the proposed 3S statistic with existing GoF statistics for the SPP. Then in Sections 6.2 and 6.3, we evaluate our OoD detection approach on simulated and real-world data, respectively. The experiments were run on machine with 1080Ti GPU. Details on the setup and datasets construction are provided in Appendix E & F.
6.1 Standard Poisson process
In Section 3 we mentioned several failure modes of existing GoF statistics for the SPP. Then, in Section 4.1 introduced the 3S statistic that was supposed address these limitations. Hence, the goal of this section is to compare the proposed statistic with the existing ones in the task of GoF testing for the SPP. We consider four test statistics: (1) KS statistic on arrival times (Equation 5), (2) KS statistic on inter-event times (Equation 6), (3) chi-squared statistic on the arrival times (Cox, 1955; Tao et al., 2018), and (4) the proposed 3S statistic (Equation 7).
To quantitatively compare the discriminative power of different statistics, we adopt an evaluation strategy similar to Gerhard & Gerstner (2010); Yang et al. (2019). First, generate consisting of 1000 SPP realizations. We use model to compute the empirical distribution function of each statistic s(Z) under Ho. Then, define two sets: DID (consisting of samples from moodel. the SPP) and DOOD (consisting of samples from Q, another TPP), each with 1000 sequences. Importantly, in this and following experiments, the training and test are always disjoint.
D
We follow the GoF testing procedure described at the end Section 4.1, which corresponds to the hypothesis test in Equation 2. That is, compute the p-value (Equation 3) for each sequence in the test sets using the EDF of s(2) on Duudel- A good test statistic s(2) should assign lower p- p-values to the OoD sequences from DOOD than ID sequences from DID allowing us discriminate between samples from Q and P model- We quantify how well a given statistic separates the two distributions by computing the area under the ROC curve (ROC AUC). This effectively averages the performance of statistic for the GoF hypothesis test over different significance levels a.
Datasets. We consider six choices for the distribution Q:
RATE, a homogeneous Poisson process with intensity μ 1; STOPPING, where events stop after some time tstpp E [0, V];
RENEWAL, where inter-event times are drawn i.i.d. from the Gamma distribution;
HAWKES, where events are more clustered compared to the SPP:
INIIOMOGENEOUS, a Poisson process with non-constant intensity X(t) = 3 sin(w));;
SELFCORRECTING, where events are more evenly spaced compared the SPP.
For cases the last 4 cases, the expected number of events is the same as for the SPP.
For each choice of Q we define a detectability parameter 5 E [o, 1]. where higher 8 corresponds to
TPPs that are increasingly dissimilar to the SPP. That is, setting 5 = corresponds a distribution Q that is exactly equal to the SPP, and 5 = 1 corresponds to a distribution that deviates significantly from
the detectability value of 8 0 corresponds to μ and B = 0 (i.c., A cr exp(-(L 1))
the SPP. For example, for a Hawkes with conditional intensity >²(1) = 1 B E
1) making Q
7

---

+
KS arrival
-C KS inter event
chisquared
53 seasse
+ Log likelhood
ServerStop
ServerOverload
Latency
SpikeTraine
1.0
1.0
1.0
1.0
0.9
0.8
0.5
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.5
0.5
0.5
0.5
0.1
0.3
0.3
0.3
0.0
0.4
0.6
1.0
0.4
0.8
Detectability
Detectablitty
Detectability
Detectability
Figure 5: OoD detection on simulated data using different test statistics, measured with with ROC AUC (higher is better). See Section 6.2 for the description of the experimental setup.
indistinguishable from P. The value of 8 0.5 corresponds to μ = 0.5 and B 0.5, which preserves the expected number of events N but makes the arrival times f; "burstier." We describe how the parameters of each distribution Q are defined based on d Appendix E. Note that, in general, the ROC AUC scores are not guaranteed to monotonically increase as the detectability 5 is increased.
Results. In Figure we present AUC scores for different statisties as d is varied. As expected, KS arrival accurately identifies sequences that come from Q where the absolute time of events are non-uniform (as in ENHOMOGENEOUS), Similarly, KS inter-event is good detecting deviations in the distribution of inter-event times, as in RENEWAL. The performance of the chi-squared statistic similar to that of KS arrival. Nevertheless, the above statisties fail when the expected number of events, N, changes substantially as in KS arrival and chi-squared on RATE, and KS inter-event on STOPPING. These failure modes match our discussion from Section 3.
In contrast, the 3S statistic stands out as the most consistent test (best or close-to-best performance in 5 out of 6 cases) and does not completely fail in any of the scenarios. The relatively weaker performance on spacings (D' Agostino, 1986) than imbalanced spacings. The results show that the 3S statistic able to detect deviations w.r.t. both the event count N (RATE and STOPPING), as well as the distributions of the inter-event times wy WRENEWAL) or the arrival times oy (HAWKES and INHOMOGENBOUS)- something that other GoF statistics for the SPP cannot provide.
implies that the 3S statistic is less sensitive to superuniform
6.2 Detecting anomalies simulated data
In this section, test the OoD detection approach discussed in Section 4.2, i.e., we perform anomaly detection for TPP with an unknown compensator. This corresponds to the hypothesis test in Equation| 1. We use the training set Dirain to fit an RNN-based neural TPP model (Shchur al., 2020) via maximum likelihood estimation (see Appendix F for details). Then, we define test statistics for the general TPP as follows. We apply the compensator A of the learned model to each event sequence X and compute the four statistics for the SPP from Section 6.1 on the transformed sequence Z = A (X).. We highlight that these methods are not "baselines" in the usual sense- the idea of combining a GoF statistic with learned TPP model to detect anomalous event sequences is itself novel and hasn't 't been explored by earlier works. The rest of the setup is similar to Section 6. We use Draai to compute the EDF of each statistic under Ho, and then compute the ROC AUC scores on the p-values. addition to the four statistics discussed before, we consider a two-sided test on the likeliool log (X) of the learned generative model, which corresponds to the approach by Nalisnick et al. (2019).
Datasets. Like before, we define a detectability parameter 8 for each scenario that determines how dissimilar ID and OoD sequences are. SERVER-STOP SERVER-OVERIOOR OVERLOAD and LATENCY are inspired by applications in DevOps, such as detecting anomalies in server logs.
and SERVER-STOP contain data generated by multivariate Hawkes process with 3 marks, e.g., modeling network traffic among 3 hosts. In OoD sequences, we change the influence matrix simulate scenarios where host goes offline (SERVER-STOP), and where a host goes down and the traffic is routed to different host Higher implies that the change in the influence matrix happens earlier.
SERVER-OVERLL
LATENCY contains events of two types, sampled as follows. The first mark, the "trigger," sampled from a homogencous Poisson process with rate μ 3. The arrival times of the second
8

---

Table 1: ROC AUC scores for OoD detection on real-world datasets (mean & standard error are computed over 5 runs). Best result in bold, results within 2 pp. of the best underlined.
KS arrival
KS inter-event
Chi-squared
Log-likelihodd
3S statistic
LOGS
Packet corruption (1%)
57.4 = 1.7
62.1 0.9
66.6 1.8
75.9 = 0.1
95.3 = 0.3
LOGS
Packet corruption (10%)
59.2 + 2.3
97.8 + 0.6
59.1 2.3
99.0 + 0.0
99.4 - 0.1
LOGS
Packet duplication (1%)
81.1 + 5.2
82.8 + 5.0
74.6 6.5
88.1 + 0.1
90.9 = 0.3
LOGS Packet delay (frontend)
95.6 = 1.2
98.9 0.4
99.3 0.1
90.9 = 0.0
97.6 = 0.1
LOGS
Packet delay (all services)
99.8 + 0.0
94.7
99.8 + 0.0
96.1 + 0.0
99.6 + 0.1
STEAD Anchorage, AK
59.6 0.2
7.17
6.2
88.0
88.3
STEAD Aleutian Islands,
53.8 0.5
88.8 0.3
0.9
97.0
99.8 =
STEAD Helmet, CA
59.1+09
98.7 + 0.0
0.6
96.9 0.0
92.6 0.3
mark, the "response," " are obtained by shifting the times of the first mark by an offset sampled i.i.d. from (u = 1, = 0.1). ) In OoD sequences, the delay is increased by an amount proportional to 6, which emulates an increased latency in the system.
SPIKETRAINS (Stetter et al., 2012) contains sequences of firing times of 50 neurons, each represented by a distinct mark. We generate OoD sequences by shuffling the indices of neurons (e.g., switching marks 1 and 2), where higher detectability 5 implies more switches k. Here we study how different statistics behave for TPPs with a large number of marks.
Results are shown in Figure 5. The 3S statistic demonstrates excellent performance in all four scenar- ios, followed by KS arrival and chi-squared. case of SERVER- and SERVER- OVERLLLDD, the 3S statistic allows to perfectly detect the anomalies even when only 5% of the time interval are af- fected by the change in the influence structure. KS inter-event and log-likelihooe statistics completely fail on SERVER -ERVER-STOP and SERVER SERVER-OVERLOAR OVERLOAD, respectively. These statisties also struggle to discriminate OoD sequences in LATENCY and SPIKETRAIN scenarios. The non non-monotone behavior of the ROC AUC scores for some statistics (as the ó increases) indicates that these statisties poorly suited for the respective scenarios.
6.3 Detecting anomalies in real-world data
Finally, we apply our methods to detect anomalies in two real-world event sequence datasets. We keep the setup (e.g., configuration of the neural TPP model) identical to Section 6.2.
LOGS: We generate server logs using Sock Shop microservices (Weave, 2017) and represent them as marked event sequences. Sock Shop a standard testbed for research in microservice applications (Aderaldo et al., 2017) and contains web application that runs several containerized services. We generate OoD sequences by injecting various failures (e-g.. packet corruption, increased latency) among these microservices using chaos testing tool Pumba (Ledenev et al., 2016). We split one large server log into 30-second subintervals, that are then partitioned into train and test sets.
STEAD (Stanford Earthquake Dataset) (Mousavi et al., 2019) includes detailed seismic measure- ments over million earthquakes. We construct four subsets, each containing 72-hour subintervals in a period of five years within a 350km radius of a fixed geographical location. We treat se- quences corresponding the San Mateo, CA region as in-distribution data, and the remaining regions (Anchorage. AK, Alcutian Islands, AK and Helmet, CA) as OoD data.
Results. Table shows the ROC AUC scores for all scenarios. KS arrival and chi-squared achieve surprisingly low scores in 6 out scenarios, even though these two methods showed strong results simulated data in Sections 6.1 and 6.2. In contrast, KS inter-event and log-likelihood perform better here than in previous experiments, but still produce poor results on Packet corruption. The 3S statistic is the only method that consistently shows high ROC AUC scores across all scenarios. Moreover, we observe that for marked sequences (LOGS and all datasets in Section 6.2), the 3S statistic leads to more accurate detection compared to the likeeiiid statistic in out of cases.
Discussion
Limitations. Our approach assumes that the sequences in Drain were drawn i.i.d. from the true data-generating distribution Paaa (Section 2). This assumption can be violated in two ways: some of the training sequences might be anomalous there might exist dependencies between them. We have

---

considered the latter case in our experiments on SPIKETRAINS and LOGS datasets, where despite the non-i.id. nature of the data our method was able accurately detect anomalies. However, there might exist scenarios where the violation of the assumptions significantly degrades the performance.
No single test statistic can be "optimal" for either Ool detection or GoF testing, since make no assumptions about the alternative distribution Q (Section 2). We empirically showed that the proposed 3S statistic compares favorably to other choices over a range of datasets and applications domains. Still, for any fixed pair of distributions IP and Q, one can always find a statistic that will have equal or higher power s.t. the same false positive rate (Neyman & Pearson, 1933). Hence, won't 't be surprising to find cases where our (or any other chosen priori) statistic is inferior.
Broader impact. Continuous- tieeetuuuu---uueeeeeeeee aariable-enggh event sequences provide a natural representation for data such as electronic health records (Enguehard et al., 2020), server logs (He et al., 2016) and user activity traces (Zhu et al., 2020). The ability perform unsupervised anomaly detection in such data can enable practitioners to find at-risk patients, reduce DevOps costs, and automatically detect security breaches-all all of which are important tasks in the respective fields. One of the risks when applying an anomaly detection method in practice is that the statistical anomalies found by the method will not be relevant for the use case. For example, when looking for health insurance fraud, the method might instead flag legitimate patients who underwent atypically many procedures as "suspicious" and freeze their accounts. To avoid such situations, automated decisions systems should be deployed with care, especially in sensitive domains like healthcare.
Conclusion. We have presented approach for OoD detection for temporal point processes based on goodness-of-fit testing. At the core of our approach lies a new GoF test for standard Poisson processes based on the 3S statistic. Our method applies to a wide class of TPPs and is extremely easy to implement. We empirically showed that the proposed approach leads to better OoD detection accuracy compared to both popular GoF statisties for TPPs (Kolmogorov-Sminoo chi-squared) and approaches commonly used in OoD detection literature (model log-likelihood), While our analysis focuses on TPPs, we believe our discussion on similarities and distinctions between GoF testing and OoD detection offers insights to the broader machine learning community.
Funding transparency statement
The work was funded by Amazon Research.
References
Aderaldo, C. M., Mendonça, N. C., Pahl, C., and Jamshidi, P. Benchmark requirements for mi- croservices architecture research. In International Workshop on Establishing the Community Infrastructure for Architecture -Base Software Engineering, 2017.
Alizadeh, M., Scaglione, A., Davies, J., and Kurani, K. S. A scalable stochastic model for the electricity demand of electric and plug-in in hybrid vehicles. IEEE Transactions Smart Grid, 5(2), 2013.
Barnard, G. Time intervals between accidents-a note Maguire, Pearson and Wynn's paper. Biometrika, 40(1-2):212-213, 1953.
Blum, A., Hopcroft, J., and Kannan, R. Foundations of data science. 2016.
Brown, E. N., Barbieri, R., Ventura, V., Kass, R. F., and Frank, I.. M. The time-rescaling theorem and its application to neural spike train data analysis. Neural Computation, 14(2), 2002.
Choi, H., Jang, E., and Alemi, A. A. WAIC, but why? Generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392 2018.
Chwialkowski, K., Strathmann, H., and Gretton, A. A kernel test of goodness of fit. International Conference Machine Learning, 2016.
Cox, D. R. Some statistical methods connected with series of events. Journal of the Royal Statistical
Society: Series B, 17(2), 1955.
10

---

Cox, D. R.
The statistical analysis of series of events. Monographs on Applied Probability and Statistics, 1966.
D' "goostino, Goodness -f fit-techniques. 1986.
Daley, and Vere-Jones, An introduction the theory of point processes, volume 1: Elementary theory and methods. 2003.
Du, N., Dai, H., Trivedi, R., Upadhyay. U., Gomez-Rodriguez, M., and Song. L. Recurrent marked temporal point processes: Embedding event history to vector. International Conference on Knowledge Discovery and Data Mining, 2016.
Enguehard, J., Busbridge, D., Bozson, A., Woodcock, C.,, and Hammerla, N. Neural temporal point processes for modelling electronic health records. In Machine Learning for Health, 2020.
Ferragut, E. M., Laska, J., and Bridges, R. A. new, principled approach to anomaly detection. In International Conference Machine Learning and Applicaationss 2012.
Fisher, R. A. Design of experiments. British Medical Journal, 1(3923):554-554 1936.
Fisher, R. A. Answer to question 14 combining independent tests of significance. 1948.
Gerhard, F. and Gerstner, W. Rescaling, thinning complementing? On goodness-of-fit procedures for point process models and generalized linear models. Advances in Neural Information Processing Systems, 2010.
Gerhard, F., Haslinger, R., and Pipa, G. Applying the multivariate time-rescaling theorem to neural population models. Neural computation, 23(6), 2011.
Greenwood, M. The statistical study of infectious diseases. Journal of the Royal Statistical Society: Series A, 109:85-110, 1946.
He, S., Zhu, J., He, P., and Lyu, M. R. Experience report: In International Symposium on Software Reliability Engineering, 2016.
System log analysis for anomaly detection.
Hill, I. Approximating the distribution of Greenwood's statistic with Johnson distributions. Journal of the Royal Statistical Sociery: Series A, 142(3):378-380, 1979.
Janzing, D., Budhathoki, K., Minories, L., and Blöbaum, Causal structure based root cause analysis of outliers. arXiv: 1xXix:1912,02724, 2019.
Kim, S.-H. and Whitt, W. Are call center and hospital arrivals well modeled by nonhomogencous Poisson processes? Manufacturing & Service Operations Management, 16(3), 2014.
Ledenev, A. al. Pumba: Chaos testing tool for Docker. https://githut. com/ alleei-eeeeeee pumba, 2016.
Lewis, P. A. W. Some results on tests for Poisson processes. Biometrika, 52(1/2), 1965.
Li, S., Xiao, S., Zhu, S., Du, N., Xie, Y., and Song, L. Learning temporal point processes via reinforcement learning. In Advances in Neural Information Processing Systems, 2018.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. International Conference on Learning Representationss 2018.
Liu. Q., Lee, J., and Jordan, M. A kernelized Stein discrepancy for goodness-of-fit tests. In International Conference on Machine Learning, 2016.
Liu, S. and Hauskrecht, M. Detection of outlier events in continuous-time event sequences. arXiv preprint arXiv:1912.09522 2019.
Marsaglia, G., Tsang, W. W., Wang. J., and Others. Evaluating Kolmogorov's distribution. Journal of Statistical Software, 8(18), 2003.
Moran, P. The random division of an interval. Supplement to the Journal of the Royal Statistical Society, 91), 1947.
11

---

Morningstar, W. R., Ham, C., Gallagher, A. G., Lakshminarayanan, B., Alemi, A. A., and Dillon, V. Density of states estimation for out-of-distribttion detection. International Conference on Artificial Intelligence and Statistics, 2021.
Mousavi, S. M., Sheng, Y. Zhu, W., and Beroza, G. C. STanford EArthquake Dataset (STEAD): A global data set of seismic signals for AI. IEEE Access, 2019.
Nalisnick, E., Matsukawa, A., Teh, Y. W., and Lakshminarayanan, B. Detecting out-of-distribution inputs to deep generative models using test for typicality. arXiv preprint arXiv:7906.00 2019.
Neyman, J. and Pearson, E. S. On the problem of the most efficient tests of statistical hypotheses.
Philosophical Transactions of the Royal Society of London: Series A, 23 231(694-70))289- 1933.
North, B. V., Curtis, D., and Sham, P. C. A note on the calculation of empirical p-values from Monte Carlo procedures. The American Journal of Human Genetics, 71(2), 2002.
Ogata, Y. Statistical models for earthquake occurrences and residual analysis for point processes. Journal of the American Statistical association, 83(401), 1988.
Ojeda, C. A. M., Cvejoski, K., Sifa, R., Schuecker, J., and Bauckhage, C. Patterns and outliers in temporal point processes. In SAI Intelligent Systems Conference, 2019.
Pasupathy, R. Generating homogeneous Poisson processes. Wiley encyclopedia of operations research and management science, 2010.
Pillow, J. Time-rescaling methods for the estimation and assessment of non non-Poisson neural encoding models. Advances in Neural Information Processing Systems, 22, 2009.
Rao, J. and Kuo, M. Asymptotic results on the Greenwood statistic and some of its generalizations. Journal of the Royal Statistical Society: Series B, 46(2):228-22) 237, 1984.
Rasmussen, J. G. Lecture notes: Temporal point processes and the conditional intensity function. arXiv preprint arXiv: 18X6::880600022 2018.
Ren, J., Liu, P. J., Fertig, E.,, Snoek, J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, 2019.
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., and Müller, K.-R. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
Shchur, O., Biloš, M., and Günnemann, S. Intensity-free learning of temporal point processes. In International Conference on Learning Represenations, 2020.
Shehur, O., Türkmen, A. C., Januschowski, T., and Günnemann, S. Neural temporal point processes:
A review. International Joint Conference on Artificial Intelligence, 2021.
Stephens, M. A. Further percentage points for Greenwood's statistic. Journal of the Royal Statistical Society: Series A, 144(3)3644366, 1981.
Stetter, O.,, Battaglia, D.. Soriano, J., and Geisel, T. Model-free reconstruction of excitatory neuronal
connectivity from calcium imaging signals. PLoS Computational Biology, 8(8), 2012.
Tao, L., Weber, K. E., Arai, K., and Eden, U. T. A common goodness-of-fi framework for neural pop- ulation models using marked point process time-rescall rescaling. Journal of Computational Neuroscience, 45(2), 2018.
Wang, Z., Dai, B., Wipf, D., and Zhu, J. Further analysis of outlier detection with deep generative models. Advances in Neural Information Processing Systems, 2020.
Weave. Sock shop : A microservice demo application.
com/
mi microservices-demt www.de//wiiiooorriieeee-deeeeeee 2017.
12

---

Wei. S., Zhu, S., Zhang, M., and Xie, Y. Goodness-of fit test for mismatched self-exciting processes. In International Conference on Artificcal Intelligence and Statistics, 2021.
Yang. J. Rao, V., and Neville, J. A Stein-Papangelou goodness-of-fit test for point processes. In International Conference Artificial Intelligence and Statistics, 2019.
Zhu, S., Yuchi, H. S., and Xie, Y. Adversarial anomaly detection for marked spatio-temporal streaming data, In International Conference on Acoustics, Speech and Signal Processing, 2020.
13


Reviews:
[{"review_id": "zR1LbWZtMHg", "reviewer": "Reviewer_XhWV", "summary": "This paper focuses on the anomaly detection of continuous-time event sequences with temporal point processes and proposes to leverage the idea of goodness-of-fit testing to check whether sequences follow the in-distribution. Consider the limitation that the current statistics are not sensitive to the event number N, this paper proposes a new statistic called sum-of-squared-spacings (3S) to check the fitness of the sequence. The experimental results show the effectiveness of the proposed statistic for anomaly detection.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Detecting anomalous continuous-time event sequences is an important task yet under-exploited. This paper proposes a new statistic to identify the anomalous event sequences, called 3S. Evaluating on two synthetic and one real datasets shows the effectiveness of using 3S for anomaly detection. The overall paper is well-written with sufficient references and experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Two minor comments here. First, I personally would expect more explanation about the advantages of the proposed 3S statistic over the log-likelihood approach. As mentioned in the paper, many existing anomaly detection approaches are actually based on log-likelihood, because the log-likelihood approach is more intuitive. The experiments on the real-world datasets in this paper also show that log-likelihood can achieve good performance. I believe give more comparisons about these two approaches would be appreciated. Another comment may be outside the scope of this paper, but I am curious about how to detect the marked event sequences with the marked TPP model. Say, using log-likelihood, it would be easy to combine the signals about marked sequences as well as the time sequences for anomaly detection because both can be modeled via log-likelihood, but the proposed 3S statistic can only detect anomalous sequences from the time perspective. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NavgXV-Sy5o", "reviewer": "Reviewer_jvJp", "summary": "This paper discusses anomaly detection in continuous-time event sequences as out-of-distribution (OoD) detection for temporal point processes, and statistical tests for the detection. The paper proposes a new statistical test for the detection and demonstrates the usefulness of the proposed test in simulated and real-world data sets.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper. The connection between anomaly detection in continuous-time event sequences and out of distribution detection has been discussed, and such a connection is useful for anomaly detection research. The proposed 3S test is well motivated and sound. 3S test has been shown effective for anomaly detection in continuous-time event sequences in experiments.   \n\nDiscussions in the paper are focused, clear and easy to follow. The authors evaluate the proposed test in different circumstances, and the strengths and weaknesses of the proposed test have been presented. The connection between anomaly detection and out of distribution detection, and existing and proposed statistical tests are useful for anomaly detection research.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "F_vkgdjMkp", "reviewer": "Reviewer_kuGg", "summary": "This paper addresses an anomaly detection problem for event sequences as temporal point process (TPP), where they formalize the problem as out-of-distribution (OoD) detection in a batch scenario.\nThe authors proposed using goodness-of-fit (GoF) statistics to determine whether a sequence is OoD or not, which provides vast options for this purpose to us than an ordinary OoD procedure just using log-likelihood.\nThey make it possible to use GoF statistics designed for known standard Poisson process by using Neural TPP fitted to training sequences. We can convert a random event sequence distributed according to an arbitrary TPP into a realization of the standard Poisson process with a compensator derived from the fitted Neural TPP.\nThey also propose a sum-of-squared-spacings (3S) statistic that can detect anomalies w.r.t. the event count N, which is difficult to detect by Kolmogorov–Smirnov (KS) statistic, as well as anomalies w.r.t. the distribution of the arrivals or inter-event times.\nExperimental results demonstrate that the proposed statistic performs better than either other statistics or log-likelihood-based OoD.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality:\n* This work proposes a novel combination of GoF statistic and OoD for event sequences with learned neural TPP for the unknown data-generating distribution.\n\n\n### Quality:\n* The authors introduced the motivation of the proposed method and provided the necessary background, which would be informative for many readers.\n\n* The requirements of the proposed method would limit its practical usability.\n    - $T$ number of observations are required for anomaly detection with the proposed method. Also, $T$ and $V$ are assumed to be large enough in their analysis. In that case, it is challenging to perform timely anomaly detection, which is crucial for real applications, such as detecting intrusions in computer networks and detecting fraud or shifts in the market structure on financial systems. It would be better to add analysis w.r.t different configurations for $V$.\n\n* The motivation and justification of the use of 3S statistics are unclear.\n    - Is it possible to use other statistics instead of 3S statistics?\n    - In l.562 in Appendix, the authors replaced $N$ with its expectation $E[N |V ]$. I cannot find the justification for that. Is it just heuristics for considering $N$ as a random variable? Are there any other variants?\n\n* The compared methods seem naive.\n    - They analyze when KS statistic is modified by removing $\\sqrt{N}$, but they can also replace $1/N$ in the second equation with $1/V$ similar to the proposed method. It would work better.\n    - It would be better to add a comparison with the original 3S statistic without modification.\n\n\n### Clarity:\n* The discussion in l.195 is confusing since it would be mainly for KS statistic, not for 3S statistic, which does not require CDF. If so, the paragraph from l.195 would not be a help to understand the proposed method.\n\n* There is no description of how to choose hyperparameters of models, such as how they split validation set from the training set and test set.\n\n* In section 6.3, there is no description of the model they used.\n\n* Figure 1 is not mentioned in the main text. Also, in general, it is better to locate figures on top.\n\n\n### Significance:\n* The practical usability of the proposed method can be limited because of its requirements, as stated above.\n* Since the compared methods seem naive in the experiments, it is not easy to find the significance of the proposed method.\n\n\n================ Update: The motivation and novelty of the proposed method have been clarified in the rebuttal. Also, contributions on empirical evaluation should be noted, where the proposed approach worked well even on short intervals.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors formulate anomaly detection in continuous-time event sequences as an out-of-distribution detection problem for temporal point processes.  This allows them to apply a number of common goodness-of-fit measures to detect the out-of-distribution data (e.g., KS tests on arrival or inter-event times), and propose other statistics (specifically, 3S or "sum of squared spacings") and test for typicality of the statistic values under the training dataset.  Overall reviewers found the work clear and novel and recommend acceptance.  Please see the reviews for more detailed suggestions to improve the presentation.

Decision: Accept (Poster)