Detecting Anomalous Event Sequences with Temporal Point Processes

1 Introduction . Event data is abundant in the real world and is encountered in various important applications . For example , transactions in financial systems , server logs , and user activity traces can all naturally be represented as discrete events in continuous time . Detecting anomalies in such data can provide immense industrial value . For example , abnormal entries in system logs may correspond to unnoticed server failures , atypical user activity in computer networks may correspond to intrusions , and irregular patterns in financial systems may correspond to fraud or shifts in the market structure . Manual inspection of such event data is usually infeasible due to its sheer volume . At the same time , hand-crafted rules quickly become obsolete due to software updates or changing trends ( He et al. , 2016 ) . Ideally , we would like to have an adaptive system that can learn the normal behavior from the data , and automatically detect abnormal event sequences . Importantly , such a system should detect anomalies in a completely unsupervised way , as high-quality labels are usually hard to obtain . Assuming “ normal ” data is available , we can formulate the problem of detecting anomalous event sequences as an instance of out-of-distribution ( OoD ) detection . Multiple recent works consider OoD detection for image data based on deep generative models ( Ren et al. , 2019 ; Nalisnick et al. , 2019 ; Wang et al. , 2020 ) . However , none of these papers consider continuous-time event data . Deep generative models for such variable-length event sequences are known as neural temporal point processes ( TPPs ) ( Du et al. , 2016 ) . Still , the literature on neural TPPs mostly focuses on prediction tasks , and the problem of anomaly detection has not been adequately addressed by existing works ( Shchur et al. , 2021 ) . We aim to fill this gap in our paper . ∗Work done during an internship at Amazon Research . Code and datasets : https : //github.com/shchur/tpp-anomaly-detection 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our main contributions are the following : 1 . Approach for anomaly detection for TPPs . We draw connections between OoD detection and GoF testing for TPPs ( Section 2 ) . By combining this insight with neural TPPs , we propose an approach for anomaly detection that shows high accuracy on synthetic and real-world event data . 2 . A new test statistic for TPPs . We highlight the limitations of popular GoF statistics for TPPs and propose the sum-of-squared-spacings statistic that addresses these shortcomings ( Section 4 ) . The proposed statistic can be applied to both unmarked and marked TPPs . 2 Anomaly detection and goodness-of-fit testing . Background . A temporal point process ( TPP ) ( Daley & Vere-Jones , 2003 ) , denoted as P , defines a probability distribution over variable-length event sequences in an interval [ 0 , T ] . A TPP realization X consists of strictly increasing arrival times ( t1 , . . . , tN ) , where N , the number of events , is itself a random variable . A TPP is characterized by its conditional intensity function λ∗ ( t ) : = λ ( t|Ht ) that is equal to the rate of arrival of new events given the historyHt = { tj : tj < t } . Equivalently , a TPP can be specified with the integrated intensity function ( a.k.a . the compensator ) Λ∗ ( t ) = ∫ t 0 λ∗ ( u ) du . Out-of-distribution ( OoD ) detection . We formulate the problem of detecting anomalous event sequences as an instance of OoD detection ( Liang et al. , 2018 ) . Namely , we assume that we are given a large set of training sequencesDtrain = { X1 , . . . , XM } that were sampled i.i.d . from some unknown distribution Pdata over a domain X . At test time , we need to determine whether a new sequenceX was also drawn from Pdata ( i.e. , X is in-distribution or “ normal ” ) or from another distribution Q 6= Pdata ( i.e. , X is out-of-distribution or anomalous ) . We can phrase this problem as a null hypothesis test : H0 : X ∼ Pdata H1 : X ∼ Q for some Q 6= Pdata . ( 1 ) To reiterate , here we consider the case where X is a variable-length event sequence and Pdata is some unknown TPP . However , the rest of the discussion in Section 2 also applies to distributions over other data types , such as images . Goodness-of-fit ( GoF ) testing . First , we observe that the problem of OoD detection is closely related to the problem of GoF testing ( D ’ Agostino , 1986 ) . We now outline the setup and approaches for GoF testing , and then describe how these can be applied to OoD detection . The goal of a GoF test to determine whether a random elementX follows a known distribution Pmodel2 H0 : X ∼ Pmodel H1 : X ∼ Q for some Q 6= Pmodel . ( 2 ) We can perform such a test by defining a test statistic s ( X ) , where s : X → R ( Fisher , 1936 ) . For this , we compute the ( two-sided ) p-value for an observed realization x of X as3 ps ( x ) = 2×min { Pr ( s ( X ) ≤ s ( x ) |H0 ) , 1− Pr ( s ( X ) ≤ s ( x ) |H0 ) } . ( 3 ) The factor 2 accounts for the fact that the test is two-sided . We reject the null hypothesis ( i.e. , conclude that X doesn ’ t follow Pmodel ) if the p-value is below some predefined confidence level α . Note that computing the p-value requires evaluating the cumulative distribution function ( CDF ) of the sampling distribution , i.e. , the distribution test statistic s ( X ) under the null hypothesis H0 . GoF testing vs. OoD detection . The two hypothesis tests ( Equations 1 and 2 ) appear similar—in both cases the goal is to determine whetherX follows a certain distribution P and no assumptions are made about the alternative Q . This means that we can perform OoD detection using the procedure described above , that is , by defining a test statistic s ( X ) and computing the respective p-value ( Equation 3 ) . However , in case of GoF testing ( Equation 2 ) , the distribution Pmodel is known . Therefore , we can analytically compute or approximate the CDF of s ( X ) |X ∼ Pmodel , and thus the p-value . In contrast , in an OoD detection hypothesis test ( Equation 1 ) , we make no assumptions about Pdata and only 2We test a single realization X , as is common in TPP literature ( Brown et al. , 2002 ) . Note that this differs from works on univariate GoF testing that consider multiple realizations , i.e. , H0 : X1 , . . . , XM i.i.d.∼ Pmodel . 3In the rest of the paper , the difference between the random element X and its realization x is unimportant , so we denote both as X , as is usually done in the literature . have access to samples Dtrain that were drawn from this distribution . For this reason , we can not compute the CDF of s ( X ) |X ∼ Pdata analytically . Instead , we can approximate the p-value using the empirical distribution function ( EDF ) of the test statistic s ( X ) on Dtrain . The above procedure can be seen as a generalization of many existing methods for unsupervised OoD detection . These approaches usually define the test statistic based on the log-likelihood ( LL ) of a generative model fitted to Dtrain ( Choi et al. , 2018 ; Ren et al. , 2019 ; Ruff et al. , 2021 ) . However , as follows from our discussion above , there is no need to limit ourselves to LL-based statistics . For instance , we can define a test statistic for event sequences based on the rich literature on GoF testing for TPPs . We show in Section 6 that this often leads to more accurate anomaly detection compared to LL . Moreover , the difference between OoD detection and GoF testing is often overlooked . By drawing a clear distinction between the two , we can avoid some of the pitfalls encountered by other works ( Nalisnick et al. , 2019 ) , as we elaborate in Appendix A . The anomaly detection framework we outlined above can be applied to any type of data—such as images or time series—but in this work we mostly focus on continuous-time event data . This means that our main goal is to find an appropriate test statistic for variable-length continuous-time event sequences . In Section 3 , we take a look at existing GoF statistics for TPPs and analyze their limitations . Then in Section 4 , we propose a new test statistic that addresses these shortcomings and describe in more detail how it can be used for OoD detection . 3 Review of existing GoF test statistics for TPPs . Here , we consider a GoF test ( Equation 2 ) , where the goal is to determine whether an event sequence X = ( t1 , . . . , tN ) was generated by a known TPP Pmodel with compensator Λ∗ . We will return to the problem of OoD detection , where the data-generating distribution Pdata is unknown , in Section 4.2 . Many popular GoF tests for TPPs are based on the following result ( Ogata , 1988 ; Brown et al. , 2002 ) . Theorem 1 ( Random time change theorem ( Brown et al. , 2002 ) ) . A sequence X = ( t1 , . . . , tN ) is distributed according to a TPP with compensator Λ∗ on the interval [ 0 , V ] if and only if the sequence Z = ( Λ∗ ( t1 ) , . . . , Λ ∗ ( tN ) ) is distributed according to the standard Poisson process on [ 0 , Λ∗ ( V ) ] . Intuitively , Theorem 1 can be viewed as a TPP analogue of how the CDF of an arbitrary random variable over R transforms its realizations into samples from Uniform ( [ 0 , 1 ] ) . Similarly , the compensator Λ∗ converts a random event sequence X into a realization Z of the standard Poisson process ( SPP ) . Therefore , the problem of GoF testing for an arbitrary TPP reduces to testing whether the transformed sequence Z follows the SPP on [ 0 , Λ∗ ( T ) ] . In other words , we can define a GoF statistic for a TPP with compensator Λ∗ by ( 1 ) applying the compensator to X to obtain Z and ( 2 ) computing one of the existing GoF statistics for the SPP on the transformed sequence . This can also be generalized to marked TPPs ( where events can belong to one ofK classes ) by simply concatentating the transformed sequences Z ( k ) for each event type k ∈ { 1 , . . . , K } ( see Appendix D for details ) . SPP , i.e. , the Poisson process with constant intensity λ∗ ( t ) = 1 , is the most basic TPP one can conceive . However , as we will shortly see , existing GoF statistics even for this simple model have considerable shortcomings and can only detect a limited class of deviations from the SPP . More importantly , test statistics for general TPPs defined using the above recipe ( Theorem 1 ) inherit the limitations of the SPP statistics . For brevity , we denote the transformed arrival times as Z = ( v1 , . . . , vN ) = ( Λ∗ ( t1 ) , . . . , Λ∗ ( tN ) ) and the length of the transformed interval as V = Λ∗ ( T ) . One way to describe the generative process of an SPP is as follows ( Pasupathy , 2010 ) N |V ∼ Poisson ( V ) ui|N , V ∼ Uniform ( [ 0 , V ] ) for i = 1 , . . . , N. ( 4 ) An SPP realization Z = ( v1 , . . . , vN ) is obtained by sorting the ui ’ s in increasing order . This is equivalent to defining the arrival time vi as the i-th order statistic u ( i ) . We can also represent Z by the inter-event times ( w1 , . . . , wN+1 ) where wi = vi − vi−1 , assuming v0 = 0 and vN+1 = V . Barnard ( 1953 ) proposed a GoF test for the SPP based on the above description ( Equation 4 ) and the Kolmogorov–Smirnov ( KS ) statistic . The main idea of this approach is to check whether the arrival times v1 , . . . , vN are distributed uniformly in the [ 0 , V ] interval . For this , we compare F̂arr , the empirical CDF of the arrival times , with Farr ( u ) = u/V , the CDF of the Uniform ( [ 0 , V ] ) distribution . This can be done using the KS statistic on the arrival times ( KS arrival ) , defined as κarr ( Z ) = √ N · sup u∈ [ 0 , V ] |F̂arr ( u ) − Farr ( u ) | where F̂arr ( u ) = 1 N N∑ i=1 1 ( vi ≤ u ) . ( 5 ) Another popular GoF test for the SPP is based on the fact that the inter-event times wi are distributed according to the Exponential ( 1 ) distribution ( Cox , 1966 ) . The test compares F̂int , the empirical CDF of the inter-event times , and Fint ( u ) = 1− exp ( −u ) , the CDF of the Exponential ( 1 ) distribution . This leads to the KS statistic for the inter-event times ( KS inter-event ) κint ( Z ) = √ N · sup u∈ [ 0 , ∞ ) |F̂int ( u ) − Fint ( u ) | where F̂int ( u ) = 1 N + 1 N+1∑ i=1 1 ( wi ≤ u ) . ( 6 ) KS arrival and KS inter-event statistics are often presented as the go-to approach for testing the goodness-of-fit of the standard Poisson process ( Daley & Vere-Jones , 2003 ) . Combining them with Theorem 1 leads to simple GoF tests for arbitrary TPPs that are widely used to this day ( Gerhard et al. , 2011 ; Alizadeh et al. , 2013 ; Kim & Whitt , 2014 ; Tao et al. , 2018 ; Li et al. , 2018 ) . Limitations of the KS statistics . The KS statistics κarr ( Z ) and κint ( Z ) are only able to differentiate the SPP from a narrow class of alternative processes . For example , KS arrival only checks if the arrival times vi are distributed uniformly , conditioned on the event count N . But what if the observed N is itself extremely unlikely under the SPP ( Equation 4 ) ? KS inter-event can be similarly insensitive to the event count—removing all events V2 < vi ≤ V from an SPP realization Z will only result in just a single atypically large inter-event time wi , which changes the value of κint ( Z ) at most by 1N+1 . We demonstrate these limitations of κarr ( Z ) and κint ( Z ) in our experiments in Section 6.1 . Other failure modes of the KS statistics were described by Pillow ( 2009 ) . Note that ad-hoc fixes to the KS statistics do not address these problems . For example , combining multiple tests performed separately for the event count and arrival times using Fisher ’ s method ( Fisher , 1948 ; Cox , 1966 ) consistently decreases the accuracy , as we show in Appendix G. In the next section , we introduce a different test statistic that aims to address these shortcomings .

Summary: This paper addresses an anomaly detection problem for event sequences as temporal point process (TPP), where they formalize the problem as out-of-distribution (OoD) detection in a batch scenario. The authors proposed using goodness-of-fit (GoF) statistics to determine whether a sequence is OoD or not, which provides vast options for this purpose to us than an ordinary OoD procedure just using log-likelihood. They make it possible to use GoF statistics designed for known standard Poisson process by using Neural TPP fitted to training sequences. We can convert a random event sequence distributed according to an arbitrary TPP into a realization of the standard Poisson process with a compensator derived from the fitted Neural TPP. They also propose a sum-of-squared-spacings (3S) statistic that can detect anomalies w.r.t. the event count N, which is difficult to detect by Kolmogorov–Smirnov (KS) statistic, as well as anomalies w.r.t. the distribution of the arrivals or inter-event times. Experimental results demonstrate that the proposed statistic performs better than either other statistics or log-likelihood-based OoD.