Detecting Anomalous Event Sequences with Temporal Point Processes

1 Introduction . Event data is abundant in the real world and is encountered in various important applications . For example , transactions in financial systems , server logs , and user activity traces can all naturally be represented as discrete events in continuous time . Detecting anomalies in such data can provide immense industrial value . For example , abnormal entries in system logs may correspond to unnoticed server failures , atypical user activity in computer networks may correspond to intrusions , and irregular patterns in financial systems may correspond to fraud or shifts in the market structure . Manual inspection of such event data is usually infeasible due to its sheer volume . At the same time , hand-crafted rules quickly become obsolete due to software updates or changing trends ( He et al. , 2016 ) . Ideally , we would like to have an adaptive system that can learn the normal behavior from the data , and automatically detect abnormal event sequences . Importantly , such a system should detect anomalies in a completely unsupervised way , as high-quality labels are usually hard to obtain . Assuming “ normal ” data is available , we can formulate the problem of detecting anomalous event sequences as an instance of out-of-distribution ( OoD ) detection . Multiple recent works consider OoD detection for image data based on deep generative models ( Ren et al. , 2019 ; Nalisnick et al. , 2019 ; Wang et al. , 2020 ) . However , none of these papers consider continuous-time event data . Deep generative models for such variable-length event sequences are known as neural temporal point processes ( TPPs ) ( Du et al. , 2016 ) . Still , the literature on neural TPPs mostly focuses on prediction tasks , and the problem of anomaly detection has not been adequately addressed by existing works ( Shchur et al. , 2021 ) . We aim to fill this gap in our paper . ∗Work done during an internship at Amazon Research . Code and datasets : https : //github.com/shchur/tpp-anomaly-detection 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our main contributions are the following : 1 . Approach for anomaly detection for TPPs . We draw connections between OoD detection and GoF testing for TPPs ( Section 2 ) . By combining this insight with neural TPPs , we propose an approach for anomaly detection that shows high accuracy on synthetic and real-world event data . 2 . A new test statistic for TPPs . We highlight the limitations of popular GoF statistics for TPPs and propose the sum-of-squared-spacings statistic that addresses these shortcomings ( Section 4 ) . The proposed statistic can be applied to both unmarked and marked TPPs . 2 Anomaly detection and goodness-of-fit testing . Background . A temporal point process ( TPP ) ( Daley & Vere-Jones , 2003 ) , denoted as P , defines a probability distribution over variable-length event sequences in an interval [ 0 , T ] . A TPP realization X consists of strictly increasing arrival times ( t1 , . . . , tN ) , where N , the number of events , is itself a random variable . A TPP is characterized by its conditional intensity function λ∗ ( t ) : = λ ( t|Ht ) that is equal to the rate of arrival of new events given the historyHt = { tj : tj < t } . Equivalently , a TPP can be specified with the integrated intensity function ( a.k.a . the compensator ) Λ∗ ( t ) = ∫ t 0 λ∗ ( u ) du . Out-of-distribution ( OoD ) detection . We formulate the problem of detecting anomalous event sequences as an instance of OoD detection ( Liang et al. , 2018 ) . Namely , we assume that we are given a large set of training sequencesDtrain = { X1 , . . . , XM } that were sampled i.i.d . from some unknown distribution Pdata over a domain X . At test time , we need to determine whether a new sequenceX was also drawn from Pdata ( i.e. , X is in-distribution or “ normal ” ) or from another distribution Q 6= Pdata ( i.e. , X is out-of-distribution or anomalous ) . We can phrase this problem as a null hypothesis test : H0 : X ∼ Pdata H1 : X ∼ Q for some Q 6= Pdata . ( 1 ) To reiterate , here we consider the case where X is a variable-length event sequence and Pdata is some unknown TPP . However , the rest of the discussion in Section 2 also applies to distributions over other data types , such as images . Goodness-of-fit ( GoF ) testing . First , we observe that the problem of OoD detection is closely related to the problem of GoF testing ( D ’ Agostino , 1986 ) . We now outline the setup and approaches for GoF testing , and then describe how these can be applied to OoD detection . The goal of a GoF test to determine whether a random elementX follows a known distribution Pmodel2 H0 : X ∼ Pmodel H1 : X ∼ Q for some Q 6= Pmodel . ( 2 ) We can perform such a test by defining a test statistic s ( X ) , where s : X → R ( Fisher , 1936 ) . For this , we compute the ( two-sided ) p-value for an observed realization x of X as3 ps ( x ) = 2×min { Pr ( s ( X ) ≤ s ( x ) |H0 ) , 1− Pr ( s ( X ) ≤ s ( x ) |H0 ) } . ( 3 ) The factor 2 accounts for the fact that the test is two-sided . We reject the null hypothesis ( i.e. , conclude that X doesn ’ t follow Pmodel ) if the p-value is below some predefined confidence level α . Note that computing the p-value requires evaluating the cumulative distribution function ( CDF ) of the sampling distribution , i.e. , the distribution test statistic s ( X ) under the null hypothesis H0 . GoF testing vs. OoD detection . The two hypothesis tests ( Equations 1 and 2 ) appear similar—in both cases the goal is to determine whetherX follows a certain distribution P and no assumptions are made about the alternative Q . This means that we can perform OoD detection using the procedure described above , that is , by defining a test statistic s ( X ) and computing the respective p-value ( Equation 3 ) . However , in case of GoF testing ( Equation 2 ) , the distribution Pmodel is known . Therefore , we can analytically compute or approximate the CDF of s ( X ) |X ∼ Pmodel , and thus the p-value . In contrast , in an OoD detection hypothesis test ( Equation 1 ) , we make no assumptions about Pdata and only 2We test a single realization X , as is common in TPP literature ( Brown et al. , 2002 ) . Note that this differs from works on univariate GoF testing that consider multiple realizations , i.e. , H0 : X1 , . . . , XM i.i.d.∼ Pmodel . 3In the rest of the paper , the difference between the random element X and its realization x is unimportant , so we denote both as X , as is usually done in the literature . have access to samples Dtrain that were drawn from this distribution . For this reason , we can not compute the CDF of s ( X ) |X ∼ Pdata analytically . Instead , we can approximate the p-value using the empirical distribution function ( EDF ) of the test statistic s ( X ) on Dtrain . The above procedure can be seen as a generalization of many existing methods for unsupervised OoD detection . These approaches usually define the test statistic based on the log-likelihood ( LL ) of a generative model fitted to Dtrain ( Choi et al. , 2018 ; Ren et al. , 2019 ; Ruff et al. , 2021 ) . However , as follows from our discussion above , there is no need to limit ourselves to LL-based statistics . For instance , we can define a test statistic for event sequences based on the rich literature on GoF testing for TPPs . We show in Section 6 that this often leads to more accurate anomaly detection compared to LL . Moreover , the difference between OoD detection and GoF testing is often overlooked . By drawing a clear distinction between the two , we can avoid some of the pitfalls encountered by other works ( Nalisnick et al. , 2019 ) , as we elaborate in Appendix A . The anomaly detection framework we outlined above can be applied to any type of data—such as images or time series—but in this work we mostly focus on continuous-time event data . This means that our main goal is to find an appropriate test statistic for variable-length continuous-time event sequences . In Section 3 , we take a look at existing GoF statistics for TPPs and analyze their limitations . Then in Section 4 , we propose a new test statistic that addresses these shortcomings and describe in more detail how it can be used for OoD detection . 3 Review of existing GoF test statistics for TPPs . Here , we consider a GoF test ( Equation 2 ) , where the goal is to determine whether an event sequence X = ( t1 , . . . , tN ) was generated by a known TPP Pmodel with compensator Λ∗ . We will return to the problem of OoD detection , where the data-generating distribution Pdata is unknown , in Section 4.2 . Many popular GoF tests for TPPs are based on the following result ( Ogata , 1988 ; Brown et al. , 2002 ) . Theorem 1 ( Random time change theorem ( Brown et al. , 2002 ) ) . A sequence X = ( t1 , . . . , tN ) is distributed according to a TPP with compensator Λ∗ on the interval [ 0 , V ] if and only if the sequence Z = ( Λ∗ ( t1 ) , . . . , Λ ∗ ( tN ) ) is distributed according to the standard Poisson process on [ 0 , Λ∗ ( V ) ] . Intuitively , Theorem 1 can be viewed as a TPP analogue of how the CDF of an arbitrary random variable over R transforms its realizations into samples from Uniform ( [ 0 , 1 ] ) . Similarly , the compensator Λ∗ converts a random event sequence X into a realization Z of the standard Poisson process ( SPP ) . Therefore , the problem of GoF testing for an arbitrary TPP reduces to testing whether the transformed sequence Z follows the SPP on [ 0 , Λ∗ ( T ) ] . In other words , we can define a GoF statistic for a TPP with compensator Λ∗ by ( 1 ) applying the compensator to X to obtain Z and ( 2 ) computing one of the existing GoF statistics for the SPP on the transformed sequence . This can also be generalized to marked TPPs ( where events can belong to one ofK classes ) by simply concatentating the transformed sequences Z ( k ) for each event type k ∈ { 1 , . . . , K } ( see Appendix D for details ) . SPP , i.e. , the Poisson process with constant intensity λ∗ ( t ) = 1 , is the most basic TPP one can conceive . However , as we will shortly see , existing GoF statistics even for this simple model have considerable shortcomings and can only detect a limited class of deviations from the SPP . More importantly , test statistics for general TPPs defined using the above recipe ( Theorem 1 ) inherit the limitations of the SPP statistics . For brevity , we denote the transformed arrival times as Z = ( v1 , . . . , vN ) = ( Λ∗ ( t1 ) , . . . , Λ∗ ( tN ) ) and the length of the transformed interval as V = Λ∗ ( T ) . One way to describe the generative process of an SPP is as follows ( Pasupathy , 2010 ) N |V ∼ Poisson ( V ) ui|N , V ∼ Uniform ( [ 0 , V ] ) for i = 1 , . . . , N. ( 4 ) An SPP realization Z = ( v1 , . . . , vN ) is obtained by sorting the ui ’ s in increasing order . This is equivalent to defining the arrival time vi as the i-th order statistic u ( i ) . We can also represent Z by the inter-event times ( w1 , . . . , wN+1 ) where wi = vi − vi−1 , assuming v0 = 0 and vN+1 = V . Barnard ( 1953 ) proposed a GoF test for the SPP based on the above description ( Equation 4 ) and the Kolmogorov–Smirnov ( KS ) statistic . The main idea of this approach is to check whether the arrival times v1 , . . . , vN are distributed uniformly in the [ 0 , V ] interval . For this , we compare F̂arr , the empirical CDF of the arrival times , with Farr ( u ) = u/V , the CDF of the Uniform ( [ 0 , V ] ) distribution . This can be done using the KS statistic on the arrival times ( KS arrival ) , defined as κarr ( Z ) = √ N · sup u∈ [ 0 , V ] |F̂arr ( u ) − Farr ( u ) | where F̂arr ( u ) = 1 N N∑ i=1 1 ( vi ≤ u ) . ( 5 ) Another popular GoF test for the SPP is based on the fact that the inter-event times wi are distributed according to the Exponential ( 1 ) distribution ( Cox , 1966 ) . The test compares F̂int , the empirical CDF of the inter-event times , and Fint ( u ) = 1− exp ( −u ) , the CDF of the Exponential ( 1 ) distribution . This leads to the KS statistic for the inter-event times ( KS inter-event ) κint ( Z ) = √ N · sup u∈ [ 0 , ∞ ) |F̂int ( u ) − Fint ( u ) | where F̂int ( u ) = 1 N + 1 N+1∑ i=1 1 ( wi ≤ u ) . ( 6 ) KS arrival and KS inter-event statistics are often presented as the go-to approach for testing the goodness-of-fit of the standard Poisson process ( Daley & Vere-Jones , 2003 ) . Combining them with Theorem 1 leads to simple GoF tests for arbitrary TPPs that are widely used to this day ( Gerhard et al. , 2011 ; Alizadeh et al. , 2013 ; Kim & Whitt , 2014 ; Tao et al. , 2018 ; Li et al. , 2018 ) . Limitations of the KS statistics . The KS statistics κarr ( Z ) and κint ( Z ) are only able to differentiate the SPP from a narrow class of alternative processes . For example , KS arrival only checks if the arrival times vi are distributed uniformly , conditioned on the event count N . But what if the observed N is itself extremely unlikely under the SPP ( Equation 4 ) ? KS inter-event can be similarly insensitive to the event count—removing all events V2 < vi ≤ V from an SPP realization Z will only result in just a single atypically large inter-event time wi , which changes the value of κint ( Z ) at most by 1N+1 . We demonstrate these limitations of κarr ( Z ) and κint ( Z ) in our experiments in Section 6.1 . Other failure modes of the KS statistics were described by Pillow ( 2009 ) . Note that ad-hoc fixes to the KS statistics do not address these problems . For example , combining multiple tests performed separately for the event count and arrival times using Fisher ’ s method ( Fisher , 1948 ; Cox , 1966 ) consistently decreases the accuracy , as we show in Appendix G. In the next section , we introduce a different test statistic that aims to address these shortcomings .

Summary: This paper focuses on the anomaly detection of continuous-time event sequences with temporal point processes and proposes to leverage the idea of goodness-of-fit testing to check whether sequences follow the in-distribution. Consider the limitation that the current statistics are not sensitive to the event number N, this paper proposes a new statistic called sum-of-squared-spacings (3S) to check the fitness of the sequence. The experimental results show the effectiveness of the proposed statistic for anomaly detection.