Cardinality-Regularized Hawkes-Granger Model

1 Introduction . The Hawkes process [ 14 ] is one of the most popular models for analyzing temporal events in the machine learning ( ML ) community . It has been applied in a variety of application areas including analysis of human activities on social networks [ 23 , 36 , 39 , 15 , 38 , 22 , 2 , 44 ] , healthcare event analysis [ 6 , 12 ] , search query analysis [ 21 , 20 ] , and even water pipe maintenance [ 48 ] . In the studies of Hawkes processes , there have been two major milestones to date . One is the minorizationmaximization ( MM ) algorithm [ 16 ] . The other is Granger causal analysis through Hawkes processes . The first milestone was marked by Veen and Schoenberg [ 42 ] . Based on the intuition of branching process of earthquake aftershocks , they introduced the first MM-based maximum likelihood algorithm , which is often loosely referred to as EM ( expectation-maximization ) due to their similarity ( See Eq . ( 14 ) and the discussion below ) . As their paper pointed out , the standard gradient-based maximum likelihood estimation ( MLE ) of multivariate Hawkes processes suffered from numerical stability issues , limiting their applicability in practice . The second milestone was achieved by a few pioneers including Kim et al . [ 18 ] who first proposed an approach to Granger causal learning through the Hawkes process ; Zhou et al . [ 51 ] who introduced ` 1 regularized MLE of a multivariate Hawkes process in the context of Granger causal analysis ; and Eichler et al . [ 9 ] who theoretically established the equivalence between the Hawkes-based causality and the Granger causality [ 13 ] . Given these achievements and the well-known importance of sparsity in Granger causal learning [ 3 , 24 ] , the MM algorithm combined with a sparsity-enforcing regularizer would seem to be a promising path for Granger-causal analysis for stochastic events . This is especially true when the main interest is in analyzing causal triggering mechanism of event instances since the MM framework provides instance-wise triggering probabilities as a side product . Interestingly , however , the likelihood 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . function of the MM algorithm has a singularity that in fact prohibits any sparse solutions . Despite its significance , to date little attention has been paid to this issue in the ML community . In this paper , we provide a mathematically well-defined solution to sparse causal learning on temporal event data for the first time . Specifically , we introduce a novel cardinality-regularized MM framework for the Hawkes process . Although cardinality- or ` 0-regularization is known as an ideal way of enforcing sparsity , it makes optimization problems generally hard to solve . Even checking the feasibility is NP-complete [ 17 ] . We will show that the Hawkes process in fact has a semi-analytic solution under an ` 0-constraint , which is free from pathological issues due to the logarithmic singularity at zero , unlike existing ` 1-regularization approaches [ 51 , 46 ] . By leveraging the well-defined sparsity and the side product provided by the MM framework , we tackle the task of instance-wise causal diagnosis in real world domains . Since any event instance occurring in the past is a potential candidate cause of a given event of interest , achieving sufficient sparsity in the learned causal structure has proven critical in these applications . See also Fig . 1 for illustration . To the best of our knowledge , this work represents the first ` 0-regularized MM learning framework for the Hawkes process applied to simultaneous instance- and type-level causal analysis . 2 Related work . Related work relevant to this paper can be categorized into the following five categories . Graphical Granger models : For multivariate temporal data , sparse graphical modeling with a ( group ) lasso penalty [ 3 , 24 , 25 ] has been a standard approach to Granger causal analysis in the ML community . It replaced subtle independence tests with sparse matrix estimation in a systematic way , and significantly expanded the application field of Granger causal analysis . Their neural extension has also been proposed recently [ 40 ] . However , most of those studies use vector autoregressive ( VAR ) or state-space models ( including recurrent neural networks ( RNNs ) ) and thus assume as the input multivariate time-series data with a regular time interval . As a result , they can not handle event sequences without a certain aggregation/smoothing operation and thus it is not possible to analyze causal structure of event instances . We will show that event aggregation can seriously impact significantly impact the accuracy of causal analysis Sec . 6 . Conventional MLE : Since Hawkes ’ original proposal [ 14 ] , gradient-based MLE has been a mainstream approach to fitting Hawkes processes ( see , e.g . [ 34 ] ) . In particular , it has been used extensively in the field of seismology [ 32 ] . However , as Veen and Schoenberg [ 42 ] and Mark and Weber [ 26 ] convincingly demonstrated , gradient-based MLE tends to have numerical instability issues . Unless detailed prior knowledge on the data generating mechanism is available , which is indeed the case in seismology , its applicability to industrial stochastic event data can be limited . Stochastic optimization can be leveraged ( e.g . [ 31 ] ) , but it brings in extra variability to inference . In addition to conventional MLE , there are a few other methods to fit multivariate Hawkes processes , such as least squares [ 47 , 5 ] and moment matching [ 1 ] . They trade off the direct linkage to the likelihood principle for computational stability , and are out of our scope . This paper focuses on how to enhance the MM algorithm through cardinality regularization for a sparser and thus more interpretable solution . Neural Hawkes models : Neural point process models typically replace all or part of the terms of the Hawkes intensity function to capture nonlinear temporal dependencies [ 8 , 45 , 27 , 41 , 33 , 11 , 52 ] . However , most of the works either assume regular-interval time-series or lack the perspective of instance-wise causal triggering . The transformer architecture [ 50 ] , on the other hand , allows extracting instance-level information , but the self-attention filter essentially represents auto-correlation rather than Granger causality , making it inapplicable to event-causal diagnosis . Also , the positional encoding approach , originally introduced in a machine translation task , where the notion of time-stamp does not exist , implicitly assumes a regular time grid , partly sharing the same limitation as VAR- and RNNbased methods . Finally , it lacks systematic sparsity-enforcing mechanisms for better interpretability . Sparse MM algorithms : As discussed in Introduction , Zhou et al . [ 51 ] and Xu et al . [ 46 ] are the pioneers who first attempted to combine sparsity-enforcing regularizers with the MM algorithm . Unfortunately , due to the logarithmic singularity of the likelihood function , sparsity can only be achieved as a numerical artifact under ` 1- and ` 2,1 constraints . Sections 5 and 6 will demonstrate this point theoretically and empirically . Applications of Hawkes processes : Somewhat related to our motivation , there are a few recent studies that pay particular attention to the duality between microscopic ( instance-level ) and macroscopic ( type- or aggregate-level ) random processes . For instance , Wang et al . [ 43 ] discuss the relationship between a point process and a rate equation at an aggregate level , Li et al . [ 19 ] discuss macroscopic information diffusion through human interactions , and Zhang et al . [ 49 ] discuss failure propagation in compressor stations . None of them , however , explicitly performs simultaneous instance- and type-level causal analysis . As discussed in Sec . 6 , our framework can be viewed as a disciplined and effective solution to the issue of “ warning fatigue , ” which prevails across many industries [ 10 , 28 , 7 ] . 3 Preliminaries . This section provides the problem setting and recapitulates the basics of stochastic point processes . 3.1 Problem setting . Before getting into the formal definitions , let us take a brief look at a concrete use-case from cloud data center management ( See Sec . 6 for more details ) . In a data center , various computer and network devices continuously produce numerous event logs , which can be viewed as marked temporal event data with “ marks ” being the warning types . Due to inter-connectivity of the devices , one event from a device , such as a warning of type “ response time too long , ” may trigger many related events downstream . The more critical the original error , the more numerous the resulting set of events tend to be . Thus for a given event instance of interest ( called the target ) , it is desirable to find which event instances in the past are causally related to it . We call this task event causal diagnosis . There are two essential requirements in event causal diagnosis . The first one is the ability of providing instance-specific causal information in addition to the type-level causality . For instance , even if the i-th event type is on average likely to have a causal relationship with the j-th type , one specific instance of the i-th event type may have occurred spontaneously . A practical solution for event causal analysis must therefore perform type- and instance-level causal analysis simultaneously . The second requirement is the ability of providing sparse causal relationship . Since the number of event instances can be large , the capability of effectively shortlisting the candidates that may be causally related to a target event is essential in practice . To the best of our knowledge , our sparse Hawkes-Granger model is the first that meets these two requirements . We are given an event sequence of N + 1 event instances : D = { ( t0 , d0 ) , ( t1 , d1 ) , . . . , ( tN , dN ) } , ( 1 ) where tn and dn are the timestamp and the event type of the n-th event , respectively . The timestamps have been sorted in non-decreasing order t0 ≤ t1 ≤ · · · ≤ tN . There areD event types { 1 , 2 , . . . , D } with D N . We take the first time stamp t0 as the time origin . Hence , the remaining N instances are thought of as realization of random variables , given d0 . As a general rule , we use t or u as a free variable representing time while those with a subscript denote an instance . The main goal of event causal diagnosis , which is an unsupervised learning task , is to compute the instance triggering probabilities { qn , i } , where qn , i is the probability for the n-th event ( n = 1 , . . . , N ) instance to be triggered by the i-th event ( i = 0 , . . . , n ) . By definition , n ≥ i , and n∑ i=0 qn , i = 1 , ∀n ∈ { 1 , . . . , N } . ( 2 ) We call qn , n the self triggering ( or simply self ) probability . Note that providing { qn , i } amounts to providing weighted ranking of candidate triggering ( or causing ) events , where the weights sum to one . We wish to have as few candidates as possible with the aid of sparse causal learning .

Summary: This work introduces a l_0-regularized multivariate Hawkes process model for learning sparse Granger causality from temporal event data.  On the one hand, the authors show that the prior related work cannot infer the true sparsity using l_1 or l_{1,2} regularizers.  On the other aspect, most neural point processes for learning Granger causality, assume the equi-time-input, which prevents them from analyzing continuously observed events.  The authors used synthetic data to compare their introduced one with neural point processes, and the counterpart model using l_1 and l_{1,2} regularizers, and demonstrate the new method can infer the true sparse impact matrix behind the events.