Cardinality-Regularized Hawkes-Granger Model

1 Introduction . The Hawkes process [ 14 ] is one of the most popular models for analyzing temporal events in the machine learning ( ML ) community . It has been applied in a variety of application areas including analysis of human activities on social networks [ 23 , 36 , 39 , 15 , 38 , 22 , 2 , 44 ] , healthcare event analysis [ 6 , 12 ] , search query analysis [ 21 , 20 ] , and even water pipe maintenance [ 48 ] . In the studies of Hawkes processes , there have been two major milestones to date . One is the minorizationmaximization ( MM ) algorithm [ 16 ] . The other is Granger causal analysis through Hawkes processes . The first milestone was marked by Veen and Schoenberg [ 42 ] . Based on the intuition of branching process of earthquake aftershocks , they introduced the first MM-based maximum likelihood algorithm , which is often loosely referred to as EM ( expectation-maximization ) due to their similarity ( See Eq . ( 14 ) and the discussion below ) . As their paper pointed out , the standard gradient-based maximum likelihood estimation ( MLE ) of multivariate Hawkes processes suffered from numerical stability issues , limiting their applicability in practice . The second milestone was achieved by a few pioneers including Kim et al . [ 18 ] who first proposed an approach to Granger causal learning through the Hawkes process ; Zhou et al . [ 51 ] who introduced ` 1 regularized MLE of a multivariate Hawkes process in the context of Granger causal analysis ; and Eichler et al . [ 9 ] who theoretically established the equivalence between the Hawkes-based causality and the Granger causality [ 13 ] . Given these achievements and the well-known importance of sparsity in Granger causal learning [ 3 , 24 ] , the MM algorithm combined with a sparsity-enforcing regularizer would seem to be a promising path for Granger-causal analysis for stochastic events . This is especially true when the main interest is in analyzing causal triggering mechanism of event instances since the MM framework provides instance-wise triggering probabilities as a side product . Interestingly , however , the likelihood 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . function of the MM algorithm has a singularity that in fact prohibits any sparse solutions . Despite its significance , to date little attention has been paid to this issue in the ML community . In this paper , we provide a mathematically well-defined solution to sparse causal learning on temporal event data for the first time . Specifically , we introduce a novel cardinality-regularized MM framework for the Hawkes process . Although cardinality- or ` 0-regularization is known as an ideal way of enforcing sparsity , it makes optimization problems generally hard to solve . Even checking the feasibility is NP-complete [ 17 ] . We will show that the Hawkes process in fact has a semi-analytic solution under an ` 0-constraint , which is free from pathological issues due to the logarithmic singularity at zero , unlike existing ` 1-regularization approaches [ 51 , 46 ] . By leveraging the well-defined sparsity and the side product provided by the MM framework , we tackle the task of instance-wise causal diagnosis in real world domains . Since any event instance occurring in the past is a potential candidate cause of a given event of interest , achieving sufficient sparsity in the learned causal structure has proven critical in these applications . See also Fig . 1 for illustration . To the best of our knowledge , this work represents the first ` 0-regularized MM learning framework for the Hawkes process applied to simultaneous instance- and type-level causal analysis . 2 Related work . Related work relevant to this paper can be categorized into the following five categories . Graphical Granger models : For multivariate temporal data , sparse graphical modeling with a ( group ) lasso penalty [ 3 , 24 , 25 ] has been a standard approach to Granger causal analysis in the ML community . It replaced subtle independence tests with sparse matrix estimation in a systematic way , and significantly expanded the application field of Granger causal analysis . Their neural extension has also been proposed recently [ 40 ] . However , most of those studies use vector autoregressive ( VAR ) or state-space models ( including recurrent neural networks ( RNNs ) ) and thus assume as the input multivariate time-series data with a regular time interval . As a result , they can not handle event sequences without a certain aggregation/smoothing operation and thus it is not possible to analyze causal structure of event instances . We will show that event aggregation can seriously impact significantly impact the accuracy of causal analysis Sec . 6 . Conventional MLE : Since Hawkes ’ original proposal [ 14 ] , gradient-based MLE has been a mainstream approach to fitting Hawkes processes ( see , e.g . [ 34 ] ) . In particular , it has been used extensively in the field of seismology [ 32 ] . However , as Veen and Schoenberg [ 42 ] and Mark and Weber [ 26 ] convincingly demonstrated , gradient-based MLE tends to have numerical instability issues . Unless detailed prior knowledge on the data generating mechanism is available , which is indeed the case in seismology , its applicability to industrial stochastic event data can be limited . Stochastic optimization can be leveraged ( e.g . [ 31 ] ) , but it brings in extra variability to inference . In addition to conventional MLE , there are a few other methods to fit multivariate Hawkes processes , such as least squares [ 47 , 5 ] and moment matching [ 1 ] . They trade off the direct linkage to the likelihood principle for computational stability , and are out of our scope . This paper focuses on how to enhance the MM algorithm through cardinality regularization for a sparser and thus more interpretable solution . Neural Hawkes models : Neural point process models typically replace all or part of the terms of the Hawkes intensity function to capture nonlinear temporal dependencies [ 8 , 45 , 27 , 41 , 33 , 11 , 52 ] . However , most of the works either assume regular-interval time-series or lack the perspective of instance-wise causal triggering . The transformer architecture [ 50 ] , on the other hand , allows extracting instance-level information , but the self-attention filter essentially represents auto-correlation rather than Granger causality , making it inapplicable to event-causal diagnosis . Also , the positional encoding approach , originally introduced in a machine translation task , where the notion of time-stamp does not exist , implicitly assumes a regular time grid , partly sharing the same limitation as VAR- and RNNbased methods . Finally , it lacks systematic sparsity-enforcing mechanisms for better interpretability . Sparse MM algorithms : As discussed in Introduction , Zhou et al . [ 51 ] and Xu et al . [ 46 ] are the pioneers who first attempted to combine sparsity-enforcing regularizers with the MM algorithm . Unfortunately , due to the logarithmic singularity of the likelihood function , sparsity can only be achieved as a numerical artifact under ` 1- and ` 2,1 constraints . Sections 5 and 6 will demonstrate this point theoretically and empirically . Applications of Hawkes processes : Somewhat related to our motivation , there are a few recent studies that pay particular attention to the duality between microscopic ( instance-level ) and macroscopic ( type- or aggregate-level ) random processes . For instance , Wang et al . [ 43 ] discuss the relationship between a point process and a rate equation at an aggregate level , Li et al . [ 19 ] discuss macroscopic information diffusion through human interactions , and Zhang et al . [ 49 ] discuss failure propagation in compressor stations . None of them , however , explicitly performs simultaneous instance- and type-level causal analysis . As discussed in Sec . 6 , our framework can be viewed as a disciplined and effective solution to the issue of “ warning fatigue , ” which prevails across many industries [ 10 , 28 , 7 ] . 3 Preliminaries . This section provides the problem setting and recapitulates the basics of stochastic point processes . 3.1 Problem setting . Before getting into the formal definitions , let us take a brief look at a concrete use-case from cloud data center management ( See Sec . 6 for more details ) . In a data center , various computer and network devices continuously produce numerous event logs , which can be viewed as marked temporal event data with “ marks ” being the warning types . Due to inter-connectivity of the devices , one event from a device , such as a warning of type “ response time too long , ” may trigger many related events downstream . The more critical the original error , the more numerous the resulting set of events tend to be . Thus for a given event instance of interest ( called the target ) , it is desirable to find which event instances in the past are causally related to it . We call this task event causal diagnosis . There are two essential requirements in event causal diagnosis . The first one is the ability of providing instance-specific causal information in addition to the type-level causality . For instance , even if the i-th event type is on average likely to have a causal relationship with the j-th type , one specific instance of the i-th event type may have occurred spontaneously . A practical solution for event causal analysis must therefore perform type- and instance-level causal analysis simultaneously . The second requirement is the ability of providing sparse causal relationship . Since the number of event instances can be large , the capability of effectively shortlisting the candidates that may be causally related to a target event is essential in practice . To the best of our knowledge , our sparse Hawkes-Granger model is the first that meets these two requirements . We are given an event sequence of N + 1 event instances : D = { ( t0 , d0 ) , ( t1 , d1 ) , . . . , ( tN , dN ) } , ( 1 ) where tn and dn are the timestamp and the event type of the n-th event , respectively . The timestamps have been sorted in non-decreasing order t0 ≤ t1 ≤ · · · ≤ tN . There areD event types { 1 , 2 , . . . , D } with D N . We take the first time stamp t0 as the time origin . Hence , the remaining N instances are thought of as realization of random variables , given d0 . As a general rule , we use t or u as a free variable representing time while those with a subscript denote an instance . The main goal of event causal diagnosis , which is an unsupervised learning task , is to compute the instance triggering probabilities { qn , i } , where qn , i is the probability for the n-th event ( n = 1 , . . . , N ) instance to be triggered by the i-th event ( i = 0 , . . . , n ) . By definition , n ≥ i , and n∑ i=0 qn , i = 1 , ∀n ∈ { 1 , . . . , N } . ( 2 ) We call qn , n the self triggering ( or simply self ) probability . Note that providing { qn , i } amounts to providing weighted ranking of candidate triggering ( or causing ) events , where the weights sum to one . We wish to have as few candidates as possible with the aid of sparse causal learning .

Summary: The authors a sparse Granger-causal learning framework for temporal event data following Hawkes process. The method utilizes the L0-regularized minorization-maximization (MM) learning framework, which remedies the issues of singularity in existing approaches.  The authors validated the proposed framework with two real use-cases, one from the power grid and the other from the cloud data center management domain.  The contributions of this paper seem to be as follows: (1) The proposed method is the ﬁrst L0-regularized MM learning framework for the Hawkes process applied to simultaneous instance- and type-level causal analysis. (2) The proposed method can explicitly perform simultaneous instance- and type-level causal analysis.