Deep Interaction Processes for Time-Evolving Graphs

1 INTRODUCTION . Representation learning over graph data has become a core machine learning task with a wide range of applications including e-commerce , finance , social networks , and bioinformatics . Various neural graph representations such as ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ; Wang et al. , 2016 ; Kipf & Welling , 2017 ; Defferrard et al. , 2016 ; Scarselli et al. , 2009 ; Ying et al. , 2018 ; Hamilton et al. , 2017b ; Monti et al. , 2017 ; Den Berg et al. , 2017 ) have been proposed to learn from static graph data and successfully used for downstream tasks ( e.g. , classification ) . Graph data , however , are often dynamic in practice ; nodes and interactions between them can grow and shrink . A straightforward approach to handle dynamic graphs is to compress them into one or several static graphs . The drawbacks of this approach are multifold ; we not only blur temporal structural information but also miss time information that can be critical for real-world applications . An illustrative example is given in figure 1 . To handle continuous time-evolving graph , we can approximate a by a sequence of snapshot graphs , each of which includes all interactions that occur during a user-specified discrete-time interval , as shown in ( Goyal et al. , 2018 ; Leskovec et al. , 2007 ; Zhou et al. , 2018 ; Sankar et al. , 2019 ) . This treatment reduces time resolution and it is tricky to specify the appropriate aggregation granularity . To avoid these problems , Nguyen et al . ( 2018 ) proposed continuous-time dynamic networks ( CTDNE ) that generalize deep walk methods to learn time-dependent network embedding . As a transductive method , CTDNE can not handle the growth of new nodes . Dai et al . ( 2016 ) applied temporal point processes to model time-evolving graphs and , as a nonparametric Bayesian approach , their approach can naturally cope with the growth of new nodes and interactions . They used recurrent neural networks ( RNNs ) to define an intensity function in temporal point processes . These RNN models are shallow and one-step unrolled , making it easy to compute but relatively limited in modeling power . Trivedi et al . ( 2019 ) extended this approach by modeling two-time scale and adopting temporal-attention mechanism . In this paper we present a powerful deep neural approach that models continuous time-evolving graphs at multiple time resolutions based on a temporal point process framework . We name the new approach deep interaction processes ( DIPs ) . To model the dependency between latent dynamic representations of each node , we define a mixture of temporal cascades in which a node ’ s neural representation depends on not only this node ’ s previous representations but also the previous representations of related nodes that have interacted with this node . We generalize LSTM on this temporal cascade mixture and introduce novel time gates to model time intervals between interactions . Furthermore , We introduce a selection mechanism that gives important nodes large influence in both k−depth subgraphs of nodes in an interaction . To obtain representations from fine-to-coarse time-resolutions , we stack our neural representations in several layers and fuse them based on attention . Based on the temporal point process framework , our approach can naturally handle growth of graph nodes and interactions , making it inductive . The rest of the paper is organized as follows . In Section 2 we give background on temporal point processes and in Section 3 we present the new DIP approach . In Section 4 we discuss related works . In Section 5 we report experimental results on multiple interaction prediction and classification tasks including an important real-world anti-fraud financial application , demonstrating superior performance of the new approach over the alternatives . 2 TEMPORAL POINT PROCESSES . We first describe temporal point processes ( a class of nonparametric Bayesian models ) that our approach is based on . Specifically , a temporal point process is a stochastic process that generates a sequence of discrete events localized at times { ti } Ni=1 in any given observed time window [ 0 , T ] , where N is the number of events . An important way to characterize temporal point processes is via the conditional intensity function λ ( t|Ht ) -the stochastic model for the next event time t given all historical events before time t , denoted as Ht = { ti|ti < t } . Formally , within a small time window [ t , t + dt ) , λ ( t|Ht ) dt is the probability for the occurrence for a new event given the Ht : λ ( t|Ht ) dt = P { event in [ t , t+ dt ) |Ht } . From the survival analysis theory ( Aalen et al. , 2008 ) , given the times of the past events { t1 , t2 , . . . , ti } , the conditional density that an event occurs at ti+1 is given as follows : p ( ti+1|Hti+1 ) = λ ( ti+1|Hti+1 ) exp { − ∫ ti+1 ti λ ( t|Ht ) dt } , where the exponential part in the above equation means the conditional probability that no event happens during [ ti , ti+1 ) . The functional forms of the conditional intensity function λ ( t|Ht ) can represent certain forms of dependencies of the historical events . For instance , for Poisson processes ( Kingman , 2005 ) we set λ to be constant – making the assumption that the process is stationary and the temporal events in history are independent of each other . For classical Hawkes processes ( Hawkes , 1971 ) , the intensity function λ is often set to be a sum of multiple exponential functions , assuming that the mutual excitation among events is positive , additive over the past events , and exponentially decaying with time . Mei & Eisner ( 2017a ) removed these limiting assumptions using LSTM to learn λ from data . 3 DEEP INTERACTION PROCESSES . In this section , we present a new neural non-parametric Bayesian approach over continuous-time evolving graphs . First , we present a temporal dependency graph that is a mixture of the temporal cascades , to model interdependence between graph nodes ( as well as latent node representations ) . Then we present a novel deep model to learn dynamic node representations in the temporal dependency graph . This model naturally generalizes a chain-structured LSTM to a temporal Graph-structured LSTM equipped with time gates to handle interaction with irregular time intervals . Furthermore , we propose the FUSION and SELECTION methods to enhance the dynamic interactive nodes representation . Given the dynamic node representations , we define deep interaction processes that model potential interactions between any two nodes over time and we finally layout the maximum likelihood estimation method to optimize it . A toy example is shown in Figure 2 to introduce the corresponding concepts and the whole procedure of computing enhanced dynamic representation is given in Figure 3 . 3.1 TEMPORAL DEPENDENCY GRAPH . Consider a collection of people-movie interactions at different time points ( e.g. , David saw the movie The Skull at t6 . ) as shown in figure 2 ( a ) . The people and movies form a temporal dependency graph in which each person or movie is a node and interactions happen over time ( in figure 2 ( b ) ) . After one interaction occurs , we update the neural representation of the two nodes linked to this interaction ; e.g. , right after time t6 , we update the representations for David and movie The Skull . The new neural representation of David depends on both his current and previous interactions – as a result , depending on the representations of the two nodes associated with the previous interaction . This naturally forms a dependency cascade . Similarly we can obtain a dependency cascade for Lucy ’ s representations . Because of the common movies David and Lucy saw , their dependency cascades overlap and form a cascade mixture . Formally , we denote a dynamic interaction or link at time t by lu , v , t where u and v are two nodes associated with this interaction . We denote the node u at time t by u ( t ) and the two nodes associated with u ’ s precedent interaction at time t− as u1 ( t ) and u2 ( t ) . For later usage , we denote the k−depth temporal dependency subgraph for u ( t ) as subgraph ( u ( t ) , k ) as shown in figure 2 ( b ) . 3.2 DIP NEURAL UNIT AND DYNAMIC REPRESENTATION . Now we present a novel neural unit to update dynamic latent representations of nodes over the temporal dependency graph which is illustrated in figure 2 ( c ) and figure 2 ( d ) . First , let us denote node u ’ s features or embedding ( i.e. , a static representation jointly learned from data ) at time t by xu ( t ) and denote features of interaction l by xl . The interaction feature can be empty if the interaction contains only the temporal information . The concatenation of xu ( t ) and xl is denoted by x̂u ( t ) . Let ∆ ( u , t ) = t− t− be the time interval between two consecutive interactions involving u at time t and t− . Our neural unit , i.e , DIP unit , generalizes a chain-structured LSTM unit to depict the temporal dependency on graph data . The DIP unit has an update gate s , an input gate z , an output gate o and two forget gates f over x̂u ( t ) , dynamic representation of ui ( t ) , i.e , hui ( t ) and cell states cui ( t ) ( i = 1 , 2 ) as shown in figure 2 ( c ) . Additionally , we introduce time gates g to capture the impacts of irregular time interval ∆ ( u , t ) . Specifically , hu ( t ) and cu ( t ) are updated as follows : zu ( t ) = σ ( Wzx̂u ( t ) + N∑ i=1 Rzihui ( t ) + bz ) ou ( t ) = σ ( Wox̂u ( t ) + N∑ i=1 Roihui ( t ) + bo ) su ( t ) = tanh ( Wsx̂u ( t ) + N∑ i=1 Rsihui ( t ) + bs ) fu ( t ) , ui ( t ) = σ ( Wfi x̂u ( t ) + Rfihui ( t ) + bfi ) gu ( t ) , ui ( t ) = σ ( Wgi x̂u ( t ) + Rgihui ( t ) + Mgi∆ ( u , t ) + bgi ) cu ( t ) = zu ( t ) su ( t ) + ∑N i=1 fu ( t ) , ui ( t ) cui ( t ) gu ( t ) , ui ( t ) hu ( t ) = ou ( t ) tanh ( cu ( t ) ) ( 1 ) where σ , tanh and represent the sigmoid function , the hyperbolic tangent function , and the Hadamard product ( pointwise multiplication ) , respectively , and parameters in the unit including the recurrent weights Rzi , Roi , Rsi , Rfi and Rgi , the projection matrices Wz , Wo , Ws , Wfi and Wgi , the bias vectors bz , bo , bs , bfi and bgi and the time weight matrix Mgi are learned from data . For convenience , we use DIP-UNIT ( · ) to summarize the above equations , then the dynamic representation of node u ( t ) is given as follows : hu ( t ) , cu ( t ) = DIP-UNIT ( x̂u ( t ) , cu1 ( t ) , cu2 ( t ) , hu1 ( t ) , hu2 ( t ) , ∆ ( u , t ) , Θ ) ( 2 ) where Θ represent all the parameters . Considering the computational cost in practice , when obtaining nodes ’ dynamic representation given an happened interaction , we only utilize history information in the subgraph ( u ( t ) , k ) as illustrated in figure 2 ( d ) , which is similar to a chain-structured LSTM training unfolded with the max k steps . The k is a hyper-parameter .

Summary: The paper focuses on the problem of modeling interaction processes over dynamically evolving graphs and perform inference tasks such future interaction prediction and interaction classification. Specifically, the paper proposes a temporal point process based formulation to model the interaction dynamics where the conditional intensity function is parameterized by a recurrent network. With an occurrence of any event, the recurrent architecture updates the embeddings of the nodes involved in that event which then affects the intensity function and hence the likelihood of future events. The paper uses intensity based likelihood to train for future interaction prediction task while cross-entropy based loss for classification task. The paper demonstrates the efficacy of the method through experiments across multiple datasets and compare against representative baselines and further provides ablation analysis for the proposed architecture.