Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs

1 . Introduction . If political relations between two countries becomes more tense , will it affect the international trades between them ? If yes , which industries will bear the brunt ? Modeling the relevant events that can be temporarily affected by international relations is the key to answer this question . However , the issue of how to model these complicated temporal events is an intriguing question . A possible way is to embed events in a temporal knowledge graph , which is a graph-structured multi-relational database that stores an event in the form of a quadruple . Events are point processes and point process models , in the past , have been widely applied to many real-world applications such as the analysis of social networks [ Zhou et al. , 2013 ] , the prediction of recurrent user behaviors [ Du et al. , 2016 ] , and the estimation ∗ . Corresponding author of consumer behaviors in finance [ Bacry et al. , 2016 ] . The well known Poisson process [ Palm , 1943 ] is limited to modeling temporal events that occur independently of one another . Hawkes [ 1971 ] proposed a self-exciting point process , which is now known as the Hawkes process , which assumes that past events have an excitation effect on the likelihood of future events , and such excitation exponentially decays with time . This model has been shown to be effective in modeling earthquakes [ Ogata , 1998 ] . However , it is unable to capture some real-world patterns where past events of a different type may have inhibitory effects on future events , i.e. , a skateboard purchase may inhibit a bike purchase . To address this limitation , the neural Hawkes process [ Mei and Eisner , 2017 ] generalized the Hawkes process using recurrent neural networks with continuous state spaces such that past events can excite and inhibit future events in a complex and realistic way . Nevertheless , the neural Hawkes process is only capable of modeling event sequences with a small number of event types and fails to accurately capture the mutual influence in large-scale temporal multi-relational data . An example would be the evolving links in a dynamic graph sequence where the connections between nodes can be considered as different event types . The problem becomes even more challenging when the links are directional and labeled . In order to model the dynamics of directional and labeled links in a graph sequence , we develop a novel Graph Hawkes Process and apply it to large-scale temporal multi-relational databases , such as temporal knowledge graphs . Before introducing temporal knowledge graphs , we briefly review semantic knowledge graphs ( semantic KGs ) , which are multi-relational knowledge bases for storing factual information . Semantic KGs such as the Google Knowledge Graph [ Singhal , 2012 ] represent an event as a semantic triplet ( s , p , o ) in which s ( subject ) and o ( object ) are entities ( nodes ) , and p ( predicate ) is a directional labeled link ( edge ) . Latent feature models [ Ma et al. , 2018a , Nickel et al. , 2011 ] and graph feature models [ Minervini et al. , 2014 , Liu and Lü , 2010 ] are two popular approaches to develop statistical models for semantic KGs . However , in contrast to static multi-relational data in semantic KGs , relations between entities in many real-world scenarios are not fixed and may change over time . Such temporal events can be represented as a quadruple ( s , p , o , t ) by extending the semantic triplet with a time instance t describing when these events occurred . Further an event may last for a period of time . For example , ( John , lives in , Vancouver ) could be true for many time steps , and ( Alice , knows , John ) might be true always . We can simply discretize such an event into a sequence of time-stamped events to store it in the form of quadruples . Appendix A shows an example of a temporal KG . By considering time , the semantic KGs are augmented into temporal knowledge graphs ( tKGs ) , which creates the need for statistical learning that can capture dynamic relations between entities in tKGs . Modeling dynamic relations between entities over tKGs becomes more challenging than normal event streams since the number of event types is of order N2e ·Np , where Ne and Np are the number of entities and predicates respectively . Recent studies on tKGs reasoning focused on augmenting entity embeddings with time-dependent components in a low-dimensional space [ Kazemi et al. , 2019 , Sankar et al. , 2018 ] . However , the existing temporal KG models either lack a principled way to predict the occurrence time of future events or ignore the concurrent facts within the same time slice . In this paper , we propose a novel deep learning architecture to capture temporal dependencies on tKGs , called Graph Hawkes Neural Network ( GHNN ) . More specifically , our main contributions are : • We propose a Graph Hawkes Neural Network for predicting future events on large-scale tKGs . This is the first work that uses the Hawkes process to interpret and capture the underlying temporal dynamics of tKGs . • Different from the previous tKG models with discrete state spaces , we model the occurrence probability of an event in continuous time . In this way , our model can compute the probability of an event at an arbitrary timestamp , which considerably enhances model ’ s flexibility . • We analyze previous problematic evaluation metrics and propose a new ranking metric for link prediction on temporal knowledge graphs . • Compared to state-of-the-art time prediction models on tKGs , our approach can achieve more accurate results . 2 . Background and Related Work . 2.1 The Hawkes Process . The Hawkes process is a stochastic process for modeling sequential discrete events occurring in continuous time where the time intervals between neighboring events may not be identical . Moreover , the Hawkes process supposes that past events can temporarily excite future events , which is characterized via the intensity function . The intensity function λk ( t ) represents the expected number of events with type k in the interval of unit length . Thus , according to the survival analysis theory [ Aalen et al. , 2008 ] , the density function that an event with the type k occurs at ti is defined as pk ( ti ) = λk ( ti ) exp ( − ∫ ti tL ∑ k λk ( s ) ds ) , ( 1 ) where tL denotes the latest occurrence of any event without regarding its event type . 2.2 Future Prediction on Temporal Knowledge Graphs . Temporal knowledge graphs are multi-relational , directed graphs with labeled timestamped edges ( predicates ) between nodes ( entities ) . Each timestamped edge represents a specific event that is formed by a predicate edge p between a subject entity s and an object entity o at a timestamp t and is denoted by a quadruple ( es , ep , eo , t ) , where es , eo ∈ { 1 , ... , Ne } , ep ∈ { 1 , ... , Np } , t ∈ R+ . A tKG can therefore be represented as an ordered sequence of quadruples , E = { ei = ( esi , epi , eoi , ti ) } Ni=1 , where 0 ≤ t1 ≤ ... ≤ tn . A classic task in tKGs is to predict either a missing subject entity ( ? , epi , eoi , ti ) or a missing object entity ( esi , epi , ? , ti ) . While one aims to predict the missing links in the existing graphs in the context of a semantic knowledge graph , one wants to predict the future links at a future timestamp ti based on observed events that occurred before ti . Besides predicting what will happen in the future , another challenging problem is to predict when an event will happen , which is referred as the time prediction task . More concretely , one can precisely answer questions like : • Object prediction . Which country will Emmanuel Macron visit next ? • Subject prediction . Who is the wife of Emmanuel Macron ? • Time prediction . When will Emmanuel Macron tweet again ? Recently , several studies focussed on temporal knowledge graph reasoning . Esteban et al . [ 2016 ] introduced an event model for modeling the temporal evolution of KGs where the prediction of future events is based on the latent representations of the knowledge graph tensor and of the time-specific representations from the observed event tensor . Jiang et al . [ 2016 ] augmented existing static knowledge graph models with temporal consistency constraints such as temporal order information and formulated the time-aware inference as an Integer Linear Program problem . In addition , Ma et al . [ 2018b ] developed extensions of static knowledge graph models by adding a timestamp embedding to their score functions . Besides , Leblay and Chekol [ 2018 ] incorporated time presentations into score functions of several static KG models such as TransE [ Bordes et al. , 2013 ] and RESCAL [ Nickel et al. , 2011 ] in different ways . Additionally , Garćıa-Durán et al . [ 2018 ] suggested a straight forward extension of some existing static knowledge graph models that utilize a recurrent neural network ( RNN ) to encode predicates with temporal tokens derived by decomposing given timestamps . However , these models can not generalize to unseen timestamps because they only learn embeddings for observed timestamps . In contrast , LiTSEE [ Xu et al. , 2019 ] directly incorporates time as a scale into entity representations by utilizing the linear time series decomposition . Also , Know-Evolve [ Trivedi et al. , 2017 ] learns evolving entity representations using the Rayleigh process , being able to capture the dynamic characteristics of tKGs . Additionally , RE-Net [ Jin et al. , 2019 ] augmented the R-GCN model [ Schlichtkrull et al. , 2018 ] to tKGs and uses the order of history event for predicting the future . 3 . Notation . Throughout the following sections , ei denotes an event consisting of ( esi , epi , eoi ) where esi , eoi and epi written not in bold represent the subject entity , object entity and predicate of the event ei , respectively . Additionally , we use ti to denote the timestamp when the event ei occurred . Besides , esi , epi , eoi written in bold represent their embeddings . We denote vectors by bold lowercase letters , such as c , and matrices by bold capital Roman letters , e.g. , W. Additionally , subscripted bold letters denote specific vectors or matrices such as km . Moreover , scalar quantities , such as λk , are written without bold . We denote the upper limits of scalar quantities by capitalized scalars , for example , 1 ≤ n ≤ N . 4 . Our Model . In this section , we present the Graph Hawkes Neural Network ( GHNN ) for modeling sequences of discrete large-scale multi-relational graphs in continuous time . The GHNN consists of the following two major modules : • A neighborhood aggregation module for capturing the information from concurrent events that happened at the same timestamp . • A Graph Hawkes Process for modeling the occurrence of a future fact where we use a recurrent neural network to learn this temporal point process . We take the temporal knowledge graph as an example and show how our model deals with the link prediction task and the time prediction task . Besides , GHNN also learns latent representations specified for entities and predicates . In the rest of this section , we first define the relevant historical event sequence for each inference task , which is the input of GHNN , and then provide details on the proposed modules in GHNN .

Summary: This paper introduces a new model based on the Hawkes process to model the temporal knowledge base. The proposed method is based on previous work Mei & Eisen, 2017. The previous work target event prediction and time prediction, but in this paper, the task is subject/object prediction and time prediction. Furthermore, the author showed result improvement over strong baselines on several datasets.