Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs

1 INTRODUCTION . Reasoning , a process of inferring new knowledge from available facts , has long been considered an essential topic in AI research . Recently , reasoning on knowledge graphs ( KG ) has gained increasing interest ( Das et al. , 2017 ; Ren et al. , 2020 ; Hildebrandt et al. , 2020 ) . A knowledge graph is a graphstructured knowledge base that stores factual information in the form of triples ( s , p , o ) , e.g. , ( Alice , livesIn , Toronto ) . In particular , s ( subject ) and o ( object ) are expressed as nodes and p ( predicate ) as an edge type . Most knowledge graph models assume that the underlying graph is static . However , in the real world , facts and knowledge can change with time . For example , ( Alice , livesIn , Toronto ) becomes invalid after Alice moves to Vancouver . To accommodate time-evolving multi-relational data , temporal KGs have been introduced ( Boschee et al. , 2015 ) , where a temporal fact is represented as a quadruple by extending the static triple with a timestamp t indicating the triple is valid at t , i.e . ( Barack Obama , visit , India , 2010-11-06 ) . In this work , we focus on forecasting on temporal KGs , where we infer future events based on past events . Forecasting on temporal KGs can improve a plethora of downstream applications such as decision support in personalized health care and finance . The use cases often require the predictions made by the learning models to be interpretable , such that users can understand and trust the predictions . However , current machine learning approaches ( Trivedi et al. , 2017 ; Jin et al. , 2019 ) for temporal KG forecasting operate in a black-box fashion , where they design an embedding-based score function to estimate the plausibility of a quadruple . These models can not clearly show which evidence contributes to a prediction and lack explainability to the forecast , making them less suitable for many real-world applications . ∗Equal contribution . †Corresponding authors . Explainable approaches can generally be categorized into post-hoc interpretable methods and integrated transparent methods ( Došilović et al. , 2018 ) . Post-hoc interpretable approaches ( Montavon et al. , 2017 ; Ying et al. , 2019 ) aim to interpret the results of a black-box model , while integrated transparent approaches ( Das et al. , 2017 ; Qiu et al. , 2019 ; Wang et al. , 2019 ) have an explainable internal mechanism . In particular , most integrated transparent ( Lin et al. , 2018 ; Hildebrandt et al. , 2020 ) approaches for KGs employ path-based methods to derive an explicit reasoning path and demonstrate a transparent reasoning process . The path-based methods focus on finding the answer to a query within a single reasoning chain . However , many complicated queries require multiple supporting reasoning chains rather than just one reasoning path . Recent work ( Xu et al. , 2019 ; Teru et al. , 2019 ) has shown that reasoning over local subgraphs substantially boosts performance while maintaining interpretability . However , these explainable models can not be applied to temporal graph-structured data because they do not take time information into account . This work aims to design a transparent forecasting mechanism on temporal KGs that can generate informative explanations of the predictions . In this paper , we propose an explainable reasoning framework for forecasting future links on temporal knowledge graphs , xERTE , which employs a sequential reasoning process over local subgraphs . To answer a query in the form of ( subject eq , predicate pq , ? , timestamp tq ) , xERTE starts from the query subject , iteratively samples relevant edges of entities included in the subgraph and propagates attention along the sampled edges . After several rounds of expansion and pruning , the missing object is predicted from entities in the subgraph . Thus , the extracted subgraph can be seen as a concise and compact graphical explanation of the prediction . To guide the subgraph to expand in the direction of the query ’ s interest , we propose a temporal relational graph attention ( TRGA ) mechanism . We pose temporal constraints on passing messages to preserve the causal nature of the temporal data . Specifically , we update the time-dependent hidden representation of an entity ei at a timestamp t by attentively aggregating messages from its temporal neighbors that were linked with ei prior to t. We call such temporal neighbors as prior neighbors of ei . Additionally , we use an embedding module consisting of stationary entity embeddings and functional time encoding , enabling the model to capture both global structural information and temporal dynamics . Besides , we develop a novel representation update mechanism to mimic human reasoning behavior . When humans perform a reasoning process , their perceived profiles of observed entities will update , as new clues are found . Thus , it is necessary to ensure that all entities in a subgraph can receive messages from prior neighbors newly added to the subgraph . To this end , the proposed representation update mechanism enables every entity to receive messages from its farthest prior neighbors in the subgraph . The major contributions of this work are as follows . ( 1 ) We develop xERTE , the first explainable model for predicting future links on temporal KGs . The model is based on a temporal relational attention mechanisms that preserves the causal nature of the temporal multi-relational data . ( 2 ) Unlike most black-box embedding-based models , xERTE visualizes the reasoning process and provides an interpretable inference graph to emphasize important evidence . ( 3 ) The dynamical pruning procedure enables our model to perform reasoning on large-scale temporal knowledge graphs with millions of edges . ( 4 ) We apply our model for forecasting future links on four benchmark temporal knowledge graphs . The results show that our method achieves on average a better performance than current state-of-the-art methods , thus providing a new baseline . ( 5 ) We conduct a survey with 53 respondents to evaluate whether the extracted evidence is aligned with human understanding . 2 RELATED WORK . Representation learning is an expressive and popular paradigm underlying many KG models . The embedding-based approaches for knowledge graphs can generally be categorized into bilinear models ( Nickel et al. , 2011 ; Yang et al. , 2014 ; Ma et al. , 2018a ) , translational models ( Bordes et al. , 2013 ; Lv et al. , 2018 ; Sun et al. , 2019 ; Hao et al. , 2019 ) , and deep-learning models ( Dettmers et al. , 2017 ; Schlichtkrull et al. , 2018 ) . However , the above methods are not able to use rich dynamics available on temporal knowledge graphs . To this end , several studies have been conducted for temporal knowledge graph reasoning ( Garcı́a-Durán et al. , 2018 ; Ma et al. , 2018b ; Jin et al. , 2019 ; Goel et al. , 2019 ; Lacroix et al. , 2020 ; Han et al. , 2020a ; b ; Zhu et al. , 2020 ) . The published approaches are largely black-box , lacking the ability to interpret their predictions . Recently , several explainable reasoning methods for knowledge graphs have been proposed ( Das et al. , 2017 ; Xu et al. , 2019 ; Hildebrandt et al. , 2020 ; Teru et al. , 2019 ) . However , the above explainable methods can only deal with static KGs , while our model is designed for interpretable forecasting on temporal KGs . 3 PRELIMINARIES . Let E and P represent a finite set of entities and predicates , respectively . A temporal knowledge graph is a collection of timestamped facts written as quadruples . A quadruple q = ( es , p , eo , t ) represents a timestamped and labeled edge between a subject entity es ∈ E and an object entity eo ∈ E , where p ∈ P denotes the edge type ( predicate ) . The temporal knowledge graph forecasting task aims to predict unknown links at future timestamps based on observed past events . Definition 1 ( Temporal KG forecasting ) . Let F represent the set of all ground-truth quadruples , and let ( eq , pq , eo , tq ) ∈ F denote the target quadruple . Given a query ( eq , pq , ? , tq ) derived from the target quadruple and a set of observed prior facts O = { ( ei , pk , ej , tl ) ∈ F|tl < tq } , the temporal KG forecasting task is to predict the missing object entity eo . Specifically , we consider all entities in the set E as candidates and rank them by their likelihood to form a true quadruple together with the given subject-predicate-pair at timestamp tq1 . For a given query q = ( eq , pq , ? , tq ) , we build an inference graph Ginf to visualize the reasoning process . Unlike in temporal KGs , where a node represents an entity , each node in Ginf is an entitytimestamp pair . The inference graph is a directed graph in which a link points from a node with an earlier timestamp to a node with a later timestamp . Definition 2 ( Node in Inference Graph and its Temporal Neighborhood ) . Let E represent all entities , F denote all ground-truth quadruples , and let t represent a timestamp . A node in an inference graph Ginf is defined as an entity-timestamp pair v = ( ei , t ) , ei ∈ E . We define the set of one-hop prior neighbors of v as Nv= ( ei , t ) = { ( ej , t′ ) | ( ei , pk , ej , t′ ) ∈ F ∧ ( t′ < t ) } 2 . For simplicity , we denote one-hop prior neighbors as Nv . Similarly , we define the set of one-hop posterior neighbors of v as N v= ( ei , t ) = { ( ej , t′ ) | ( ej , pk , ei , t ) ∈ F ∧ ( t′ > t ) } . We denote them as N v for short . We provide an example in Figure 4 in the appendix to illustrate the inference graph . 4 OUR MODEL . We describe xERTE in a top-down fashion where we provide an overview in Section 4.1 and then explain each module from Section 4.2 to 4.6 . 4.1 SUBGRAPH REASONING PROCESS . Our model conducts the reasoning process on a dynamically expanded inference graph Ginf extracted from the temporal KG . We show a toy example in Figure 1 . Given query q = ( eq , pq , ? , tq ) , we initialize Ginf with node vq = ( eq , tq ) consisting of the query subject and the query time . The inference graph expands by sampling prior neighbors of vq . For example , suppose that ( eq , pk , ej , t′ ) is a valid quadruple where t′ < tq , we add the node v1 = ( ej , t′ ) into Ginf and link it with vq where the link is labeled with pk and points from vq to v1 . We use an embedding module to assign each node and predicate included in Ginf a temporal embedding that is shared across queries . The main goal of the embedding module is to let the nodes access query-independent information and get a broad view of the graph structure since the following temporal relational graph attention ( TRGA ) layer only performs query-dependent message passing locally . Next , we feed the inference graph into the TRGA layer that takes node embeddings and predicate embeddings as the input , produces a query-dependent representation for each node by passing messages on the small inference graph , and computes a query-dependent attention score for each edge . As explained in Section 4.7 , we propagate the attention of each node to its prior neighbors using the edge attention scores . Then we further expand Ginf by sampling the prior neighbors of the nodes in Ginf . The expansion will grow 1Throughout this work , we add reciprocal relations for every quadruple , i.e. , we add ( eo , p−1 , es , t ) for every ( es , p , eo , t ) . Hence , the restriction to predict object entities does not lead to a loss of generality . 2Prior neighbors linked with ei as subject entity , e.g. , ( ej , pk , ei , t ) , are covered using reciprocal relations . rapidly and cover almost all nodes after a few steps . To prevent the inference graph from exploding , we reduce the edge amount by pruning the edges that gain less attention . As the expansion and pruning iterate , Ginf allocates more and more information from the temporal KG . After running L inference steps , the model selects the entity with the highest attention score in Ginf as the prediction of the missing query object , where the inference graph itself serves as a graphical explanation .

Summary: This paper proposes xERTE, a comprehensive set of strategies (i.e. a temporal relational attention mechanism and a human-mimic representation update scheme, temporal neighborhood sampling and pruning, etc.) for link forecasting in temporal knowledge graphs (tKGs). Experiments on real-world tKGs show significant improvements and better explainability on KG forecasting.