Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs

1 INTRODUCTION . Reasoning , a process of inferring new knowledge from available facts , has long been considered an essential topic in AI research . Recently , reasoning on knowledge graphs ( KG ) has gained increasing interest ( Das et al. , 2017 ; Ren et al. , 2020 ; Hildebrandt et al. , 2020 ) . A knowledge graph is a graphstructured knowledge base that stores factual information in the form of triples ( s , p , o ) , e.g. , ( Alice , livesIn , Toronto ) . In particular , s ( subject ) and o ( object ) are expressed as nodes and p ( predicate ) as an edge type . Most knowledge graph models assume that the underlying graph is static . However , in the real world , facts and knowledge can change with time . For example , ( Alice , livesIn , Toronto ) becomes invalid after Alice moves to Vancouver . To accommodate time-evolving multi-relational data , temporal KGs have been introduced ( Boschee et al. , 2015 ) , where a temporal fact is represented as a quadruple by extending the static triple with a timestamp t indicating the triple is valid at t , i.e . ( Barack Obama , visit , India , 2010-11-06 ) . In this work , we focus on forecasting on temporal KGs , where we infer future events based on past events . Forecasting on temporal KGs can improve a plethora of downstream applications such as decision support in personalized health care and finance . The use cases often require the predictions made by the learning models to be interpretable , such that users can understand and trust the predictions . However , current machine learning approaches ( Trivedi et al. , 2017 ; Jin et al. , 2019 ) for temporal KG forecasting operate in a black-box fashion , where they design an embedding-based score function to estimate the plausibility of a quadruple . These models can not clearly show which evidence contributes to a prediction and lack explainability to the forecast , making them less suitable for many real-world applications . ∗Equal contribution . †Corresponding authors . Explainable approaches can generally be categorized into post-hoc interpretable methods and integrated transparent methods ( Došilović et al. , 2018 ) . Post-hoc interpretable approaches ( Montavon et al. , 2017 ; Ying et al. , 2019 ) aim to interpret the results of a black-box model , while integrated transparent approaches ( Das et al. , 2017 ; Qiu et al. , 2019 ; Wang et al. , 2019 ) have an explainable internal mechanism . In particular , most integrated transparent ( Lin et al. , 2018 ; Hildebrandt et al. , 2020 ) approaches for KGs employ path-based methods to derive an explicit reasoning path and demonstrate a transparent reasoning process . The path-based methods focus on finding the answer to a query within a single reasoning chain . However , many complicated queries require multiple supporting reasoning chains rather than just one reasoning path . Recent work ( Xu et al. , 2019 ; Teru et al. , 2019 ) has shown that reasoning over local subgraphs substantially boosts performance while maintaining interpretability . However , these explainable models can not be applied to temporal graph-structured data because they do not take time information into account . This work aims to design a transparent forecasting mechanism on temporal KGs that can generate informative explanations of the predictions . In this paper , we propose an explainable reasoning framework for forecasting future links on temporal knowledge graphs , xERTE , which employs a sequential reasoning process over local subgraphs . To answer a query in the form of ( subject eq , predicate pq , ? , timestamp tq ) , xERTE starts from the query subject , iteratively samples relevant edges of entities included in the subgraph and propagates attention along the sampled edges . After several rounds of expansion and pruning , the missing object is predicted from entities in the subgraph . Thus , the extracted subgraph can be seen as a concise and compact graphical explanation of the prediction . To guide the subgraph to expand in the direction of the query ’ s interest , we propose a temporal relational graph attention ( TRGA ) mechanism . We pose temporal constraints on passing messages to preserve the causal nature of the temporal data . Specifically , we update the time-dependent hidden representation of an entity ei at a timestamp t by attentively aggregating messages from its temporal neighbors that were linked with ei prior to t. We call such temporal neighbors as prior neighbors of ei . Additionally , we use an embedding module consisting of stationary entity embeddings and functional time encoding , enabling the model to capture both global structural information and temporal dynamics . Besides , we develop a novel representation update mechanism to mimic human reasoning behavior . When humans perform a reasoning process , their perceived profiles of observed entities will update , as new clues are found . Thus , it is necessary to ensure that all entities in a subgraph can receive messages from prior neighbors newly added to the subgraph . To this end , the proposed representation update mechanism enables every entity to receive messages from its farthest prior neighbors in the subgraph . The major contributions of this work are as follows . ( 1 ) We develop xERTE , the first explainable model for predicting future links on temporal KGs . The model is based on a temporal relational attention mechanisms that preserves the causal nature of the temporal multi-relational data . ( 2 ) Unlike most black-box embedding-based models , xERTE visualizes the reasoning process and provides an interpretable inference graph to emphasize important evidence . ( 3 ) The dynamical pruning procedure enables our model to perform reasoning on large-scale temporal knowledge graphs with millions of edges . ( 4 ) We apply our model for forecasting future links on four benchmark temporal knowledge graphs . The results show that our method achieves on average a better performance than current state-of-the-art methods , thus providing a new baseline . ( 5 ) We conduct a survey with 53 respondents to evaluate whether the extracted evidence is aligned with human understanding . 2 RELATED WORK . Representation learning is an expressive and popular paradigm underlying many KG models . The embedding-based approaches for knowledge graphs can generally be categorized into bilinear models ( Nickel et al. , 2011 ; Yang et al. , 2014 ; Ma et al. , 2018a ) , translational models ( Bordes et al. , 2013 ; Lv et al. , 2018 ; Sun et al. , 2019 ; Hao et al. , 2019 ) , and deep-learning models ( Dettmers et al. , 2017 ; Schlichtkrull et al. , 2018 ) . However , the above methods are not able to use rich dynamics available on temporal knowledge graphs . To this end , several studies have been conducted for temporal knowledge graph reasoning ( Garcı́a-Durán et al. , 2018 ; Ma et al. , 2018b ; Jin et al. , 2019 ; Goel et al. , 2019 ; Lacroix et al. , 2020 ; Han et al. , 2020a ; b ; Zhu et al. , 2020 ) . The published approaches are largely black-box , lacking the ability to interpret their predictions . Recently , several explainable reasoning methods for knowledge graphs have been proposed ( Das et al. , 2017 ; Xu et al. , 2019 ; Hildebrandt et al. , 2020 ; Teru et al. , 2019 ) . However , the above explainable methods can only deal with static KGs , while our model is designed for interpretable forecasting on temporal KGs . 3 PRELIMINARIES . Let E and P represent a finite set of entities and predicates , respectively . A temporal knowledge graph is a collection of timestamped facts written as quadruples . A quadruple q = ( es , p , eo , t ) represents a timestamped and labeled edge between a subject entity es ∈ E and an object entity eo ∈ E , where p ∈ P denotes the edge type ( predicate ) . The temporal knowledge graph forecasting task aims to predict unknown links at future timestamps based on observed past events . Definition 1 ( Temporal KG forecasting ) . Let F represent the set of all ground-truth quadruples , and let ( eq , pq , eo , tq ) ∈ F denote the target quadruple . Given a query ( eq , pq , ? , tq ) derived from the target quadruple and a set of observed prior facts O = { ( ei , pk , ej , tl ) ∈ F|tl < tq } , the temporal KG forecasting task is to predict the missing object entity eo . Specifically , we consider all entities in the set E as candidates and rank them by their likelihood to form a true quadruple together with the given subject-predicate-pair at timestamp tq1 . For a given query q = ( eq , pq , ? , tq ) , we build an inference graph Ginf to visualize the reasoning process . Unlike in temporal KGs , where a node represents an entity , each node in Ginf is an entitytimestamp pair . The inference graph is a directed graph in which a link points from a node with an earlier timestamp to a node with a later timestamp . Definition 2 ( Node in Inference Graph and its Temporal Neighborhood ) . Let E represent all entities , F denote all ground-truth quadruples , and let t represent a timestamp . A node in an inference graph Ginf is defined as an entity-timestamp pair v = ( ei , t ) , ei ∈ E . We define the set of one-hop prior neighbors of v as Nv= ( ei , t ) = { ( ej , t′ ) | ( ei , pk , ej , t′ ) ∈ F ∧ ( t′ < t ) } 2 . For simplicity , we denote one-hop prior neighbors as Nv . Similarly , we define the set of one-hop posterior neighbors of v as N v= ( ei , t ) = { ( ej , t′ ) | ( ej , pk , ei , t ) ∈ F ∧ ( t′ > t ) } . We denote them as N v for short . We provide an example in Figure 4 in the appendix to illustrate the inference graph . 4 OUR MODEL . We describe xERTE in a top-down fashion where we provide an overview in Section 4.1 and then explain each module from Section 4.2 to 4.6 . 4.1 SUBGRAPH REASONING PROCESS . Our model conducts the reasoning process on a dynamically expanded inference graph Ginf extracted from the temporal KG . We show a toy example in Figure 1 . Given query q = ( eq , pq , ? , tq ) , we initialize Ginf with node vq = ( eq , tq ) consisting of the query subject and the query time . The inference graph expands by sampling prior neighbors of vq . For example , suppose that ( eq , pk , ej , t′ ) is a valid quadruple where t′ < tq , we add the node v1 = ( ej , t′ ) into Ginf and link it with vq where the link is labeled with pk and points from vq to v1 . We use an embedding module to assign each node and predicate included in Ginf a temporal embedding that is shared across queries . The main goal of the embedding module is to let the nodes access query-independent information and get a broad view of the graph structure since the following temporal relational graph attention ( TRGA ) layer only performs query-dependent message passing locally . Next , we feed the inference graph into the TRGA layer that takes node embeddings and predicate embeddings as the input , produces a query-dependent representation for each node by passing messages on the small inference graph , and computes a query-dependent attention score for each edge . As explained in Section 4.7 , we propagate the attention of each node to its prior neighbors using the edge attention scores . Then we further expand Ginf by sampling the prior neighbors of the nodes in Ginf . The expansion will grow 1Throughout this work , we add reciprocal relations for every quadruple , i.e. , we add ( eo , p−1 , es , t ) for every ( es , p , eo , t ) . Hence , the restriction to predict object entities does not lead to a loss of generality . 2Prior neighbors linked with ei as subject entity , e.g. , ( ej , pk , ei , t ) , are covered using reciprocal relations . rapidly and cover almost all nodes after a few steps . To prevent the inference graph from exploding , we reduce the edge amount by pruning the edges that gain less attention . As the expansion and pruning iterate , Ginf allocates more and more information from the temporal KG . After running L inference steps , the model selects the entity with the highest attention score in Ginf as the prediction of the missing query object , where the inference graph itself serves as a graphical explanation .

Summary: Authors have presented a method to forecast future links on temporal knowledge graphs (KGs). They use attention mechanisms to extract a query-dependent subgraph. According to the authors, this extracted subgraph provides a graphical explanation of the prediction. Authors have performed an ablation study to denote the effect of different components (e.g., updating the representation of nodes, time encoding, sampling strategy) in their method. They have tested the performance of their approach on 3 datasets and have shown that their approach outperforms other baselines in terms of Hits and MRR.