OxKBC: Outcome Explanation for Factorization Based Knowledge Base Completion

and YAGO ) are based on tensor factorization ( TF ) , e.g , DistMult , ComplEx . While they produce good results , they can not expose any rationale behind their predictions , potentially reducing the trust of a user in the outcome of the model . Previous works have explored creating an inherently explainable model , e.g . Neural Theorem Proving ( NTP ) , DeepPath , MINERVA , but explainability in them comes at the cost of performance . Others have tried to create an auxiliary explainable model having high fidelity with the underlying TF model , but unfortunately , they do not scale well to large KBs . In this work , we propose OXKBC – an Outcome eXplanation engine for KBC , which provides a post-hoc explanation for every triple inferred by a ( uninterpretable ) factorization based model . It first augments the underlying Knowledge Graph by introducing weighted edges between entities based on their similarity given by the underlying model . It then defines a notion of human-understandable explanation paths along with a language to generate them . Depending on the edges , the paths are aggregated into second–order templates for further selection . The best template with its grounding is then selected by a neural selection module that is trained with minimal supervision by a novel loss function . Experiments over Mechanical Turk demonstrate that users overwhelmingly find our explanations more trustworthy compared to rule mining . 1 . Introduction . Knowledge bases ( KBs ) store facts in the form of triples r ( s , o ) , which indicate that the head entity s is related to the tail entity o via the relation r , e.g. , capitalOf ( WashingtonDC , USA ) . These KBs are typically incomplete [ Bollacker et al. , 2008 ] and the task of KB Completion ( KBC ) aims to infer new facts from a given KB . Many state-of-the-art models for KBC on large-scale KBs use tensor factorization ( TF ) [ Sedghi and Sabharwal , 2018 , Lacroix et al. , 2018 , Jain et al. , 2018 , Xue et al. , 2018 ] . They learn embeddings for each entity and relation in a continuous vector space and define a scoring function to evaluate the validity of a given triple . At test time , given a query r ( s , ? ) they output a ranked list of entities o by sorting the model scores for r ( s , o ) for all possible tail entities . Though TF models achieve ∗ . Joint second author . excellent task performance , they are inherently opaque – it is very difficult to uncover a rationale for why the model scores an entity high for a query , because neither the dimensions nor the scoring function are human interpretable . Since users are often the final consumers of a KB ( e.g. , via a KB-based QA task , or an entity retrieval task [ Xiao et al. , 2016 ] ) , we believe providing a rationale behind a prediction to them is an important sub-task for building trust in these systems . Contributions : We present OXKBC : a post-hoc Outcome eXplanation engine , which is faithful to a given TF -based KBC model ( M ) . It works on the underlying Knowledge Graph , KG , corresponding to the facts given in the Knowledge Base , KB . It first augments the KG with weighted edges between entities based on their similarity as defined by the underlying model M . It then posits that an explanation for a prediction o for query r ( s , ? ) is a path between s and o in this augmented graph . A key contribution of our work is that similar paths are grouped into second-order templates and OXKBC trains a neural template selection module for explaining a given prediction . We define novel unsupervised and semi-supervised loss functions , so that this module can be trained with no or minimal training data . OXKBC outputs the highest scoring path within the selected template as an explanation to the user . We evaluate OXKBC explanations via a user study over Amazon Mechanical Turk . We find that workers overwhelmingly prefer OXKBC explanations compared to rule-mining techniques proposed earlier . 2 . Related Work . KBC : Other than TF based models , research has also explored path based approaches for KBC . They use random walk with restarts [ Lao and Cohen , 2010 ] and more recently , reinforcement learning [ Das et al. , 2018 , Xiong et al. , 2017 ] to construct multi-hop paths in the knowledge graph and use those to make predictions . The paths may offer natural interpretability [ Nickel et al. , 2016 ] , but the performance of these models is significantly lower than TF , especially on large KBs like FB15K [ Bordes et al. , 2013 ] , and YAGO [ Suchanek et al. , 2007 ] . See Appendix A.1 for a detailed comparison . [ Rocktäschel and Riedel , 2017 ] inductively learn first order rules to generate paths by using a differentiable version of backward chaining . Unfortunately , it does not scale beyond 10K facts in KB.1 In contamporaneous work , [ Minervini et al. , 2020 ] speed up neural theorem proving for this task . Unfortunately , we could not get their method to complete even after a week of continuous computation on FB15K , suggesting that their method still is not scalable to large KBs . Earlier , research had also considered symbolic Horn-clause style inference [ Lao and Cohen , 2010 , Schoenmackers et al. , 2010 ] , though these are no longer competitive with modern neural methods . TF methods , like DistMult [ Yang et al. , 2015 ] and ComplEx [ Trouillon et al. , 2016 ] , calculate the score SM ( s , r , o ) , of a triple r ( s , o ) via a latent tensor factorization over entity embeddings ( s , t ) and relation embeddings ( r ) . Key idea in these methods is to learn embeddings such that the score SM of the facts present in the KB is high . In this case , DistMult [ Yang et al. , 2015 ] aligns relation embeddings to the Hadamard product of the embeddings of head and tail entitites . It uses SM ( s , r , o ) = rT ( s • o ) , where • is the element-wise product , also known as Hadamard product . While in principle OXKBC may be adapted to explain any TF model ’ s predictions , in our experiments , we use TypedDistMult 2 [ Jain et al. , 2018 ] as our underlying model M . The key idea is to augment the scoring function used by DistMult to reduce type errors . Two new terms are introduced to account for head and tail entity type compatibility . The approach requires no explicit 1. https : //www.akbc.ws/2017/slides/sebastian-riedel-slides.pdf 2 . Code taken from https : //github.com/dair-iitd/KBI specification of type hierarchies or type supervision . The scoring function SM scores r ( s , o ) as SM ( s , r , o ) = σ ( rTw ( sw • ow ) ) σ ( rTtsst ) σ ( rTtoot ) , where σ is the sigmoid function . The subscript w denotes the world knowledge embedding , t denotes the type embedding of an entity and ts and to denote head and tail type embeddings of a relation , respectively . Explainability in KBC : There are broadly two approaches for explainable AI . One is to design a transparent model , which can inherently reveal its functioning ( e.g. , linear models , decision trees , Horn-clause inference for KBC ) . Unfortunately , transparent models often result in compromised performance in general [ Weld and Bansal , 2018 ] , as well as specifically for KBC ( as discussed earlier in this section ) . An alternative is to provide post-hoc explanations for the behaviour of a high-quality black box model [ Wachter et al. , 2017 ] . This approach is further divided into two categories : ‘ Model Explanation ’ ( MX ) and ‘ Outcome Explanation ’ ( OX ) [ Costabello et al. , 2019 ] . In MX , once the black–box model is learnt , a global explainable auxiliary model is built using the predictions of the black-box model such that the auxiliary model has high fidelity with the original black-box model . Rule mining [ Yang et al. , 2015 ] is an MX approach , which mines first order rules based on embeddings learnt by M . It identifies paths ( of length ≤ 3 ) using the notion of support and confidence borrowed from the frequent itemset mining literature [ Pei et al. , 2000 ] . Search space is pruned by considering only those relations which are close to each other in the embedding space . The extracted first order rules may be used to explain a prediction . We compare against this approach in our experiments . The other MX approach is known as the pedagogical approach – it trains a global auxiliary interpretable model ( weighted Horn clause rules ) from the output of the non-interpretable but accurate TF model [ Gusmão et al. , 2018 ] . Its experiments were on a toy dataset with only 13 relations , and unfortunately , we were unable to scale this to the size of our dataset ( see Appendix A.2 ) . OX , attempts to achieve a higher fidelity between explainable model and M by creating local model approximations around each prediction independently , e.g . [ Ribeiro et al. , 2016 ] . This approach often trains a new classifier for each prediction , which is slow if a large number of predictions need to be explained . In response , OXKBC takes a novel OX approach , which decomposes the explanation engine into two components : a global trained model that , for a given prediction , selects the most plausible second-order explanation template ( which quantifies over both relations and entities ) , and an explanation generator that outputs the best instantiation of the template for this prediction . This approach scales better , since the trained model is global and easy to train , and no retraining is required per prediction . Finally , a related task to OX is fact checking – verifying the truthfulness of a ( predicted ) triple r ( s , o ) based on existing facts in the KB . Many fact-checking approaches [ Shiralkar et al. , 2017 , Shi and Weninger , 2016 ] use path based techniques to check the veracity of a triple and generate paths as evidence . However , these methods do not use algorithms that use information from the underlying TF model . Thus , while these paths may be useful evidence , they should not be used as explanations , as they are not faithful to the TF model . In the future , it may be interesting to combine fact checking ideas with statistical information in TF models to develop a better outcome explanation engine . 3 . OXKBC : The Outcome Explanation Engine . We are given a knowledge base of facts , KB , with relations R and entities E , and a model M , trained on KB . M scores each triple r ( s , o ) using the score SM ( s , r , o ) . The goal of OXKBC is to show an end-user a reason why M predicts a tail entity o for the query r ( s , ? ) . It does so by finding an explanation path between s and o which entails the given relation r .

Summary: The authors propose a novel method OxKBC for generating post-hoc explanations of a trained factorization model for link prediction. The authors identify five different templates which explain a predicted triple (s,r,o): Similarity between tail entities, similarity between relations, both occurring simultaneously, similarity between r and the hadamard product of a two-hop relation in the KB, and o being a frequent tail entity for r. Since the predefined scoring functions do not obviously generalize between different templates, OxKBC performs a two-step computation for each predicted triple where first the best template is selected, and second that template is grounded in the knowledge graph. The authors evaluate OxKBC using a paired comparisons test with Amazon Mechanical Turk, comparing against a rule-mining baseline. Results show OxKBC to handily outperform the baseline.