Probability Calibration for Knowledge Graph Embedding Models

1 INTRODUCTION . Knowledge graph embedding models are neural architectures that learn vector representations ( i.e . embeddings ) of nodes and edges of a knowledge graph . Such knowledge graph embeddings have applications in knowledge graph completion , knowledge discovery , entity resolution , and link-based clustering , just to cite a few ( Nickel et al. , 2016a ) . Despite burgeoning research , the problem of calibrating such models has been overlooked , and existing knowledge graph embedding models do not offer any guarantee on the probability estimates they assign to predicted facts . Probability calibration is important whenever you need the predictions to make probabilistic sense , i.e. , if the model predicts a fact is true with 80 % confidence , it should to be correct 80 % of the times . Prior art suggests to use a sigmoid layer to turn logits returned by models into probabilities ( Nickel et al. , 2016a ) ( also called the expit transform ) , but we show that this provides poor calibration . Figure 1 shows reliability diagrams for off-the-shelf TransE and ComplEx . The identity function represents perfect calibration . Both models are miscalibrated : all TransE combinations in Figure 1a under-forecast the probabilities ( i.e . probabilities are too small ) , whereas ComplEx under-forecasts or over-forecasts according to which loss is used ( Figure1b ) . Calibration is crucial in high-stakes scenarios such as drug-target discovery from biological networks , where end-users need trustworthy and interpretable decisions . Moreover , since probabilities are not calibrated , when classifying triples ( i.e . facts ) as true or false , users must define relationspecific thresholds , which can be awkward for graphs with a great number of relation types . To the best of our knowledge , this is the first work to focus on calibration for knowledge embeddings . Our contribution is two-fold : First , we use Platt Scaling and isotonic regression to calibrate knowledge graph embedding models on datasets that include ground truth negatives . One peculiar feature of knowledge graphs is that they usually rely on the open world assumption ( facts not present are not necessarily false , they are simply unknown ) . This makes calibration troublesome because of the lack of ground truth negatives . For this reason , our second and main contribution is a calibration heuristics that combines Platt-scaling or isotonic regression with synthetically generated negatives . Experimental results show that we obtain better-calibrated models and that it is possible to calibrate knowledge graph embedding models even when ground truth negatives are not present . We also experiment with triple classification , and we show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds . 2 RELATED WORK . A comprehensive survey of knowledge graph embedding models is out of the scope of this paper . Recent surveys such as ( Nickel et al. , 2016a ) and ( Cai et al. , 2017 ) summarize recent literature . TransE ( Bordes et al. , 2013 ) is the forerunner of distance-based methods , and spun a number of models commonly referred to as TransX . The intuition behind the symmetric bilinear-diagonal model DistMult ( Yang et al. , 2015 ) paved the way for its asymmetric evolutions in the complex space , RotatE ( Sun et al. , 2019 ) and ComplEx ( Trouillon et al. , 2016 ) ( a generalization of which uses hypercomplex representations ( Zhang et al. , 2019 ) ) . HolE relies instead on circular correlation ( Nickel et al. , 2016b ) . The recent TorusE ( Ebisu & Ichise , 2018 ) operates on a lie group and not in the Euclidean space . While the above models can be interpreted as multilayer perceptrons , others such as ConvE ( Dettmers et al. , 2018 ) or ConvKB ( Nguyen et al. , 2018 ) include convolutional layers . More recent works adopt capsule networks architectures ( Nguyen et al. , 2019 ) . Adversarial learning is used by KBGAN ( Cai & Wang , 2018 ) , whereas attention mechanisms are instead used by ( Nathani et al. , 2019 ) . Some models such as RESCAL ( Nickel et al. , 2011 ) , TuckER ( Balažević et al. , 2019 ) , and SimplE ( Kazemi & Poole , 2018 ) rely on tensor decomposition techniques . More recently , ANALOGY adopts a differentiable version of analogical reasoning ( Liu et al. , 2017 ) . In this paper we limit our analysis to four popular models : TransE , DistMult , ComplEx and HolE . They do not address the problem of assessing the reliability of predictions , leave aside calibrating probabilities . Besides well-established techniques such as Platt scaling ( Platt et al. , 1999 ) and isotonic regression ( Zadrozny & Elkan , 2002 ) , recent interest in neural architectures calibration show that modern neural architectures are poorly calibrated and that calibration can be improved with novel methods . For example , ( Guo et al. , 2017 ) successfully proposes to use temperature scaling for calibrating modern neural networks in classification problems . On the same line , ( Kuleshov et al. , 2018 ) proposes a procedure based on Platt scaling to calibrate deep neural networks in regression problems . The Knowledge Vault pipeline in ( Dong et al. , 2014 ) extracts triples from unstructured knowledge and is equipped with Platt scaling calibration , but this is not applied to knowledge graph embedding models . KG2E ( He et al. , 2015 ) proposes to use normally-distributed embeddings to account for the uncertainty , but their model does not provide the probability of a triple being true , so KG2E would also benefit from the output calibration we propose here . To the best of our knowledge , the only work that adopts probability calibration to knowledge graph embedding models is Krompaß & Tresp ( 2015 ) . The authors propose to use ensembles in order to improve the results of knowledge graph embedding tasks . For that , they propose to calibrate the models with Platt scaling , so they operate on the same scale . No further details on the calibration procedure are provided . Besides , there is no explanation on how to handle the lack of negatives . 3 PRELIMINARIES . Knowledge Graph . Formally , a knowledge graph G = { ( s , p , o ) } ⊆ E × R × E is a set of triples t = ( s , p , o ) , each including a subject s ∈ E , a predicate p ∈ R , and an object o ∈ E . E and R are the sets of all entities and relation types of G. Triple Classification . Binary classification task where G ( which includes only positive triples ) is used as training set , and T = { ( s , p , o ) } ⊆ E × R × E is a disjoint test set of labeled triples to classify . Note T includes positives and negatives . Since the learned models are not calibrated , multiple decision thresholds τi must be picked , where 0 < i < |R| , i.e . one for each relation type . This is done using a validation set ( Bordes et al. , 2013 ) . Classification metrics apply ( e.g . accuracy ) . Link Prediction . Given a training set G that includes only positive triples , the goal is assigning a score f ( t ) ∈ R proportional to the likelihood that each unlabeled triple t included in a held-out set S is true . Note S does not have ground truth positives or negatives . This task is cast as a learning to rank problem , and uses metrics such as mean rank ( MR ) , mean reciprocal rank ( MRR ) or Hits @ N. Knowledge Graph Embeddings . Knowledge graph embedding models are neural architectures that encode concepts from a knowledge graph G ( i.e . entities E and relation types R ) into lowdimensional , continuous vectors ∈ Rk ( i.e , the embeddings ) . Embeddings are learned by training a neural architecture over G. Although such architectures vary , the training phase always consists in minimizing a loss function L that includes a scoring function fm ( t ) , i.e . a model-specific function that assigns a score to a triple t = ( s , p , o ) ( more precisely , the input of fm are the embeddings of the subject es , the predicate rp , and the object eo ) . The goal of the optimization procedure is learning optimal embeddings , such that the scoring function fm assigns high scores to positive triples t+ and low scores to triples unlikely to be true t− . Existing models propose scoring functions that combine the embeddings es , rp , eo ∈ Rk using different intuitions . Table 1b lists the scoring functions of the most common models . For example , the scoring function of TransE computes a similarity between the embedding of the subject es translated by the embedding of the predicate ep and the embedding of the object eo , using the L1 or L2 norm || · || . Such scoring function is then used on positive and negative triples t+ ∈ G , t− ∈ N in the loss function . This is usually a pairwise margin-based loss ( Bordes et al. , 2013 ) , negative log-likelihood , or multi-class log-likelihood ( Lacroix et al. , 2018 ) . Since the training set usually includes positive statements , we generate synthetic negatives t− ∈ N required for training . We do so by corrupting one side of the triple at a time ( i.e . either the subject or the object ) , following the protocol proposed by ( Bordes et al. , 2013 ) . Calibration . Given a knowledge graph embedding model identified by its scoring function fm , with fm ( t ) = p̂ , where p̂ is the estimated confidence level that a triple t = ( s , p , o ) is true , we define fm to be calibrated if p̂ represents a true probability . For example , if fm ( · ) predicts 100 triples all with confidence p̂ = 0.7 , we expect exactly 70 to be actually true . Calibrating a model requires reliable metrics to detect miscalibration , and effective techniques to fix such distortion . Appendix A.1 includes definitions and background on the calibration metrics adopted in the paper . 4 CALIBRATING KNOWLEDGE GRAPH EMBEDDING MODELS PREDICTIONS . We propose two scenario-dependent calibration techniques : we first address the case with ground truth negatives t− ∈ N . The second deals with the absence of ground truth negatives . Calibration with Ground Truth Negatives . We propose to use off-the-shelf Platt scaling and isotonic regression , techniques proved to be effective in literature . It is worth reiterating that to calibrate a model negative triples N are required from a held-out dataset ( which could be the validation set ) . Such negatives are usually available in triple classification datasets ( FB13 , WN11 , YAGO39K ) Calibration with Synthetic Negatives . Our main contribution is for the case where no ground truth negatives are provided at all , which is in fact the usual scenario for link prediction tasks . We propose to adopt Platt scaling or isotonic regression and to synthetically generate corrupted triples as negatives , while using sample weights to guarantee that the frequencies adhere to the base rate of the population ( which is problem-dependent and must be user-specified ) . It is worth noting that it is not possible to calibrate a model without implicit or explicit base rate . If it is not implicit on the dataset ( the ratio of positives to totals ) , it must be explicitly provided . We generate synthetic negatives N following the standard protocol proposed by ( Bordes et al. , 2013 ) 1 : for every positive triple t = ( s , p , o ) , we corrupt one side of the triple at a time ( i.e . either the subject s or the object o ) by replacing it with other entities in E . The number of corruptions generated per positive is defined by the user-defined corruption rate η ∈ N. Since the number of negatives N = |N | can be much greater than the number of positive triples P = |G| , when dealing with calibration with synthetically generated corruptions , we weigh the positive and negative triples to make the calibrated model match the population base rate α = P/ ( P + N ) ∈ [ 0 , 1 ] , otherwise the base rate would depend on the arbitrary choice of η . Given a positive base rate α , we propose the following weighting scheme : ω+ = η for positive triples G ω− = 1 α − 1 for negative triples N ( 1 ) where ω+ ∈ R is the weight associated to the positive triples and ω− ∈ R to the negatives . The ω+ weight removes the imbalance determined by having a higher number of corruptions than positive triples in each batch . The ω− weight guarantees that the given positive base rate α is respected . The above can be verified as follows . For the unweighted problem , the positive base rate is simply the ratio of positive examples to the total number of examples : α = P P +N ( 2 ) If we add uniform weights to each class , we have : α = ω+P ω−N + ω+P ( 3 ) By defining ω+ = η , i.e . adopting the ratio of negatives to positives ( corruption rate ) , we then have : α = P NP Nω− + P N P = N ω−N +N = 1 ω− + 1 ( 4 ) Thus , the negative weights is : ω− = 1 α − 1 ( 5 )

Summary: The paper studies probability calibration for three different knowledge graph embedding methods, with a focus on TransE evaluated on the task of knowledge graph triple classification. It studies Brier and log loss performance of Platt scaling and isotonic regression probability calibration on WN11 for TransE and claims that better calibration yields better performance as measured by mean reciprocal rank. Calibration plots for other datasets are also included as evidence. Furthermore, evidence is presented that probability calibration can lead to better performance for the task of triple classification. The main contributions of the paper also include the adaption of sampling techniques introduced by Bordes et al. (2013) adapted for estimating negatives for probability calibrations.