Probability Calibration for Knowledge Graph Embedding Models

1 INTRODUCTION . Knowledge graph embedding models are neural architectures that learn vector representations ( i.e . embeddings ) of nodes and edges of a knowledge graph . Such knowledge graph embeddings have applications in knowledge graph completion , knowledge discovery , entity resolution , and link-based clustering , just to cite a few ( Nickel et al. , 2016a ) . Despite burgeoning research , the problem of calibrating such models has been overlooked , and existing knowledge graph embedding models do not offer any guarantee on the probability estimates they assign to predicted facts . Probability calibration is important whenever you need the predictions to make probabilistic sense , i.e. , if the model predicts a fact is true with 80 % confidence , it should to be correct 80 % of the times . Prior art suggests to use a sigmoid layer to turn logits returned by models into probabilities ( Nickel et al. , 2016a ) ( also called the expit transform ) , but we show that this provides poor calibration . Figure 1 shows reliability diagrams for off-the-shelf TransE and ComplEx . The identity function represents perfect calibration . Both models are miscalibrated : all TransE combinations in Figure 1a under-forecast the probabilities ( i.e . probabilities are too small ) , whereas ComplEx under-forecasts or over-forecasts according to which loss is used ( Figure1b ) . Calibration is crucial in high-stakes scenarios such as drug-target discovery from biological networks , where end-users need trustworthy and interpretable decisions . Moreover , since probabilities are not calibrated , when classifying triples ( i.e . facts ) as true or false , users must define relationspecific thresholds , which can be awkward for graphs with a great number of relation types . To the best of our knowledge , this is the first work to focus on calibration for knowledge embeddings . Our contribution is two-fold : First , we use Platt Scaling and isotonic regression to calibrate knowledge graph embedding models on datasets that include ground truth negatives . One peculiar feature of knowledge graphs is that they usually rely on the open world assumption ( facts not present are not necessarily false , they are simply unknown ) . This makes calibration troublesome because of the lack of ground truth negatives . For this reason , our second and main contribution is a calibration heuristics that combines Platt-scaling or isotonic regression with synthetically generated negatives . Experimental results show that we obtain better-calibrated models and that it is possible to calibrate knowledge graph embedding models even when ground truth negatives are not present . We also experiment with triple classification , and we show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds . 2 RELATED WORK . A comprehensive survey of knowledge graph embedding models is out of the scope of this paper . Recent surveys such as ( Nickel et al. , 2016a ) and ( Cai et al. , 2017 ) summarize recent literature . TransE ( Bordes et al. , 2013 ) is the forerunner of distance-based methods , and spun a number of models commonly referred to as TransX . The intuition behind the symmetric bilinear-diagonal model DistMult ( Yang et al. , 2015 ) paved the way for its asymmetric evolutions in the complex space , RotatE ( Sun et al. , 2019 ) and ComplEx ( Trouillon et al. , 2016 ) ( a generalization of which uses hypercomplex representations ( Zhang et al. , 2019 ) ) . HolE relies instead on circular correlation ( Nickel et al. , 2016b ) . The recent TorusE ( Ebisu & Ichise , 2018 ) operates on a lie group and not in the Euclidean space . While the above models can be interpreted as multilayer perceptrons , others such as ConvE ( Dettmers et al. , 2018 ) or ConvKB ( Nguyen et al. , 2018 ) include convolutional layers . More recent works adopt capsule networks architectures ( Nguyen et al. , 2019 ) . Adversarial learning is used by KBGAN ( Cai & Wang , 2018 ) , whereas attention mechanisms are instead used by ( Nathani et al. , 2019 ) . Some models such as RESCAL ( Nickel et al. , 2011 ) , TuckER ( Balažević et al. , 2019 ) , and SimplE ( Kazemi & Poole , 2018 ) rely on tensor decomposition techniques . More recently , ANALOGY adopts a differentiable version of analogical reasoning ( Liu et al. , 2017 ) . In this paper we limit our analysis to four popular models : TransE , DistMult , ComplEx and HolE . They do not address the problem of assessing the reliability of predictions , leave aside calibrating probabilities . Besides well-established techniques such as Platt scaling ( Platt et al. , 1999 ) and isotonic regression ( Zadrozny & Elkan , 2002 ) , recent interest in neural architectures calibration show that modern neural architectures are poorly calibrated and that calibration can be improved with novel methods . For example , ( Guo et al. , 2017 ) successfully proposes to use temperature scaling for calibrating modern neural networks in classification problems . On the same line , ( Kuleshov et al. , 2018 ) proposes a procedure based on Platt scaling to calibrate deep neural networks in regression problems . The Knowledge Vault pipeline in ( Dong et al. , 2014 ) extracts triples from unstructured knowledge and is equipped with Platt scaling calibration , but this is not applied to knowledge graph embedding models . KG2E ( He et al. , 2015 ) proposes to use normally-distributed embeddings to account for the uncertainty , but their model does not provide the probability of a triple being true , so KG2E would also benefit from the output calibration we propose here . To the best of our knowledge , the only work that adopts probability calibration to knowledge graph embedding models is Krompaß & Tresp ( 2015 ) . The authors propose to use ensembles in order to improve the results of knowledge graph embedding tasks . For that , they propose to calibrate the models with Platt scaling , so they operate on the same scale . No further details on the calibration procedure are provided . Besides , there is no explanation on how to handle the lack of negatives . 3 PRELIMINARIES . Knowledge Graph . Formally , a knowledge graph G = { ( s , p , o ) } ⊆ E × R × E is a set of triples t = ( s , p , o ) , each including a subject s ∈ E , a predicate p ∈ R , and an object o ∈ E . E and R are the sets of all entities and relation types of G. Triple Classification . Binary classification task where G ( which includes only positive triples ) is used as training set , and T = { ( s , p , o ) } ⊆ E × R × E is a disjoint test set of labeled triples to classify . Note T includes positives and negatives . Since the learned models are not calibrated , multiple decision thresholds τi must be picked , where 0 < i < |R| , i.e . one for each relation type . This is done using a validation set ( Bordes et al. , 2013 ) . Classification metrics apply ( e.g . accuracy ) . Link Prediction . Given a training set G that includes only positive triples , the goal is assigning a score f ( t ) ∈ R proportional to the likelihood that each unlabeled triple t included in a held-out set S is true . Note S does not have ground truth positives or negatives . This task is cast as a learning to rank problem , and uses metrics such as mean rank ( MR ) , mean reciprocal rank ( MRR ) or Hits @ N. Knowledge Graph Embeddings . Knowledge graph embedding models are neural architectures that encode concepts from a knowledge graph G ( i.e . entities E and relation types R ) into lowdimensional , continuous vectors ∈ Rk ( i.e , the embeddings ) . Embeddings are learned by training a neural architecture over G. Although such architectures vary , the training phase always consists in minimizing a loss function L that includes a scoring function fm ( t ) , i.e . a model-specific function that assigns a score to a triple t = ( s , p , o ) ( more precisely , the input of fm are the embeddings of the subject es , the predicate rp , and the object eo ) . The goal of the optimization procedure is learning optimal embeddings , such that the scoring function fm assigns high scores to positive triples t+ and low scores to triples unlikely to be true t− . Existing models propose scoring functions that combine the embeddings es , rp , eo ∈ Rk using different intuitions . Table 1b lists the scoring functions of the most common models . For example , the scoring function of TransE computes a similarity between the embedding of the subject es translated by the embedding of the predicate ep and the embedding of the object eo , using the L1 or L2 norm || · || . Such scoring function is then used on positive and negative triples t+ ∈ G , t− ∈ N in the loss function . This is usually a pairwise margin-based loss ( Bordes et al. , 2013 ) , negative log-likelihood , or multi-class log-likelihood ( Lacroix et al. , 2018 ) . Since the training set usually includes positive statements , we generate synthetic negatives t− ∈ N required for training . We do so by corrupting one side of the triple at a time ( i.e . either the subject or the object ) , following the protocol proposed by ( Bordes et al. , 2013 ) . Calibration . Given a knowledge graph embedding model identified by its scoring function fm , with fm ( t ) = p̂ , where p̂ is the estimated confidence level that a triple t = ( s , p , o ) is true , we define fm to be calibrated if p̂ represents a true probability . For example , if fm ( · ) predicts 100 triples all with confidence p̂ = 0.7 , we expect exactly 70 to be actually true . Calibrating a model requires reliable metrics to detect miscalibration , and effective techniques to fix such distortion . Appendix A.1 includes definitions and background on the calibration metrics adopted in the paper . 4 CALIBRATING KNOWLEDGE GRAPH EMBEDDING MODELS PREDICTIONS . We propose two scenario-dependent calibration techniques : we first address the case with ground truth negatives t− ∈ N . The second deals with the absence of ground truth negatives . Calibration with Ground Truth Negatives . We propose to use off-the-shelf Platt scaling and isotonic regression , techniques proved to be effective in literature . It is worth reiterating that to calibrate a model negative triples N are required from a held-out dataset ( which could be the validation set ) . Such negatives are usually available in triple classification datasets ( FB13 , WN11 , YAGO39K ) Calibration with Synthetic Negatives . Our main contribution is for the case where no ground truth negatives are provided at all , which is in fact the usual scenario for link prediction tasks . We propose to adopt Platt scaling or isotonic regression and to synthetically generate corrupted triples as negatives , while using sample weights to guarantee that the frequencies adhere to the base rate of the population ( which is problem-dependent and must be user-specified ) . It is worth noting that it is not possible to calibrate a model without implicit or explicit base rate . If it is not implicit on the dataset ( the ratio of positives to totals ) , it must be explicitly provided . We generate synthetic negatives N following the standard protocol proposed by ( Bordes et al. , 2013 ) 1 : for every positive triple t = ( s , p , o ) , we corrupt one side of the triple at a time ( i.e . either the subject s or the object o ) by replacing it with other entities in E . The number of corruptions generated per positive is defined by the user-defined corruption rate η ∈ N. Since the number of negatives N = |N | can be much greater than the number of positive triples P = |G| , when dealing with calibration with synthetically generated corruptions , we weigh the positive and negative triples to make the calibrated model match the population base rate α = P/ ( P + N ) ∈ [ 0 , 1 ] , otherwise the base rate would depend on the arbitrary choice of η . Given a positive base rate α , we propose the following weighting scheme : ω+ = η for positive triples G ω− = 1 α − 1 for negative triples N ( 1 ) where ω+ ∈ R is the weight associated to the positive triples and ω− ∈ R to the negatives . The ω+ weight removes the imbalance determined by having a higher number of corruptions than positive triples in each batch . The ω− weight guarantees that the given positive base rate α is respected . The above can be verified as follows . For the unweighted problem , the positive base rate is simply the ratio of positive examples to the total number of examples : α = P P +N ( 2 ) If we add uniform weights to each class , we have : α = ω+P ω−N + ω+P ( 3 ) By defining ω+ = η , i.e . adopting the ratio of negatives to positives ( corruption rate ) , we then have : α = P NP Nω− + P N P = N ω−N +N = 1 ω− + 1 ( 4 ) Thus , the negative weights is : ω− = 1 α − 1 ( 5 )

Summary: This is the first work that studies probability calibration for knowledge graph embedding models. In the case where ground-truth negatives are available the authors directly use off-the-shelf established calibration techniques (Platt scaling, isotonic regression). When ground-truth negatives are not available they propose to synthetically generate corrupted triples as negatives and use sample weights to guarantee that the frequencies adhere to the base rate.