Revisiting Evaluation of Knowledge Base Completion Models

led to accurate models for existing KG completion benchmarks . However , due to the open-world assumption of existing KGs , evaluation of KG completion uses ranking metrics and triple classification with negative samples , and is thus unable to directly assess models on the goals of the task : completion . In this paper , we first study the shortcomings of these evaluation metrics . Specifically , we demonstrate that these metrics ( 1 ) are unreliable for estimating how calibrated the models are , ( 2 ) make strong assumptions that are often violated , and 3 ) do not sufficiently , and consistently , differentiate embedding methods from each other , or from simpler approaches . To address these issues , we gather a semi-complete KG referred as YAGO3-TC , using a random subgraph from the test and validation data of YAGO3-10 , which enables us to compute accurate triple classification accuracy on this data . Conducting thorough experiments on existing models , we provide new insights and directions for the KG completion research . Along with the dataset and the open source implementation of the models , we also provide a leaderboard for knowledge graph completion that consists of a hidden , and growing , test set , available at https : //pouyapez.github.io/yago3-tc/ . 1 . Introduction . Knowledge graphs ( KGs ) are essential components of a wide range of tasks in scientific and industrial processes [ Zhang et al. , 2016 , Zhu et al. , 2018 ] . Most knowledge graphs , in practice , are often substantially incomplete and contain noise even in the edges they do have , prompting the need for models for knowledge graph completion ( KGC ) , also called link prediction . In recent years , models that have led to accurate link prediction are based primarily on relational embeddings [ Bordes et al. , 2013a , Yang et al. , 2015 ] , where dense vectors are learned for each entity and relation in the KG . By using different scoring functions that capture the uncertainty in each fact [ Trouillon et al. , 2017 , Dettmers et al. , 2018 , Sun et al. , 2019a ] , such knowledge graph completion models have achieved incredible success on existing benchmarks . Unfortunately , the lack of a complete and accurate KGs is a problem for evaluation as well . Since it is not possible to list all possible true and false facts for a KG of interest , existing evaluation of KGC consists of gathering known true facts , and using : ( 1 ) ranking metrics , such as Hits @ N and Mean Reciprocal Rank ( MRR ) , to calculate the relative rank of these known true facts against all unknown facts ( thus implicitly treated as negative ) , and ( 2 ) classification accuracy of individual facts , by treating random corruptions of a known true fact as negative/false facts . In spite of steady and significant progress on these models , it is not clear whether these metrics correspond to the true performance on link prediction , making it difficult to decide whether they are ready for real-world deployment . Further , due to the strong assumptions made by these evaluation metrics , the strengths , shortcomings , and reasoning capabilities underlying these link prediction methods is difficult to determine , hindering further progress of the field . In this paper , we study significant issues with the current evaluation metrics for knowledge graph completion models , in particular , highlighting the impact of the assumptions made by these metrics on model performance . We show that the ranking metrics often do not correlate well with the actual performance of the model , make it incredibly challenging to determine whether these models are well-calibrated or not ( an essential property for real-world deployment ) , and do not correlate well with the reasoning power of the models . For triple classification , upon a detailed examination of several commonly used benchmarks , we show that the metric is heavily sensitive to the choice of negative sampling , and that there is a significant mismatch between accuracy and the ranking metrics . To address these shortcomings in existing benchmarks , we introduce YAGO3-TC , a high-quality , manually-annotated dense sub-graph of the YAGO3-10 KG . Along with the true facts that are already present in test and validation splits of the existing benchmark , YAGO3-TC also includes related facts involving the same entities that are annotated to be true or false via crowdsourcing . These related facts are designed to be somewhat challenging to discriminate since they are high-scoring by recent accurate models , resulting in 28,364 labeled facts out of which 2,976 are positive . Since we ensure the quality of the annotations , classification metrics such as accuracy , precision/recall , etc . can be used to appropriately evaluate models of knowledge graph completion . We also provide a comprehensive analysis of recent KG completion models , given the highquality annotations in YAGO3-TC , using triple classification metrics . We are able to provide accurate calibration results for completion models , showing that they are significantly overconfident ( consistent with existing results for neural networks , but different from other observations for KGC ) . Further , we observe that there is a significant mismatch between ranking metrics and performance on the completion task ( e.g. , there is more than 20 % gap between Hits @ 1 and Precision ) . Most importantly , we show that the progress in performance indicated by ranking metrics does not align with actual completion task ; simple methods achieve similar performance to state-of-art models . 2 . Background and Notation . In this section , we introduce notations , benchmarks , evaluation procedures , and a brief overview of existing relational embedding approaches to knowledge graph completion . More details on existing benchmarks and implementation details are provided in the Appendix . Embedding Based KGC : To represent the facts in the KGs , we use triples of subject , relation , and object , 〈s , r , o〉 , where s , o ∈ ξ , the set of entities , and r ∈ R , the set of relations . To model the KG for link prediction , a scoring function ψ : ξ × R × ξ → R is learned to evaluate whether any given fact is true . In this work , we will primarily study DistMult [ Yang et al. , 2015 ] due to its simplicity and popularity , and RotatE [ Sun et al. , 2019a ] and Tucker [ Balazevic et al. , 2019 ] because of their state-of-the-art performance . Only true facts are included in the training data , thus training the model involves , for each observed triple , sampling L negative ones ( value of L is usually treated as a hyperparameter ) by randomly corrupting either the subject or the object of the triple . Using the model ’ s probability of truth as σ ( ψ ( s , r , o ) ) for 〈s , r , o〉 , following Yang et al . [ 2015 ] , Trouillon et al . [ 2016 ] , the binary cross-entropy loss is defined as : L ( G ) = ∑ ( s , r ) ∈G ∑ o ys , ro log ( σ ( ψ ( s , r , o ) ) ) + ( 1− ys , ro ) log ( 1− σ ( ψ ( s , r , o ) ) ) . ( 1 ) where ys , ro represents whther the fact is “ true ” ( y s , r o = 1 for observed facts , y s , r o = 0 otherwise ) . Ranking Metrics : To evaluate the performance of the KG completion models , we rank test triples against all possible negative samples , generated by corrupting the subject or object of the target triple . Ranking metrics have been used since existing KGs are open-world and the ground truth label for all negative and positive samples is not available . In the filtered setting , which we consider in this work , we only treat triples that do not appear in the training , validation , or test set , as the possible negative samples . To quantify the ranking of target triples , we use standard evaluation metrics such as Mean Reciprocal Rank ( MRR ) , which is the average inverse rank of all test triples , and Hits @ N , which is the percentage of test triples whose rank is lower ( better ) than or equal to N. Triple classification : Triple classification is the task of binary classification on the KGs triples . This task is important because , if appropriately set up , it directly evaluates the capability of KGC models in identifying missing links . Specifically , given a target triple 〈s , r , o〉 , we want to identify if this is a positive/true fact or a negative one . For this task , previous approaches learn a specific threshold τr for each relation , over validation data . In order to create the negative samples in these approaches , for both validation and test data , they corrupt subject or object of the target triple with a random entity form the KG . After learning thresholds τr , a triple assumed to be positive if its score is higher than the threshold for the triple ’ s relation . 3 . Issues in Existing KG Completion Evaluation . In this section , we discuss some issues prevalent in current evaluation metrics , and provide empirical evidence for their shortcomings . First , we observe that the assumptions underlying ranking metrics are often incompatible with the goals of the completion itself . Then , we show that evaluating how well completion models are calibrated is challenging since the results are incredibly sensitive to the setup design choices . Finally , we show that the results of the ranking metrics are often inconsistent with the results of the triple classification evaluation . 3.1 Assumptions in Ranking Metrics . In the past few years , we have observed tremendous progress in the performance of KG completion models , based on ranking metrics . As these models become increasingly accurate and potentially ready for real-world deployment , it is now useful to understand the extent to which these ranking metrics align with the actual goals of the completion task . Let us consider a simple example . Assume we want to validate whether triple 〈s , r , o〉 is true or not . According to the current procedure , our only option is to rank the score of all possible objects ( triples of the form 〈s , r , o′〉 ) and subjects ( of the form 〈s′ , r , o〉 ) and compute the rank of our target triple . In this case , the ranking metrics such as Hits @ N can only tell us whether this triple appears in the top N possible triples . If the relation of our target triple can accept only one true object , i.e . the relation is N-1 , this ranking is meaningful , however , if multiple objects can be true for our target subject and relation , this ranking is incomplete since does not capture other triples that are ranked higher than the target fact are themselves true or not . A similar observation holds for relations for which multiple subjects can be true for the same object , i.e . a 1-N relation . Unfortunately , on studying two commonly used KGs WN18RR and YAGO3-10 , we notice that this phenomenon happens in a huge portion of the data , as a result of the existence of semi-inverse relations . Semi-Inverse Relations in WN18RR : On conducting simple statistical analysis on WN18RR , we notice that this KG is not completely free of relations that are inverse of each other ( which make the completion task trivial ) . We notice in more than 90 % appearance of three relations _derivationally_related_form , _verb_group , and _similar_to , and also more than 60 % appearance of _also_see , another triple with the same entities but the opposite direction of the original relation appears in the training data . Together , these relations consist 37 % of this KG . Moreover , around one-third of triples in the test data contain one of these relations ; the KGC models achieve more than 0.95 MRR performance on this subset , significantly affecting the overall performance . Semi-Inverse Relations in YAGO3-10 : Along the similar lines , in YAGO3-10 , for 75 % of the triples with relation isAffiliatedTo , the triple with relation playsFor appears between the same object and subject ( 87 % for reverse ) . These relations consist of 63 % of test data , 81 % of which have the other relation between the same subject and object in the training data . Note that embedding methods achieve around 0.95 of MRR performance on these triples . The existence of these semi-inverse relations results in two important conclusions . First , it indicates that the performance based on ranking metrics on these KGs is not trustworthy since it is very easy to predict these triples . Second , since these relations can accept many objects ( and subjects ) for the same subject ( and object ) , and embedding methods score all of those objects ( and subject ) very highly ( since their semi-inverse version of them appear in the training data ) , low values of ranking metrics are clearly not accurate assessment of completion on these triples . More specifically , for a target triple with one of the mentioned relations , as the number of true objects or subjects with semi-inverse relation increases , the chance of obtaining a worse ranking increases as well .

Summary: This paper first analyzes several popular KB-completion datasets and their evaluation methods.  Several issues have been highlighted and discussed, such as the assumptions made in the ranking metrics, skewed distributions on semi-inverse relations (in WN18RR & YAGO3-10), confidence scores by popular methods are not calibrated.  In addition, the authors also suggest that some simple baselines are actually quite robust.  Based on their finding, the author creates a binary triple classification dataset.  Effectively, every triples in their dataset are examined by multiple Turkers to ensure the label quality and also to avoid the potential error due to the "close-world" assumption behind some existing datasets.