Behavioral Testing of Knowledge Graph Embedding Models for Link Prediction

1 . Introduction . Knowledge Graphs ( KGs ) are graph databases that represent information about entities and their relationships in the form of canonical ( head , relation , tail ) -triples . KGs are used in many downstream tasks such as question answering [ Huang et al. , 2019 ] , recommender systems [ Guo et al. , 2020 ] , information extraction [ Gashteovski et al. , 2020 ] , and named entity linking [ Shen et al. , 2012 ] . A common challenge when working with KGs is that they suffer from incompleteness [ West et al. , 2014 ] . For example , the popular KG DBpedia [ Lehmann et al. , 2015 ] contains information about the entities Joe Biden and United States , but it does not contain the fact expressing the presidentOf relationship between the two entities . Motivated by the incompleteness of most KGs , there is a large body of work on link prediction in knowledge graphs , that is , on deriving missing triples from the set of existing triples . KG Embeddings ( KGE ) have been shown to be effective at predicting missing links [ Minervini et al. , 2015 , Ruffinelli et al. , 2020 ] . KGE methods learn low-dimensional representations of entities and relation types in a vector space ( see also the more detailed definition in Section 2.1 ) . KGE models are typically evaluated by measuring their ability to rank candidate entities averaged over triples in a held out test set . This average-based evaluation , however , poses several problems ( Bianchi et al . [ 2020 ] , Kadlec et al . [ 2017 ] , Sun et al . [ 2020b ] , Mohamed et al . [ 2020 ] ; for a closer discussion see also Section 2.2 ) and leaves open the question of what models fail to learn . As a step towards a better evaluation of KGE models , we propose the use of behavioral tests . Behavioral testing , a standard-practice in software engineering , is concerned with testing behaviors of a ( black-box ) software system by feeding it various inputs and observing and analyzing the system ’ s behavior . Such behavioral tests have several advantages . First , it is possible to find out if a model makes systematic mistakes for a certain capability of interest . Second , one can compare different models in a more detailed and fine-grained manner , allowing us to understand under which circumstances different models offer an advantage . Third , the tests could be used to uncover particular issues in the training data , which could then be corrected ( e.g . in the context of KGs by adding more entities or relations of a certain type ) . Fourth , by using a more rigorous testing approach , we can increase the trust of stakeholders in production settings . In this work , we kick-start a new evaluation framework by exploring possible testable capabilities and by defining detailed tests for two of these capabilities . First , we explore how well KGE models handle symmetric relations ( e.g . the relation spouseOf is symmetric ) , which has been a much discussed property ( Sun et al . [ 2019 ] , Peng and Zhang [ 2020 ] , Zhang et al . [ 2020 ] , Trouillon et al . [ 2016 ] , Wang et al . [ 2014 ] ; inter alia ) but so far lacks a systematic evaluation . Second , entities in a KG are often associated with an entity type ( e.g . Diane Sawyer is of type Actor ) and we evaluate how well KGE models have learnt to respect entity types . We run these tests on six KGE models and find that the model ranking on the original test set does not reflect the same ranking when testing for specific capabilities . For instance , we find that the model ComplEx [ Trouillon et al. , 2016 ] ranks second to last on the original test set but is the best at predicting unseen triples for symmetric relations . 2 . Link Prediction for Knowledge Graphs . A knowledge graph K consists of a set of entities E , a set of relation types R , and a set of triples d = ( h , r , t ) , with head h ∈ E , relation type r ∈ R and tail t ∈ E . For example , the triple ( Porto , locatedIn , Portugal ) represents the information that the entity Porto is located in the entity Portugal . Knowledge graphs are typically incomplete . Link prediction is the task of inferring new triples based on the triples contained in the knowledge graph . This problem can be framed as a tail and head prediction query of the form ( h , r , ? ) and ( ? , r , t ) , where one seeks to find substitutions for ? that result in a new correct triple . 2.1 Knowledge Graph Embedding Models . A Knowledge Graph Embedding ( KGE ) model consists of three main components . First , the KGE model ’ s parameters w. Typically , the parameters are vectors associated with each entity and relation type . Second , given the model ’ s parameters w we have a scoring function φ ( d ; w ) which maps the parameters of the head , tail , and relation type occurring in a triple d to a real-valued number . KGE models differ mainly in the particular choice of scoring functions . Third , the model ’ s parameters are learnt based on a training set Dtrain of known triples , where a commonly used loss function aims to maximize the score of known triples while minimizing the score of randomly sampled ( much more likely to be incorrect ) triples . Once a model is trained , it can predict a score for an unseen triple that indicates the likelihood of this being true . 2.2 Standard Evaluation Metrics and their Shortcomings . The standard approach for evaluating link prediction methods uses a ( held-out ) test set Dtest of correct triples . For each of these ( h , r , t ) -triples , the model scores all possible substitutions of the queries ( h , r , ? ) and ( ? , r , t ) . Then , the entities are ranked in descending order according to their predicted scores . There are two commonly used metrics for evaluating the resulting rankings—MRR and Hits @ k ( t may be replaced with h for the reverse direction ) : MRR : = 1 |Dtest| ∑ ( h , r , t ) ∈Dtest 1 rank ( t ) ; Hits @ k : = 1 |Dtest| ∑ ( h , r , t ) ∈Dtest 1 [ rank ( t ) ≤ k ] . The MRR ( mean reciprocal rank ; lower is better ) metric identifies the gold tail t ’ s rank rank ( t ) according to the model ’ s scores and computes the mean of the reciprocal ranks over all test triples . The Hits @ k metric computes the mean , over all test triples , of the event that the gold tail occurs in the top k ranked entities . Using the above metrics as averages over a test set is a global and coarse-grained measure of accuracy . It conflates different relation and entity types and does not allow the user to understand the types of errors the KGE model makes on particular groups of triples . For example , the model could perform poorly on specific relation types , either due to a weakness of the model or misspecified training data . The need for an evaluation approach that can analyze and explore the behavior of KGE methods in a more fine-grained manner was also expressed in prior work [ Bianchi et al. , 2020 ] . To move towards a better evaluation paradigm for KGE models , we propose a new , extendable evaluation framework which can compare KGE models in a more systematic manner . In addition to the coarse-grained nature of standard evaluation measures , prior work has identified additional shortcomings of common evaluation procedures . First , Kadlec et al . [ 2017 ] found that hyperparameter tuning can be more important than model architecture changes , which caused the authors to raise doubts about the traditional accuracy based evaluation . By using and extending our framework of more fine-grained behavioral testing , it will be possible to determine if and under which circumstances different model architecture perform better . Second , it has been found that recently published ( and seemingly superior ) KGE models have been evaluated incorrectly [ Sun et al. , 2020b ] . By introducing a common test framework , such inconsistency could be avoided . Third , Mohamed et al . [ 2020 ] has shown that the traditional accuracy-based metrics overestimate the performance of KGE models because they magnify the accuracy on frequently occurring entities and/or relations . The less frequent , “ long-tail ” triples , however , are often the triples that one cares the most about [ Ilievski et al. , 2020 ] . 3 . Behavioral Testing . Overview . We propose to use behavioral tests as a new evaluation framework that allows one to better understand what KGE models succeed or fail to learn . Behavioral testing is a software engineering principle where capabilities of software systems are tested by treating them as black boxes and analyzing their behavior for specifically designed inputs . For example , a software engineer might write a function that expects positive integers inputs . One behavioral test for this function could be a test that checks what happens if the function is given a negative integer instead - does the function handle this disallowed case appropriately ? The idea of behavioral testing has also recently been applied to NLP [ Ribeiro et al. , 2020 ] . For example , Ribeiro et al . [ 2020 ] analyzed the robustness of named entity recognition systems by replacing the city name in specific sentences with different city names , and observing the resulting change in behavior of the system . Here , we transfer the idea of behavioral testing to analyze the behavior of KGE models in a more fine-grained manner . Knowledge graphs are typically associated with a predefined schema [ Auer et al. , 2007 ] , which imposes constraints on the set of possible triples . For instance , a given relation type might have entity type constraints for its domain and range ( e.g. , the DBpedia relation type birthPlace must have an entity of type person as head and an entity of type location as tail ) . We propose to use such schemas to design behavioral tests and to check the consistency of KGE models with regards to the schema constraints . Based on a given KG schema , fine-grained tests can be designed to assess KGE models with respect to their behavior for particular capabilities . As a result , this type of evaluation can offer more detailed insights into where KGE models perform well and where they might make systematic mistakes . A graphical overview of our proposed approach is depicted in Figure 1 . Possible Capabilities . Our framework can be easily extended by adding new capabilities and corresponding tests . Adding various such capabilities and tests to one joint evaluation framework would ensure better comparison between different KGE models in the future . Some possible capabilities are : • Relation symmetry : KGs often contain symmetric relations , where a relation r ∈ R is symmetric if ∀x , y ∈ E , ( x , r , y ) =⇒ ( y , r , x ) . For example , if a KGE model was trained on the symmetric relation ( x , spouseOf , y ) , a test could check if it correctly predicts ( y , spouseOf , x ) . Ensuring that a KGE model can handle symmetric relations has been the focus of several recently proposed KGE models ( Sun et al . [ 2019 ] , Peng and Zhang [ 2020 ] , Zhang et al . [ 2020 ] , Trouillon et al . [ 2016 ] , Wang et al . [ 2014 ] ; inter alia ) . The idea can also be extended to other properties of relation types , such as antisymmetry , inversion , and composition . • Entity hierarchy : Entities can be associated with an entity type . For example , the FB15k-237 entity Diane Sawyer is of type Actor . The entity types themselves are organized in a hierarchical ( hypernym ) taxonomy ; e.g . Actor → Artist → Person . Various KGE models have specifically been designed to learn entity hierarchies well [ Zhang et al. , 2020 , Kolyvakis et al. , 2020 , Balazevic et al. , 2019 , Chami et al. , 2020 , Sun et al. , 2020a ] . Tests could for example explore how well KGE models work at different hierarchy levels . • Entity distributions : Relations can be grouped into 4 categories with regards to how many correct heads/tails they may have : 1-to-1 , 1-to-many , many-to-1 and many-to-many . For example , some relations by definition can only have one correct tail as an answer ( e.g . birthPlace ) . Prior work studied how well KGE models perform in such different scenarios ( e.g . [ Bordes et al. , 2013a , Peng and Zhang , 2020 ] ) and this type of analysis could easily be added to the evaluation framework . • Robustness to adversarial attacks : Pezeshkpour et al . [ 2019 ] study how KGE models are effected by adversarial modifications . This idea could be transformed into a capability that checks how prone KGE models are to adversarial attacks . • Relation/entity frequency : The observation that KGE models perform better on frequently occurring entities/relations [ Mohamed et al. , 2020 ] could be systematically tested by creating different test sets where this frequency is varied . The tests defined for a capability explore the behavior of a model under a particular setting or condition . They can be utilised in two ways . First , they can serve as evaluation benchmarks with which different systems can be compared to each other . Based on this , we can choose the best model for a particular capability of interest . Second , they can be used to determine the failure rate of a system by making a binary decision for each triple , e.g . by defining a cut-off point for each tested triple.1 With this view , we can determine if a model is good enough to be deployed . While the latter is ultimately more important in a production setting , we focus here on the former , with which we explore how known KGE models can be compared against each other . 1 . For example , for a system to succeed a triple ’ s gold tail has to be in the top 3 of a model ’ s prediction , else it is counted as failure . Evaluation Setup . To kick-start the new evaluation framework , we define several tests for two capabilities and then use the tests to evaluate different KGE models . For the evaluation , we employ six KGE models : DistMult [ Yang et al. , 2015 ] , ComplEx [ Trouillon et al. , 2016 ] , RotatE [ Sun et al. , 2019 ] , HyperKG [ Kolyvakis et al. , 2020 ] , LinearRE [ Peng and Zhang , 2020 ] , HAKE [ Zhang et al. , 2020 ] . For an overview of their scoring functions , see Table 1 in the appendix . The first capability we explore is relation symmetry . All proposed methods can , in principle , learn that a relation type is symmetric . DistMult in particular treats all relation types as such . However , the main evaluation for these models was conducted on the standard test sets with averaged metrics . While ablation studies regarding symmetric capability are sometimes included [ Sun et al. , 2019 , Peng and Zhang , 2020 ] ,2 they are neither systematic nor comparable across papers . Therefore , it is difficult to judge to what degree they can handle symmetry and which model might be the best with regards to this capability . To be able to answer this question , we design targeted tests which allow us to measure the performance specifically for symmetric capability . In turn , we can analyse in detail under which settings which model performs best . The second capability we investigate is entity hierarchy . Many KGs are hierarchical by nature ; as an example see Figure 3 for entity types in DBpedia . This encodes important additional knowledge ; for instance , certain relations only apply to certain entity types . For example , the relation starsIn is typically associated with tail entities of type Actor , which is at level 4 in the hierarchy ( see Figure 3 ) . In contrast , the relation birthPlace can have tail entities of the higher ranked level 1 type Place ( e.g . City or Town , because both entity types are a type of Place ) . For this capability , we design tests that explore how well KGE models have learnt to handle different entity types at varying levels in the entity type hierarchy . For this capability , we can test whether KGE models that have specifically been designed to learn hierarchical structure ( HyperKG , HAKE ) perform better than other models . The models are trained on FB15k-273 [ Toutanova et al. , 2015 ] and we evaluate the models which achieved highest performance on the original FB15k-273 test set . All hyperparameters are included in Section C of the appendix .

Summary: This work proposes to apply behavioral testing in the evaluation of KGE models for link prediction. The authors first pointed out the drawbacks in the current evaluation method which typically averaging the accuracy metrics on all the triples in the test set. Then they suggested to apply the behavioral test evaluates the system's performance on different capabilities of the system. As two examples, the authors conducted such evaluation for symmetric relations and type hierarchies and further exposed discrepancy in the original evaluation.