Behavioral Testing of Knowledge Graph Embedding Models for Link Prediction

1 . Introduction . Knowledge Graphs ( KGs ) are graph databases that represent information about entities and their relationships in the form of canonical ( head , relation , tail ) -triples . KGs are used in many downstream tasks such as question answering [ Huang et al. , 2019 ] , recommender systems [ Guo et al. , 2020 ] , information extraction [ Gashteovski et al. , 2020 ] , and named entity linking [ Shen et al. , 2012 ] . A common challenge when working with KGs is that they suffer from incompleteness [ West et al. , 2014 ] . For example , the popular KG DBpedia [ Lehmann et al. , 2015 ] contains information about the entities Joe Biden and United States , but it does not contain the fact expressing the presidentOf relationship between the two entities . Motivated by the incompleteness of most KGs , there is a large body of work on link prediction in knowledge graphs , that is , on deriving missing triples from the set of existing triples . KG Embeddings ( KGE ) have been shown to be effective at predicting missing links [ Minervini et al. , 2015 , Ruffinelli et al. , 2020 ] . KGE methods learn low-dimensional representations of entities and relation types in a vector space ( see also the more detailed definition in Section 2.1 ) . KGE models are typically evaluated by measuring their ability to rank candidate entities averaged over triples in a held out test set . This average-based evaluation , however , poses several problems ( Bianchi et al . [ 2020 ] , Kadlec et al . [ 2017 ] , Sun et al . [ 2020b ] , Mohamed et al . [ 2020 ] ; for a closer discussion see also Section 2.2 ) and leaves open the question of what models fail to learn . As a step towards a better evaluation of KGE models , we propose the use of behavioral tests . Behavioral testing , a standard-practice in software engineering , is concerned with testing behaviors of a ( black-box ) software system by feeding it various inputs and observing and analyzing the system ’ s behavior . Such behavioral tests have several advantages . First , it is possible to find out if a model makes systematic mistakes for a certain capability of interest . Second , one can compare different models in a more detailed and fine-grained manner , allowing us to understand under which circumstances different models offer an advantage . Third , the tests could be used to uncover particular issues in the training data , which could then be corrected ( e.g . in the context of KGs by adding more entities or relations of a certain type ) . Fourth , by using a more rigorous testing approach , we can increase the trust of stakeholders in production settings . In this work , we kick-start a new evaluation framework by exploring possible testable capabilities and by defining detailed tests for two of these capabilities . First , we explore how well KGE models handle symmetric relations ( e.g . the relation spouseOf is symmetric ) , which has been a much discussed property ( Sun et al . [ 2019 ] , Peng and Zhang [ 2020 ] , Zhang et al . [ 2020 ] , Trouillon et al . [ 2016 ] , Wang et al . [ 2014 ] ; inter alia ) but so far lacks a systematic evaluation . Second , entities in a KG are often associated with an entity type ( e.g . Diane Sawyer is of type Actor ) and we evaluate how well KGE models have learnt to respect entity types . We run these tests on six KGE models and find that the model ranking on the original test set does not reflect the same ranking when testing for specific capabilities . For instance , we find that the model ComplEx [ Trouillon et al. , 2016 ] ranks second to last on the original test set but is the best at predicting unseen triples for symmetric relations . 2 . Link Prediction for Knowledge Graphs . A knowledge graph K consists of a set of entities E , a set of relation types R , and a set of triples d = ( h , r , t ) , with head h ∈ E , relation type r ∈ R and tail t ∈ E . For example , the triple ( Porto , locatedIn , Portugal ) represents the information that the entity Porto is located in the entity Portugal . Knowledge graphs are typically incomplete . Link prediction is the task of inferring new triples based on the triples contained in the knowledge graph . This problem can be framed as a tail and head prediction query of the form ( h , r , ? ) and ( ? , r , t ) , where one seeks to find substitutions for ? that result in a new correct triple . 2.1 Knowledge Graph Embedding Models . A Knowledge Graph Embedding ( KGE ) model consists of three main components . First , the KGE model ’ s parameters w. Typically , the parameters are vectors associated with each entity and relation type . Second , given the model ’ s parameters w we have a scoring function φ ( d ; w ) which maps the parameters of the head , tail , and relation type occurring in a triple d to a real-valued number . KGE models differ mainly in the particular choice of scoring functions . Third , the model ’ s parameters are learnt based on a training set Dtrain of known triples , where a commonly used loss function aims to maximize the score of known triples while minimizing the score of randomly sampled ( much more likely to be incorrect ) triples . Once a model is trained , it can predict a score for an unseen triple that indicates the likelihood of this being true . 2.2 Standard Evaluation Metrics and their Shortcomings . The standard approach for evaluating link prediction methods uses a ( held-out ) test set Dtest of correct triples . For each of these ( h , r , t ) -triples , the model scores all possible substitutions of the queries ( h , r , ? ) and ( ? , r , t ) . Then , the entities are ranked in descending order according to their predicted scores . There are two commonly used metrics for evaluating the resulting rankings—MRR and Hits @ k ( t may be replaced with h for the reverse direction ) : MRR : = 1 |Dtest| ∑ ( h , r , t ) ∈Dtest 1 rank ( t ) ; Hits @ k : = 1 |Dtest| ∑ ( h , r , t ) ∈Dtest 1 [ rank ( t ) ≤ k ] . The MRR ( mean reciprocal rank ; lower is better ) metric identifies the gold tail t ’ s rank rank ( t ) according to the model ’ s scores and computes the mean of the reciprocal ranks over all test triples . The Hits @ k metric computes the mean , over all test triples , of the event that the gold tail occurs in the top k ranked entities . Using the above metrics as averages over a test set is a global and coarse-grained measure of accuracy . It conflates different relation and entity types and does not allow the user to understand the types of errors the KGE model makes on particular groups of triples . For example , the model could perform poorly on specific relation types , either due to a weakness of the model or misspecified training data . The need for an evaluation approach that can analyze and explore the behavior of KGE methods in a more fine-grained manner was also expressed in prior work [ Bianchi et al. , 2020 ] . To move towards a better evaluation paradigm for KGE models , we propose a new , extendable evaluation framework which can compare KGE models in a more systematic manner . In addition to the coarse-grained nature of standard evaluation measures , prior work has identified additional shortcomings of common evaluation procedures . First , Kadlec et al . [ 2017 ] found that hyperparameter tuning can be more important than model architecture changes , which caused the authors to raise doubts about the traditional accuracy based evaluation . By using and extending our framework of more fine-grained behavioral testing , it will be possible to determine if and under which circumstances different model architecture perform better . Second , it has been found that recently published ( and seemingly superior ) KGE models have been evaluated incorrectly [ Sun et al. , 2020b ] . By introducing a common test framework , such inconsistency could be avoided . Third , Mohamed et al . [ 2020 ] has shown that the traditional accuracy-based metrics overestimate the performance of KGE models because they magnify the accuracy on frequently occurring entities and/or relations . The less frequent , “ long-tail ” triples , however , are often the triples that one cares the most about [ Ilievski et al. , 2020 ] . 3 . Behavioral Testing . Overview . We propose to use behavioral tests as a new evaluation framework that allows one to better understand what KGE models succeed or fail to learn . Behavioral testing is a software engineering principle where capabilities of software systems are tested by treating them as black boxes and analyzing their behavior for specifically designed inputs . For example , a software engineer might write a function that expects positive integers inputs . One behavioral test for this function could be a test that checks what happens if the function is given a negative integer instead - does the function handle this disallowed case appropriately ? The idea of behavioral testing has also recently been applied to NLP [ Ribeiro et al. , 2020 ] . For example , Ribeiro et al . [ 2020 ] analyzed the robustness of named entity recognition systems by replacing the city name in specific sentences with different city names , and observing the resulting change in behavior of the system . Here , we transfer the idea of behavioral testing to analyze the behavior of KGE models in a more fine-grained manner . Knowledge graphs are typically associated with a predefined schema [ Auer et al. , 2007 ] , which imposes constraints on the set of possible triples . For instance , a given relation type might have entity type constraints for its domain and range ( e.g. , the DBpedia relation type birthPlace must have an entity of type person as head and an entity of type location as tail ) . We propose to use such schemas to design behavioral tests and to check the consistency of KGE models with regards to the schema constraints . Based on a given KG schema , fine-grained tests can be designed to assess KGE models with respect to their behavior for particular capabilities . As a result , this type of evaluation can offer more detailed insights into where KGE models perform well and where they might make systematic mistakes . A graphical overview of our proposed approach is depicted in Figure 1 . Possible Capabilities . Our framework can be easily extended by adding new capabilities and corresponding tests . Adding various such capabilities and tests to one joint evaluation framework would ensure better comparison between different KGE models in the future . Some possible capabilities are : • Relation symmetry : KGs often contain symmetric relations , where a relation r ∈ R is symmetric if ∀x , y ∈ E , ( x , r , y ) =⇒ ( y , r , x ) . For example , if a KGE model was trained on the symmetric relation ( x , spouseOf , y ) , a test could check if it correctly predicts ( y , spouseOf , x ) . Ensuring that a KGE model can handle symmetric relations has been the focus of several recently proposed KGE models ( Sun et al . [ 2019 ] , Peng and Zhang [ 2020 ] , Zhang et al . [ 2020 ] , Trouillon et al . [ 2016 ] , Wang et al . [ 2014 ] ; inter alia ) . The idea can also be extended to other properties of relation types , such as antisymmetry , inversion , and composition . • Entity hierarchy : Entities can be associated with an entity type . For example , the FB15k-237 entity Diane Sawyer is of type Actor . The entity types themselves are organized in a hierarchical ( hypernym ) taxonomy ; e.g . Actor → Artist → Person . Various KGE models have specifically been designed to learn entity hierarchies well [ Zhang et al. , 2020 , Kolyvakis et al. , 2020 , Balazevic et al. , 2019 , Chami et al. , 2020 , Sun et al. , 2020a ] . Tests could for example explore how well KGE models work at different hierarchy levels . • Entity distributions : Relations can be grouped into 4 categories with regards to how many correct heads/tails they may have : 1-to-1 , 1-to-many , many-to-1 and many-to-many . For example , some relations by definition can only have one correct tail as an answer ( e.g . birthPlace ) . Prior work studied how well KGE models perform in such different scenarios ( e.g . [ Bordes et al. , 2013a , Peng and Zhang , 2020 ] ) and this type of analysis could easily be added to the evaluation framework . • Robustness to adversarial attacks : Pezeshkpour et al . [ 2019 ] study how KGE models are effected by adversarial modifications . This idea could be transformed into a capability that checks how prone KGE models are to adversarial attacks . • Relation/entity frequency : The observation that KGE models perform better on frequently occurring entities/relations [ Mohamed et al. , 2020 ] could be systematically tested by creating different test sets where this frequency is varied . The tests defined for a capability explore the behavior of a model under a particular setting or condition . They can be utilised in two ways . First , they can serve as evaluation benchmarks with which different systems can be compared to each other . Based on this , we can choose the best model for a particular capability of interest . Second , they can be used to determine the failure rate of a system by making a binary decision for each triple , e.g . by defining a cut-off point for each tested triple.1 With this view , we can determine if a model is good enough to be deployed . While the latter is ultimately more important in a production setting , we focus here on the former , with which we explore how known KGE models can be compared against each other . 1 . For example , for a system to succeed a triple ’ s gold tail has to be in the top 3 of a model ’ s prediction , else it is counted as failure . Evaluation Setup . To kick-start the new evaluation framework , we define several tests for two capabilities and then use the tests to evaluate different KGE models . For the evaluation , we employ six KGE models : DistMult [ Yang et al. , 2015 ] , ComplEx [ Trouillon et al. , 2016 ] , RotatE [ Sun et al. , 2019 ] , HyperKG [ Kolyvakis et al. , 2020 ] , LinearRE [ Peng and Zhang , 2020 ] , HAKE [ Zhang et al. , 2020 ] . For an overview of their scoring functions , see Table 1 in the appendix . The first capability we explore is relation symmetry . All proposed methods can , in principle , learn that a relation type is symmetric . DistMult in particular treats all relation types as such . However , the main evaluation for these models was conducted on the standard test sets with averaged metrics . While ablation studies regarding symmetric capability are sometimes included [ Sun et al. , 2019 , Peng and Zhang , 2020 ] ,2 they are neither systematic nor comparable across papers . Therefore , it is difficult to judge to what degree they can handle symmetry and which model might be the best with regards to this capability . To be able to answer this question , we design targeted tests which allow us to measure the performance specifically for symmetric capability . In turn , we can analyse in detail under which settings which model performs best . The second capability we investigate is entity hierarchy . Many KGs are hierarchical by nature ; as an example see Figure 3 for entity types in DBpedia . This encodes important additional knowledge ; for instance , certain relations only apply to certain entity types . For example , the relation starsIn is typically associated with tail entities of type Actor , which is at level 4 in the hierarchy ( see Figure 3 ) . In contrast , the relation birthPlace can have tail entities of the higher ranked level 1 type Place ( e.g . City or Town , because both entity types are a type of Place ) . For this capability , we design tests that explore how well KGE models have learnt to handle different entity types at varying levels in the entity type hierarchy . For this capability , we can test whether KGE models that have specifically been designed to learn hierarchical structure ( HyperKG , HAKE ) perform better than other models . The models are trained on FB15k-273 [ Toutanova et al. , 2015 ] and we evaluate the models which achieved highest performance on the original FB15k-273 test set . All hyperparameters are included in Section C of the appendix .

Summary: This paper proposes a method for performing behavior testing for KG evaluation. Automated metrics are known to misrepresent the actual performance of the systems (see [1] for a critique of automated metrics for text generation). Additionally, automated metrics might only capture a subset of desired properties that a system is expected to have. Thus, any effort that aims to rethink evaluation strategies should be welcome by the community.