You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings

1 INTRODUCTION . Knowledge graph embedding ( KGE ) models learn algebraic representations , termed embeddings , of the entities and relations in a knowledge graph . They have been successfully applied to knowledge graph completion ( Nickel et al. , 2015 ) as well as in downstream tasks and applications such as recommender systems ( Wang et al. , 2017 ) or visual relationship detection ( Baier et al. , 2017 ) . A vast number of different KGE models for multi-relational link prediction have been proposed in the recent literature ; e.g. , RESCAL ( Nickel et al. , 2011 ) , TransE ( Bordes et al. , 2013 ) , DistMult , ComplEx ( Trouillon et al. , 2016 ) , ConvE ( Dettmers et al. , 2018 ) , TuckER ( Balazevic et al. , 2019 ) , RotatE ( Sun et al. , 2019a ) , SACN ( Shang et al. , 2019 ) , and many more . Model architectures generally differ in the way the entity and relation embeddings are combined to model the presence or absence of an edge ( more precisely , a subject-predicate-object triple ) in the knowledge graph ; they include factorization models ( e.g. , RESCAL , DistMult , ComplEx , TuckER ) , translational models ( TransE , RotatE ) , and more advanced models such as convolutional models ( ConvE ) . In many cases , the introduction of new models went along with new approaches for training these models— e.g. , new training types ( such as negative sampling or 1vsAll scoring ) , new loss functions ( such as ∗Contributed equally . { daniel , broscheit } @ informatik.uni-mannheim.de †rgemulla @ uni-mannheim.de pairwise margin ranking or binary cross entropy ) , new forms of regularization ( such as unweighted and weighted L2 ) , or the use of reciprocal relations ( Kazemi & Poole , 2018 ; Lacroix et al. , 2018 ) — and ablation studies were not always performed . Table 1 shows an overview of selected models and training techniques along with the publications that introduced them . The diversity in model training makes it difficult to compare performance results for various model architectures , especially when results are reproduced from prior studies that used a different experimental setup . Model hyperparameters are commonly tuned using grid search on a small grid involving hand-crafted parameter ranges or settings known to “ work well ” from prior studies . A grid suitable for one model may be suboptimal for another , however . Indeed , it has been observed that newer training strategies can considerably improve model performance ( Kadlec et al. , 2017 ; Lacroix et al. , 2018 ; Salehi et al. , 2018 ) . In this paper , we take a step back and aim to summarize and quantify empirically the impact of different model architectures and different training strategies on model performance . We performed an extensive set of experiments using popular model architectures and training strategies in a common experimental setup . In contrast to most prior work , we considered many training strategies as well as a large hyperparameter space , and we performed model selection using quasi-random search ( instead of grid search ) followed by Bayesian optimization . We found that this approach was able to find good ( and often superior to prior studies ) model configurations with relatively low effort . Our study complements and expands on the results of Kotnis & Nastase ( 2018 ) ( focus on negative sampling ) and Mohamed et al . ( 2019 ) ( focus on loss functions ) as well as similar studies in other areas , including language modeling ( Melis et al. , 2017 ) , generative adversarial networks ( Lucic et al. , 2018 ) , or sequence tagging ( Reimers & Gurevych , 2017 ) . We found that when trained appropriately , the performance of a particular model architecture can by far exceed the performance observed in older studies . For example , RESCAL ( Nickel et al. , 2011 ) , which constitutes one of the first KGE models but is rarely considered in newer work , showed very strong performance in our study : it was competitive to or outperformed more recent architectures such as ConvE ( Dettmers et al. , 2018 ) and TuckER ( Balazevic et al. , 2019 ) . More generally , we found that the relative performance differences between various model architectures often shrunk and sometimes even reversed when compared to prior results . This suggests that ( at least currently ) training strategies have a significant impact on model performance and may account for a substantial fraction of the progress made in recent years . We also found that suitable training strategies and hyperparameter settings vary significantly across models and datasets , indicating that a small search grid may bias results on model performance . Fortunately , as indicated above , large hyperparameter spaces can be ( and should be ) used with little additional training effort . To facilitate such efforts , we provide implementations of relevant training strategies , models , and evaluation methods as part of the open source LibKGE framework,1 which emphasizes reproducibility and extensibility . Our study focuses solely on pure KGE models , which do not exploit auxiliary information such as textual data or logical rules ( Wang et al. , 2017 ) . Since many of the studies on these non-pure models did not ( and , to be fair , could not ) use current training strategies and consequently underestimated the performance of pure KGE models , their results and conclusions need to be revisited . 1https : //github.com/uma-pi1/kge 2 KNOWLEDGE GRAPH EMBEDDINGS : MODELS , TRAINING , EVALUATION . The literature on KGE models is expanding rapidly . We review selected architectures , training methods , and evaluation protocols ; see Table 1 . The table examplarily indicates that new model architectures are sometimes introduced along with new training strategies ( marked bold ) . Reasonably recent survey articles about KGE models include Nickel et al . ( 2015 ) and Wang et al . ( 2017 ) . Multi-relational link prediction . KGE models are typically trained and evaluated in the context of multi-relational link prediction for knowledge graphs ( KG ) . Generally , given a set E of entities and a set R of relations , a knowledge graph K ⊆ E × R × E is a set of subject-predicate-object ( spo ) triples . The goal of multi-relational link prediction is to “ complete the KG ” , i.e. , to predict true but unobserved triples based on the information in K. Common approaches include rule-based methods ( Galarraga et al. , 2013 ; Meilicke et al. , 2019 ) , KGE methods ( Nickel et al. , 2011 ; Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Dettmers et al. , 2018 ) , and hybrid methods ( Guo et al. , 2018 ) . Knowledge graph embeddings ( KGE ) . A KGE model associates with each entity i ∈ E and relation k ∈ R an embedding ei ∈ Rde and rk ∈ Rdr in a low-dimensional vector space , respectively ; here de , dr ∈ N+ are hyperparameters for the embedding size . Each particular model uses a scoring function s : E × R × E → R to associate a score s ( i , k , j ) with each potential triple ( i , k , j ) ∈ E × R × E . The higher the score of a triple , the more likely it is considered to be true by the model . The scoring function takes form s ( i , k , j ) = f ( ei , rk , ej ) , i.e. , depends on i , k , and j only through their respective embeddings . Here f , which represents the model architecture , may be either a fixed function or learned ( e.g. , f may be a parameterized function ) . Evaluation . The most common evaluation task for KGE methods is entity ranking , which is a form of question answering . The available data is partitioned into a set of training , validation , and test triples . Given a test triple ( i , k , j ) ( unseen during training ) , the entity ranking task is to determine the correct answer—i.e. , the missing entity j and i , resp.—to questions ( i , k , ? ) and ( ? , k , j ) . To do so , potential answer triples are first ranked by their score in descending order . All triples but ( i , k , j ) that occur in the training , validation , or test data are subsequently filtered out so that other triples known to be true do not affect the ranking . Finally , metrics such as the mean reciprocal rank ( MRR ) of the true answer or the average HITS @ k are computed ; see the appendix . KGE models . KGE model architectures differ in their scoring function . We can roughly classify models as decomposable or monolithic : the former only allow element-wise interactions between ( relation-specific ) subject and object embeddings , whereas the latter allow arbitrary interactions . More specifically , decomposable models use scoring functions of form s ( i , k , j ) = f ( ∑ z g ( [ h1 ( ei , er ) ◦ h2 ( er , ej ) ] z ) ) , where ◦ is any element-wise function ( e.g. , multiplication ) , h1 and h2 are functions that obtain relation-specific subject and object embeddings , resp. , and g and f are scalar functions ( e.g. , identity or sigmoid ) . The most popular models in this category are perhaps RESCAL ( Nickel et al. , 2011 ) , TransE ( Bordes et al. , 2013 ) , DistMult ( Yang et al. , 2015 ) , ComplEx ( Trouillon et al. , 2016 ) , and ConvE ( Dettmers et al. , 2018 ) . RESCAL ’ s scoring function is bilinear in the entity embeddings : it uses s ( i , k , j ) = eTi Rkej , where Rk ∈ Rde×de is a matrix formed from the entries of rk ∈ Rdr ( where dr = d2e ) . DistMult and ComplEx can be seen as constrained variants of RESCAL with smaller relation embeddings ( dr = de ) . TransE is a translation-based model and uses negative distance −‖ei + rk − ej‖p between ei + rk and ej as score , commonly using the L1 norm ( p = 1 ) or the L2 norm ( p = 2 ) . Finally , ConvE uses a 2D convolutional layer and a large fully connected layer to obtain relation-specific entity embeddings ( i.e. , in h1 above ) . Other recent examples for decomposable models include TuckER ( Balazevic et al. , 2019 ) , RotatE ( Sun et al. , 2019a ) , and SACN ( Shang et al. , 2019 ) . Decomposable models are generally fast to use : once the relation-specific embeddings are ( pre- ) computed , score computations are cheap . Monolithic models—such as ConvKB or KBGAT—do not decompose into relation-specific embeddings : they take form s ( i , k , j ) = f ( ei , rk , ej ) . Such models are more flexible , but they are also considerably more costly to train and use . It is currently unclear whether monolithic models can achieve comparable or better performance than decomposable models Sun et al . ( 2019b ) . Training type . There are three commonly used approaches to train KGE models , which differ mainly in the way negative examples are generated . First , training with negative sampling ( NegSamp ) ( Bordes et al. , 2013 ) obtains for each positive triple t = ( i , k , j ) from the training data a set of ( pseudo- ) negative triples obtained by randomly perturbing the subject , relation , or object position in t ( and optionally verifying that the so-obtained triples do not exist in the KG ) . An alternative approach ( Lacroix et al. , 2018 ) , which we term 1vsAll , is to omit sampling and take all triples that can be obtained by perturbing the subject and object positions as negative examples for t ( even if these tuples exist in the KG ) . 1vsAll is generally more expensive than NegSamp , but it is feasible ( and even surprisingly fast in practice ) if the number of entities is not excessively large . Finally , Dettmers et al . ( 2018 ) proposed a training type that we term KvsAll2 : this approach ( i ) constructs batches from non-empty rows ( i , k , ∗ ) or ( ∗ , k , j ) instead of from individual triples , and ( ii ) labels all such triples as either positive ( occurs in training data ) or negative ( otherwise ) . Loss functions . Several loss functions for training KGEs have been introduced so far . RESCAL originally used squared error between the score of each triple and its label ( positive or negative ) . TransE used pairwise margin ranking with hinge loss ( MR ) , where each pair consists of a positive triple and one of its negative triples ( only applicable to NegSamp and 1vsAll ) and the margin η is a hyperparameter . Trouillon et al . ( 2016 ) proposed to use binary cross entropy ( BCE ) loss : it applies a sigmoid to the score of each ( positive or negative ) triple and uses the cross entropy between the resulting probability and that triple ’ s label as loss . BCE is suitable for multi-class and multi-label classification . Finally , Kadlec et al . ( 2017 ) used cross entropy ( CE ) between the model distribution ( softmax distribution over scores ) and the data distribution ( labels of corresponding triples , normalized to sum to 1 ) . CE is more suitable for multi-class classification ( as in NegSamp and 1vsAll ) , but it has also been used in the multi-label setting ( KvsAll ) . Mohamed et al . ( 2019 ) found that the choice of loss function can have a significant impact on model performance , and that the best choice is data and model dependent . Our experimental study provides additional evidence for this finding . Reciprocal relations . Kazemi & Poole ( 2018 ) and Lacroix et al . ( 2018 ) introduced the technique of reciprocal relations into KGE training . Observe that during evaluation and also most training methods discussed above , the model is solely asked to score subjects ( for questions of form ( ? , k , j ) ) or objects ( for questions of form ( i , k , ? ) ) . The idea of reciprocal relations is to use separate scoring functions ssub and sobj for each of these tasks , resp . Both scoring functions share entity embeddings , but they do not share relation embeddings : each relation thus has two embeddings.3 The use of reciprocal relations may decrease the computational cost ( as in the case of ConvE ) , and it may also lead to better model performance Lacroix et al . ( 2018 ) ( e.g. , for relations in which one direction is easier to predict ) . On the downside , the use of reciprocal relations means that a model does not provide a single triple score s ( i , k , j ) anymore ; generally , ssub ( i , k , j ) 6= sobj ( i , k , j ) . Kazemi & Poole ( 2018 ) proposed to take the average of the two triple scores and explored the resulting models . Regularization . The most popular form of regularization in the literature is L2 regularization on the embedding vectors , either unweighted or weighted by the frequency of the corresponding entity or relation ( Yang et al. , 2015 ) . Lacroix et al . ( 2018 ) proposed to use L3 regularization . TransE normalized the embeddings to unit norm after each update . ConvE used dropout ( Srivastava et al. , 2014 ) in its hidden layers ( and only in those ) . In our study , we additionally consider L1 regularization and the use of dropout on the entity and/or relation embeddings . Hyperparameters . Many more hyperparameters have been used in prior work . This includes , for example , different methods to initialize model parameters , different optimizers , different optimizer parameters such as the learning rate or batch size , the number of negative examples for NegSamp , the regularization weights for entities and relations , and so on . To deal with such a large search space , most prior studies favor grid search over a small grid where most of these hyperparameters remain fixed . As discussed before , this approach may lead to bias in the results , however .

Summary: The paper presents an experimental study about some KGE methods. It argues that papers often propose changes in several different dimensions, such as model, loss, training, regularizer, etc., at once without providing a sufficient investigation about the individual components' contributions. The experimental study considers two datasets (FB15k-237 and WNRR) and five different models (RESCAL, TransE, DistMult, ComplEx, ConvE). The models were selected using a quasi-random hyperparameter search, followed by a short Bayesian optimization phase to fine-tune the parameters. The performance of the best models found during this hyperparameter search are compared to first published results for the same model, as well as to a small selection of recent papers. To analyse the influence of single hyperparameters, the best found configuration is compared to the best configuration which does not use this specific value for the given hyperparameter.