Toward Understanding The Effect of Loss Function on The Performance of Knowledge Graph Embedding

1 INTRODUCTION . Knowledge is considered as commonsense facts and other information accumulated from different sources . A Knowledge Graph ( KG ) is collection of facts and is usually represented as a set of triples ( h , r , t ) where h , t are entities and r is a relation , e.g . ( iphone , hyponym , smartphone ) . Entities and relations are nodes and edges in the graph , respectively . As KGs are inherently incomplete , making prediction of missing links is a fundamental task in knowlege graph analyses . Among different approaches used for KG completion , KG Embedding ( KGE ) has recently received growing attentions . KGE embeds entities and relations as low dimensional vectors known as embeddings . To measure the degree of plausibility of a triple , a scoring function is defined over the embeddings . TransE , Translation-based Embedding model , ( Bordes et al. , 2013 ) is one of the most widely used KGE models . The original assumption of TransE is to hold : h + r = t , for every positive triple ( h , r , t ) where h , r , t ∈ Rd are embedding vectors of head ( h ) , relation ( r ) and tail ( t ) respectively . TransE and its many variants like TransH ( Wang et al. , 2014 ) and TransR ( Lin et al. , 2015b ) , underperform greatly compared to the current state-of-the-art models . That is reported to be due to the limitations of their scoring functions . For instance , ( Wang et al. , 2018 ) reports that TransE can not encode a relation pattern which is neither reflexive nor irreflexive . In most of these works the effect of the loss function is ignored and the provided proofs are based on the assumptions that are not fulfilled by the associated loss functions . For instance ( Sun et al. , 2019 ) proves that TransE is incapable of encoding symmetric relation . To this end the loss function must enforce the distance of ‖h + r− t‖ to zero , but this is never fulfilled ( or even approximated ) by the employed loss function . Similarly , ( Wang et al. , 2018 ) reports that TransE can not encode a relation pattern which is neither reflexive nor irreflexive and ( Wang et al. , 2014 ) adds that TransE can not properly encode reflexive , one-to-many , many-to-one and many-to-many relations . However , as mentioned earlier , such reported limitations are not accurate and the problem is not fully investigated due to the effect of the loss function . In this regards , although TransH , TransR and TransD ( Wang et al. , 2014 ; Lin et al. , 2015b ; Ji et al. , 2015 ) addressed the reported problem of TransE in one-to-many , many-to-one , many-to-many and reflexive etc , they were misled by the assumption ( enforcing ‖h + r− t‖ to be zero ) that was not fulfilled by the employed loss function . Considering the same assumption , ( Kazemi & Poole , 2018 ) investigated three additional limitations of TransE , FTransE ( Feng et al. , 2016 ) , STransE ( Nguyen et al. , 2016 ) , TransH and TransR models : ( i ) if the models encode a reflexive relation r , they automatically encode symmetric , ( ii ) if the models encode a reflexive relation r , they automatically encode transitive and , ( iii ) if entity e1 has relation r with every entity in ∆ ∈ E and entity e2 has relation r with one of entities in ∆ , then e2 must have the relation r with every entity in ∆ . Assuming that the loss function enforces the norm to be zero , the aforementioned works have investigated these limitations by focusing on the capability of scoring functions in encoding relation patterns . However , we prove that the selection of loss function affects the boundary of score functions ; consequently , the selection of loss functions significantly affects the limitations . Therefore , the above mentioned theories corresponding to the limitations of translation-based embedding models in encoding relation patterns are inaccurate . We pose new theories about the limitations of TransX ( X=H , D , R , etc ) models considering the loss functions . To the best of our knowledge , it is the first time that the effect of loss function is investigated to prove theories corresponding to the limitations of translation-based models . In a nutshell , the key contributions of this paper is as follows . ( i ) We show that different loss functions enforce different upper-bounds and lower-bounds for the scores of positive and negative samples respectively . This implies that existing theories corresponding the limitation of TransX models are inaccurate because the effect of loss function is ignored . We introduce new theories accordingly and prove that the proper selection of loss functions mitigates the main limitations . ( ii ) We reformulate the existing loss functions and their optimization problems as an standard constrained optimization problem . This makes perfectly clear that how each of the loss functions affect on the boundary of triples scores and consequently ability of relation pattern encoding . ( iii ) Using symmetric relation patterns , we obtain a proper upper-bound of positive triples score to enable encoding of symmetric patterns . ( iv ) We additionally investigate the theoretical capability of translation-based embedding model when translation is applied in the complex space ( TransComplEx ) . We show that TransComplEx is a more powerful embedding model with fewer theoretical limitations in encoding different relation patterns such as symmetric while it is efficient in memory and time . 2 RELATED WORKS . Most of the previous work have investigated the capability of translation-based class of embedding models considering solely the formulation of the score function . Accordingly , in this section , we review the score functions of TransE and some of its variants together with their capabilities . Then , in the next section the existing limitations of Translation-based embedding models emphasized in recent works are reviewed . These limitations will be reinvestigated in the light of score and loss functions in the section 4 . The score of TransE ( Bordes et al. , 2013 ) is defined as : fr ( h , t ) = ‖h + r − t‖ . TransH ( Wang et al. , 2014 ) projects each entity ( e ) to the relation space ( e⊥ = e − wrewTr ) . The score function is defined as fr ( h , t ) = ‖h⊥ + r − t⊥‖ . TransH can encode reflexive , one-to-many , many-to-one and many-to-many relations . However , recent theories ( Kazemi & Poole , 2018 ) prove that encoding reflexive results in encoding the both symmetric and transitive which is undesired . TransR ( Lin et al. , 2015b ) projects each entity ( e ) to the relation space by using a matrix provided for each relation ( e⊥ = eMr , Mr ∈ Rde×dr ) . TransR uses the same scoring function as TransH . TransD ( Ji et al. , 2015 ) provides two vectors for each individual entities and relations ( h , hp , r , rp , t , tp ) . Head and tail entities are projected by using the following matrices : Mrh = rTp hp + I m×n , Mrt = rTp tp + I m×n . The score function of TransD is similar to the score function of TransH . RotatE ( Sun et al. , 2019 ) rotates the head to the tail entity by using relation . RotatE embeds entities and relations in the Complex space . By inclusion of constraints on the norm of entity vectors , the model would be degenerated to TransE . The scoring function of RotatE is fr ( h , t ) = ‖h ◦ r − t‖ , where h , r , t ∈ Cd , and ◦ is element-wise product . RotatE obtains the state-of-the-art results using very big embedding dimension ( 1000 ) and a lot of negative samples ( 1000 ) . TorusE ( Ebisu & Ichise , 2018 ) fixes the problem of regularization in TransE by applying translation on a compact Lie group . The model has several variants including mapping from torus to Complex space . In this case , the model is regarded as a very special case of RotatE Sun et al . ( 2019 ) that applies rotation instead of translation in the target the Complex space . According to Sun et al . ( 2019 ) , TorusE is not defined on the entire Complex space . Therefore , it has less representation capacity . TorusE needs a very big embedding dimension ( 10000 as reported in Ebisu & Ichise ( 2018 ) ) which is a limitation . 3 THE MAIN LIMITATIONS OF TRANSLATION-BASED EMBEDDING MODELS . We review six limitations of translation-based embedding models in encoding relation patterns ( e.g . reflexive , symmetric ) mentioned in the literature ( Wang et al. , 2014 ; Kazemi & Poole , 2018 ; Wang et al. , 2018 ; Sun et al. , 2019 ) . Limitation L1 . TransE can not encode reflexive relations when the relation vector is non-zero ( Wang et al. , 2014 ) . Limitation L2 . TransE can not encode a relation which is neither reflexive nor irreflexive . To see that , if TransE encodes a relation r , which is neither reflexive nor irreflexive we have h1 + r = h1 and h2 + r 6= h2 , resulting r = 0 , r 6= 0 which is a contradiction ( Wang et al. , 2018 ) . Limitation L3 . TransE can not properly encode symmetric relation when r 6= 0 . To see that ( Sun et al. , 2019 ) , if r is symmetric , then we have : h + r = t and t + r = h. Therefore , r = 0 and so all entities appeared in head or tail parts of training triples will have the same embedding vectors . The following limitations hold for TransE , FTransE , STransE , TransH and TransR ( Feng et al. , 2016 ; Nguyen et al. , 2016 ; Kazemi & Poole , 2018 ) : Limitation L4 . If r is reflexive on ∆ ∈ E , where E is the set of all entities in the KG , then r must also be symmetric . Limitation L5 . If r is reflexive on ∆ ∈ E , r must also be transitive . Limitation L6 . If entity e1 has relation r with every entity in ∆ ∈ E and entity e2 has relation r with one of entities in ∆ , then e2 must have the relation r with every entity in ∆ . 4 OUR MODEL . TransE and its variants underperform compared to other embedding models due to their limitations we iterated in Section 3 . In this section , we reinvestigate the limitations . We show that the corresponding theoretical proofs are inaccurate because the effect of loss function is ignored . So we propose new theories and prove that each of the limitations of TransE are resolved by revising either the scoring function or the loss . In this regard , we consider several loss functions and their effects on the boundary of the TransE scoring function . For each of the loss functions , we pose theories corresponding to the limitations . We additionally investigate the limitations of TransE using each of the loss functions while translation is performed in the complex space and show that by this new approach the aforementioned limitations are lifted . Our new model , TransComplEx , with a proper selection of loss function addresses the above problems .

Summary: In this paper, the authors investigate the main limitations of TransE in the light of loss function. The authors claim that their contributions consist of two parts: 1) proving that the proper selection of loss functions is vital in KGE; 2) proposing a model called TransComplEx. The results show that the proper selection of the loss function can mitigate the limitations of TransX (X=H, D, R, etc) models.