Knowledge Graph Embedding: A Probabilistic Perspective and Generalization Bounds

We study theoretical properties of embedding methods for knowledge graph completion under the “ missing completely at random ” assumption . We prove generalization error bounds for this setting . Even though the missing completely at random setting may seem naive , it is actually how knowledge graph embedding methods are typically benchmarked in the literature . Our results provide , to a certain extent , an explanation for why knowledge graph embedding methods work ( as much as classical learning theory results provide explanations for classical learning from i.i.d . data ) . 1 INTRODUCTION . Many large knowledge bases such as YAGO ( Mahdisoltani et al. , 2015 ) and NELL ( Carlson et al. , 2010 ) are represented as knowledge graphs . A knowledge graph is a set of triples ( head , predicate , tail ) where head and tail are objects ( also frequently called entities ) and predicate is a binary relation . Each such triple represents a fact about the world , e.g . ( NewYork , IsLocatedIn , US ) . The objects can be seen as vertices of a graph and the relations as labeled directed edges ( pointing from head to tail ) , hence the name knowledge graph . Most knowledge bases are incomplete , i.e . there are facts that should be included in them but are not . A popular task is to automatically complete such knowledge bases . This is often done by embedding the objects and relations contained in them into some vector spaces and then using the embedding to predict the missing tuples . There has been a number of knowledge graph completion methods presented in the literature ( we give an overview of existing methods in Section 2.3 ) . However , to our best knowledge , there has not been any theoretical analysis of the generalization abilities of these methods in the spirit of classical learning theory . We make the first steps in this direction in the present paper . We derive expected error bounds for a class of knowledge graph embedding methods . A key challenge in deriving generalization bounds in this setting is that the training and test data do not satisfy assumptions typically used in learning theory . The knowledge bases that we receive contain only positive examples , that is , the facts already in the knowledge base . We do not know which of the facts not contained in the knowledge base are just missing and which are not contained in the knowledge base because they are actually false . This is known as learning from positive and unlabeled data ( PU-learning ) ; we refer to ( Bekker & Davis , 2018 ) for a recent survey . Only a few results have been obtained for generalization bounds for PU-learning so far , e.g . ( Niu et al. , 2016 ) , but they are not directly applicable to our setting ( we discuss the details in Section 10 ) . The bounds that we derive in this paper rely on an assumption on how the triples are missing from the knowledge graph . In particular , we assume that the triples are missing “ completely at random ” , i.e . that each tuple may be missing with probability δ > 0 independently of the others . Even though this setting may appear naive , it is actually exactly how some famous datasets ’ training and test sets are created , e.g . ( Bordes et al. , 2013 ) . Thus , the theoretical results actually do relate to what one observes in “ practice ” ( i.e . in the respective papers that use these datasets ) . The bounds that we derive here apply to knowledge graph embedding methods that use log-loss for learning the embeddings such as ComplEx ( Trouillon & Nickel , 2017 ) , SimplE ( Kazemi & Poole , 2018 ) or ConvE ( Dettmers et al. , 2018 ) . Main Technical Contributions Our main technical results are expected error bounds on the number of triples ( h , r , t ) that are predicted incorrectly when using knowledge graph embedding for knowledge graph completion ( stated in Theorems 4 and 5 ) , i.e . either to be in the knowledge graph when they are not supposed to be ( false positives ) or not to be in the knowledge graph when they are supposed to be ( false negatives ) . The bound assumes that the knowledge graph embedding is learned by maximizing the log-likelihood of a certain exponential-family distribution ( introduced in Section 5 ) , which turns out to be equivalent to how several prominent knowledge graph embedding methods ( Trouillon & Nickel , 2017 ; Kazemi & Poole , 2018 ; Dettmers et al. , 2018 ) are learned using log-loss ( this is discussed in detail in Section 10 ) . Our results therefore provide a statistical learning theory style justification for these knowledge graph embedding methods . In order to obtain the main results we also derive several auxiliary technical results which might be of independent interest ( in particular , Theorems 2 and 3 ) . 2 PRELIMINARIES . 2.1 KNOWLEDGE GRAPHS . Let O be a set of objects and R a set of relations . A knowledge graph is a set of triples ( h , r , t ) where h and t are objects and r is a relation . Here h is called the head of the triple and t is called the tail of the triple . In the formalism of first-order logic , a triple ( h , r , t ) represents a first-order logic atom r ( h , t ) . For instance the triple ( Alice , Knows , Bob ) corresponds to the first-order logic atom Knows ( Alice , Bob ) , which represents the fact that Alice knows Bob . Here , Alice and Bob are objects and Knows is a relation . Example 1 . Let us have the following objects and relations : O = { France , Germany , Berlin , Paris } , R = { CapitalOf , Borders } . Then the knowledge graph , representing our geographical knowledge about these countries and cities , can look as G = { ( France , Borders , Germany ) , ( Germany , Borders , France ) , ( Berlin , CapitalOf , Germany ) , ( Paris , CapitalOf , France ) } . Whereas , in this case , the knowledge graph is complete and accurate , in reality most knowledge graphs will lack some of the true tuples . The task of knowledge base completion is to add such missing facts . 2.2 KNOWLEDGE GRAPH EMBEDDING . Knowledge graph embedding methods represent knowledge graphs geometrically . They do so by representing objects and relations as vectors ( or matrices ) . Given a knowledge graph G on some sets of objects O and relations R , most knowledge graph embedding methods need three components : vector representations of objects , vector representations of relations and a function ψ ( xh , xr , xt ) which takes the representations of h , r and t and produces a score that is used to assess how likely it is that the relation ( h , r , t ) is contained in the knowledge graph . One can then compare the output of the function with a threshold to decide which triples are ( predicted to be ) in the knowledge graph . Note on notation . For a set of objects O and a set of relations R we will normally denote their vector representations by X . We will usually treat X as a look-up table . For an object o we will typically use xo to denote its vector representation from X and analogically for a relation r we will use xr , unless explicitly stated otherwise . With a slight abuse of notation , we will also sometimes treat X as a set of vectors and , for instance , write x ∈ X . Relations can be mapped to matrices ( denoted Mr ) or tensors instead of only vectors , and even if relations are mapped to vectors the dimension may be different from the dimension of entity embeddings . However , for the sake of simplicity , we will not introduce more notations , and X and ψ will also be used in these cases . 2.3 EXISTING METHODS . One can roughly categorize knowledge graph ( KG ) embedding approaches into several groups based on different scoring functions such as distance-based scoring functions and similarity-based scoring functions . The most well-known class of embedding methods using distance-based scoring functions is TransE ( Bordes et al. , 2013 ) and its variants . The scoring function of TransE is defined as the negative distance between xh + xr and xt , i.e. , ψ ( xh , xr , xt ) = −||xh + xr − xt||2 . In order to effectively model more general relations rather than only one-to-one relations , TransH has been proposed ( Wang et al. , 2014 ) . Given a triple ( h , r , t ) , the scoring function is defined as −||x⊥h + xr − x⊥t ||2 where x⊥h and x⊥t are the projections of entity embeddings xh and xt onto some hyperplane , respectively . TransR ( Lin et al. , 2015 ) , TransD ( Ji et al. , 2015 ) and TranSparse ( Ji et al. , 2016 ) are similar to TransH , allowing relation-specific embeddings in different ways . TransM ( Fan et al. , 2014 ; Xiao et al. , 2015 ) , ManifoldE ( Xiao et al. , 2016 ) and TransF ( Feng et al. , 2016 ) relax the requirement xh + xr ≈ xt . SE ( Bordes et al. , 2011 ) uses two matrices to project head and tail entities for each relation r , and the score is −||M1rxh −M2rxt||1 . RotatE ( Sun et al. , 2019 ) defines each relation as a rotation from the source entity to the target entity in the complex vector space , and its scoring function is −||xh ◦xr −xt||2 where ◦ is the entry-wise product . In the class of similarity-based methods , RESCAL ( Nickel et al. , 2011 ) and its variants are among the most well-known ones . RESCAL also assigns every entity a vector but every relation r a matrix Mr 1 . The scoring function is defined as x > hMrxt . Jenatton et al . ( 2012 ) further assumes that every Mr has the same set of eigenvectors . Garcı́a-Durán et al . ( 2014 ) adds other terms into the scoring function x > hMrxt+x > h xr+x > r xt+x > t Dxh whereD is a diagonal matrix independent of relations . DistMult ( Yang et al. , 2015 ) is a simplified version of RESCAL by restricting Mr being diagonal , and hence improves the efficiency of RESCAL . To combine the advantages of both methods , i.e. , the expressive power of RESCAL and the simplicity of DistMult , HolE ( Nickel et al. , 2016 ) has been proposed . ComplEx ( Yang et al. , 2015 ) extends DistMult by allowing complex-valued embeddings , which is more powerful in modeling asymmetric relations . Hayashi & Shimbo ( 2017 ) show that ComplEx is in fact equivalent to HolE . To capture analogical structures , ANALOGY ( Liu et al. , 2017 ) further requires Mr in RESCAL to be normal and mutually commutative . A class of KG embeddings is based on tensor factorization approaches such as the Canonical Polyadic ( CP ) decomposition ( Hitchcock , 1927 ) whose scoring function is similar to DistMult , but every entity has two independent embeddings . SimplE ( Kazemi & Poole , 2018 ) is a recent extension of CP decomposition , which ensures the two embeddings of every entity dependent . There are also a group of algorithms using neural networks for embedding , e.g. , SME ( Bordes et al. , 2014 ) , NTN and SLM ( Bordes et al. , 2014 ) , MLP ( Dong et al. , 2014 ) , NAM ( Liu et al. , 2016 ) and ConvE ( Dettmers et al. , 2018 ) . All the aforementioned embedding methods map entities and relations in a deterministic way . In KG2E ( He et al. , 2015 ) , representations of entities and relations are random vectors sampled from multivariate normal distributions , xh ∼ N ( µh , Σh ) xr ∼ N ( µr , Σr ) xt ∼ N ( µt , Σt ) . A possible scoring function is KL ( xt−xh||xr ) and another is the probability inner product ( Jebara et al. , 2004 ) of xt − xh and xr . TransG ( Han et al. , 2016 ) further models relations using mixtures of Gaussians .

Summary: This work conducts a theoretical study of the generalization bounds for the number of wrongly predicted triples of KG embedding methods. Under a "``missing completely at random'' assumption, the authors model the distributions of sampled KG and ground-truth KG, and derivate the bounds as a function of the KL-divergence by leveraging Pinsker's inequality. Preliminary discussions on how existing KG embedding methods fit into this theoretical study have been conducted.