On Understanding Knowledge Graph Representation

1 INTRODUCTION . Knowledge graphs are large repositories of binary relations between words ( or entities ) in the form of fact triples ( subject , relation , object ) . Many models have been developed for learning representations of entities and relations in knowledge graphs , such that known facts can be recalled and previously unknown facts can be inferred , a task known as link prediction . Recent link prediction models ( e.g . Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Balažević et al. , 2019b ) learn entity representations , or embeddings , of far lower dimensionality than the number of entities , by capturing latent structure in the data . Relations are typically represented as a mapping from the embedding of a subject entity to its related object entity embedding ( s ) . Although the performance of knowledge graphlink prediction models has steadily improved for nearly a decade , relatively little is understood of the low-rank latent structure that underpins these models , which we address in this work . We start by drawing a parallel between entity embeddings in knowledge graphs and unsupervised word embeddings , as learned by algorithms such as Word2Vec ( W2V ) ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) . We assume that words have latent features , e.g . meaning ( s ) , tense , grammatical type , that are innate and fixed , irrespective of what an embedding may capture ( which may be only a part , subject to the embedding method and/or the data source ) ; and that this same latent structure gives rise to patterns observed in the data , e.g . in word co-occurrence statistics and in which words are related to which . As such , an understanding of the latent structure from one embedding task ( e.g . word embedding ) might be useful to another ( e.g . knowledge graph entity embedding ) . Recent work ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) theoretically explains how semantic properties are encoded in word embeddings that ( approximately ) factorise a matrix of word cooccurrence pointwise mutual information ( PMI ) , e.g . as is known for W2V ( Levy & Goldberg , 2014 ) . Semantic relationships between words ( specifically similarity , relatedness , paraphrase and analogy ) are proven to manifest as linear relationships between rows of the PMI matrix ( subject to known error terms ) , of which word embeddings can be considered low-rank projections . This explains why similar words ( e.g . synonyms ) have similar embeddings ; and embeddings of analogous word pairs share a common “ vector offset ” . Importantly , this insight allows us to identify geometric relationships between such word embeddings necessary for other semantic relations to hold , such as those of knowledge graphs . These relation conditions describe relation-specific mappings between entity embeddings , i.e . relation representations , providing a “ blue-print ” against which to consider knowledge graph representation models . We find that various properties of knowledge graph representation models , including the relative performance of leading link prediction models , accord with predictions based on these relation conditions , suggesting a commonality to the latent structure learned in word embedding models and knowledge graph representation models , despite the significant differences between their training data and methodology . In summary , the key contributions of this work are : • to use recent understanding of PMI-based word embeddings to derive what a relation representation must achieve to map a subject word embedding to all related object word embeddings ( relation conditions ) , based on which relations can be categorised into three types ; • to show that properties of knowledge graph models fit predictions made from relation conditions , e.g . strength of a relation ’ s relatedness aspect is reflected in the eigenvalues of its relation matrix ; • to show that the performance per relation of leading link prediction models corresponds to the ability of the model ’ s architecture to meet the relation conditions of the relation ’ s type , i.e . the better the architecture of a knowledge graph representation model aligns with the form theoretically derived for PMI-based word embeddings , the better the model performs ; and • noting how ranking metrics can be flawed , to provide novel insight into the prediction accuracy per relation of recent knowledge graph models , an evaluation metric we recommend in future . 2 BACKGROUND . Our work draws on knowledge graph representation and word embedding . Whilst related , these tasks differ materially in their training data . The former is restricted to datasets crafted by hand or automatically generated , the latter has the vast abundance of natural language text ( e.g . Wikipedia ) . 2.1 KNOWLEDGE GRAPH REPRESENTATION . Almost all recent knowledge graph models represent entities es , eo as vectors es , eo ∈ Rde of low dimension ( e.g . de=200 ) relative to the number of entities ne ( typically of order 104 ) , and relations as transformations in the latent space from subject entity embedding to object . These models are distinguished by their score function , which defines ( i ) the form of the relation transformation , e.g . matrix multiplication , vector addition ; and ( ii ) how “ closeness ” between the transformed subject embedding and an object embedding is evaluated , e.g . dot product , Euclidean distance . Score functions can be non-linear ( e.g . Dettmers et al . ( 2018 ) ) , or linear and sub-categorised as additive , multiplicative or both . We focus on linear models due to their simplicity and strong performance at link prediction ( including state-of-the-art ) . Table 1 shows the score functions of competitive linear models that span all linear sub-categories : TransE ( Bordes et al. , 2013 ) , DistMult ( Yang et al. , 2015 ) , TuckER ( Balažević et al. , 2019b ) and MuRE ( Balažević et al. , 2019a ) . Additive models typically use Euclidean distance and contain a relation-specific translation from a ( possibly transformed ) subject to a ( possibly transformed ) object entity embedding . A generic additive score function is given by φ ( es , r , eo ) =−‖Rses+r−Roeo‖22+bs+bo=−‖e ( r ) s +r−e ( r ) o ‖22+bs+bo . The simplest example is TransE for which Rs=Ro=I and bs=bo=0 . The score function of MuRE has Ro=I and so combines multiplicative ( Rs=R ) and additive ( r ) components . Multiplicative models have the generic score function φ ( es , r , eo ) = e > s Reo = 〈e ( r ) s , eo〉 , i.e . a bilinear product of the entity embeddings and a relation-specific matrix R. DistMult is a simple example with diagonal R and so can not model asymmetric relations ( Trouillon et al. , 2016 ) . In TuckER , each relation-specific R = W×3 r is a linear combination of dr “ prototype ” relation matrices in a core tensor W∈Rde×dr×de ( where ×n denotes tensor product along mode n ) , facilitating multi-task learning across relations . 2.2 WORD EMBEDDING . Algorithms such as Word2Vec ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) generate succinct low-rank word embeddings that perform well on downstream tasks ( Baroni et al. , 2014 ) . Such models predict the context words ( cj ) observed around each target word ( wi ) in a text corpus using shallow neural networks . Whilst recent language models ( e.g . Devlin et al . ( 2018 ) ; Peters et al . ( 2018 ) ) create impressive context-specific word embeddings , we focus on the former embeddings since knowledge graph entities have no obvious context and , more importantly , they are interpretable . Levy & Goldberg ( 2014 ) show that , for a dictionary of unique words D and embedding dimension d |D| , W2V ’ s loss function is minimised when its weight matrices W , C∈Rd×|D| ( whose columns are word embeddings wi , cj ) factorise a word co-occurrence pointwise mutual information ( PMI ) matrix , subject to a shift term ( PMI ( wi , cj ) =log P ( wi , cj ) P ( wi ) P ( cj ) ) . This connects W2V to earlier countbased embeddings and specifically to PMI , which has a long history in linguistic analysis ( Turney & Pantel , 2010 ) . From its loss function , GloVe can be seen to perform a comparable factorisation . Recent work shows why word embeddings that factorise such PMI matrix encode semantic word relationships ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) . The authors show that word embeddings can be seen as low-rank projections of high dimensional PMI vectors ( rows of the PMI matrix ) , between which the semantic relationships of similarity , relatedness , paraphrase and analogy provably manifest as linear geometric relationships ( subject to defined error terms ) , which are then preserved , under a sufficiently linear projection , between word embeddings . Thus similar words have similar embeddings , and the embeddings of analogous word pairs share a common vector offset . Specifically , the PMI vectors ( px ) of an analogy “ man is to king as woman is to queen ” satisfy pQ−pW≈pK−pM because the difference between words associated with king and man ( e.g . reign , crown ) mirrors that between queen and woman . This leads to a common difference between their co-occurrence distributions ( over all words ) , giving a common difference between their PMI vectors , which projects to a common difference between embeddings . Any discrepancy in the mirroring of word associations is shown to introduce error , weakening the analogy , as does a lack of statistical independence within certain word pairs ( see ( Allen & Hospedales , 2019 ) ) . The common difference in word co-occurrence distributions , e.g . the increased association with words { reign , crown , etc . } , can be interpreted semantically as a common change in context ( context-shift ) that transforms man to king and woman to queen by adding a royal context . Under this interpretation , context can also be subtracted , e.g . “ king is to man as queen is to woman ” ( minus royal ) ; or both , e.g . “ boy is to king as girl is to queen ” ( minus youth plus royal ) . Adding context can also be interpreted as specialisation , and subtracting context as generalisation . This establishes a correspondence between common word embedding vector offsets and semantic context-shifts . Although the projection from PMI vectors to word embeddings preserves the relative relationships , and thus the above semantic interpretability of common embedding differences , a direct interpretation of dimensions themselves is obscured , not least because any embedding matrix can be arbitrarily scaled/rotated if the other is inversely transformed . 3 RELATIONSHIPS BETWEEN EMBEDDINGS OF RELATED WORDS . Our aim is to build on the understanding of PMI-based word embeddings ( henceforth word embeddings ) , to identify what a knowledge graph relation representation needs to achieve to map all subject word embeddings to all related object word embeddings . We note that if a semantic relation between two words implies a particular geometric relationship between their embeddings , then the latter serves as a necessary quantitative condition for the former to hold ( a relation condition ) . Relation conditions implicitly define a relation-specific mapping by which all subject embeddings are mapped to all related object embedding ( s ) , allowing related entities to be identified by a proximity measure ( e.g . Euclidean distance or dot product ) . Since this is the approach of many knowledge graph representation models , their performance can be contrasted with their ability to express mappings that satisfy required relation conditions . For example , similarity and context-shift relations respectively imply closeness and a relation-specific vector offset between embeddings ( S2.2 ) . Such relation conditions can be tested for by respectively making no change to the subject entity or adding the relation-specific offset , before measuring proximity with the object . We note that since relation conditions are not necessarily sufficient , they do not guarantee a relation holds , i.e . false positives may arise . In general , the data from which knowledge graph embeddings are derived differs greatly to the co-occurrence data used for word embeddings , and the latter would not be anticipated to be learned by knowledge graph models . However , word embeddings provide a known solution ( i.e . minimise the loss function ) of any knowledge graph model able to express the required mapping ( s ) derived from relation conditions , where the loss function measures proximity between mapped entities . The relation conditions for certain relation types ( underlined ) follow readily from S2.2 : • Similarity : Semantically similar words induce similar distributions over the words they co-occur with . Thus their PMI vectors ( Fig 1a ) and word embeddings are similar . • Relatedness : The relatedness of two words can be defined in terms of the words with which both co-occur similarly ( S∈D ) , which define the nature of relatedness , e.g . milk and cheese are related by S= { dairy , breakfast , ... } ; and |S| reflects the strength of relatedness . Since PMI vector components corresponding to S are similar ( Fig 1b ) , embeddings of “ S-related ” words have similar components in the subspace VS that spans the projected PMI vector dimensions corresponding to S. The rank of VS might be expected to reflect relatedness strength . In general , relatedness is a weaker , more variable relation than similarity , its limiting case with S=D and rank ( VS ) =d . • Context-shift : In the context of word embeddings , analogy typically refers to relational similarity ( Turney , 2006 ; Gladkova et al. , 2016 ) . More specifically , the relations within analogies that give a common vector offset between word embeddings require related words to have a common difference between their distributions of co-occurring words , defined as a context-shifts ( see S2.2 ) . These relations are strictly 1-to-1 and include an aspect of relatedness due to the word associations in common ( Fig 1d ) . A specialisation relation is a context-shift in which context is only added ( Fig 1c ) . • Generalised context-shift : Context-shift relations are generalised to 1-to-many , many-to-1 and many-to-many relations by letting the fully-specified added or subtracted context be one of a ( relationspecific ) context set ( Fig 1e ) , e.g . allowing an entity to be any colour or anything blue . The potential scope and size of each context set means these relations can vary greatly . The limiting case for small context sets has a single context in each , whereby the relation is an explicit context-shift ( as above ) , and the difference between embeddings is a known vector offset . In the limiting case where context sets are large , the added/subtracted context is so loosely defined that , in effect , only the relatedness aspect of the relation and thus only the common subspace component of embeddings is known . Link to set theory : Viewing PMI vectors as sets of word associations and taking intuition from Fig 1 , the above relations can be seen to reflect set operations : similarity as set equality ; relatedness as equality of a subset ; and context-shift as the set difference equalling a relation-specific set . This highlights how the relatedness aspect of a relation reflects features that must be common , and contextshift reflects features that must differ . Whilst this mirrors an intuitive notion of “ feature vectors ” , we emphasise that this is grounded in the co-occurrence statistics of PMI-based word embeddings .

Summary: The paper attempts to understand the latent structure underlying knowledge graph embedding methods. The work can be seen as an extension of understanding of PMI-based word embedding methods. They categorize knowledge graph relations into three categories based on their relation conditions: Relatedness (R), Specialisation (S), and Context-shift (C). For each category, they evaluate a representative of different types of knowledge graph embedding methods. Through results, they demonstrate that a model’s ability to represent a specific relation type depends on the limitations imposed by the model architecture with respect to satisfying the necessary relation conditions.