On Understanding Knowledge Graph Representation

1 INTRODUCTION . Knowledge graphs are large repositories of binary relations between words ( or entities ) in the form of fact triples ( subject , relation , object ) . Many models have been developed for learning representations of entities and relations in knowledge graphs , such that known facts can be recalled and previously unknown facts can be inferred , a task known as link prediction . Recent link prediction models ( e.g . Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Balažević et al. , 2019b ) learn entity representations , or embeddings , of far lower dimensionality than the number of entities , by capturing latent structure in the data . Relations are typically represented as a mapping from the embedding of a subject entity to its related object entity embedding ( s ) . Although the performance of knowledge graphlink prediction models has steadily improved for nearly a decade , relatively little is understood of the low-rank latent structure that underpins these models , which we address in this work . We start by drawing a parallel between entity embeddings in knowledge graphs and unsupervised word embeddings , as learned by algorithms such as Word2Vec ( W2V ) ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) . We assume that words have latent features , e.g . meaning ( s ) , tense , grammatical type , that are innate and fixed , irrespective of what an embedding may capture ( which may be only a part , subject to the embedding method and/or the data source ) ; and that this same latent structure gives rise to patterns observed in the data , e.g . in word co-occurrence statistics and in which words are related to which . As such , an understanding of the latent structure from one embedding task ( e.g . word embedding ) might be useful to another ( e.g . knowledge graph entity embedding ) . Recent work ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) theoretically explains how semantic properties are encoded in word embeddings that ( approximately ) factorise a matrix of word cooccurrence pointwise mutual information ( PMI ) , e.g . as is known for W2V ( Levy & Goldberg , 2014 ) . Semantic relationships between words ( specifically similarity , relatedness , paraphrase and analogy ) are proven to manifest as linear relationships between rows of the PMI matrix ( subject to known error terms ) , of which word embeddings can be considered low-rank projections . This explains why similar words ( e.g . synonyms ) have similar embeddings ; and embeddings of analogous word pairs share a common “ vector offset ” . Importantly , this insight allows us to identify geometric relationships between such word embeddings necessary for other semantic relations to hold , such as those of knowledge graphs . These relation conditions describe relation-specific mappings between entity embeddings , i.e . relation representations , providing a “ blue-print ” against which to consider knowledge graph representation models . We find that various properties of knowledge graph representation models , including the relative performance of leading link prediction models , accord with predictions based on these relation conditions , suggesting a commonality to the latent structure learned in word embedding models and knowledge graph representation models , despite the significant differences between their training data and methodology . In summary , the key contributions of this work are : • to use recent understanding of PMI-based word embeddings to derive what a relation representation must achieve to map a subject word embedding to all related object word embeddings ( relation conditions ) , based on which relations can be categorised into three types ; • to show that properties of knowledge graph models fit predictions made from relation conditions , e.g . strength of a relation ’ s relatedness aspect is reflected in the eigenvalues of its relation matrix ; • to show that the performance per relation of leading link prediction models corresponds to the ability of the model ’ s architecture to meet the relation conditions of the relation ’ s type , i.e . the better the architecture of a knowledge graph representation model aligns with the form theoretically derived for PMI-based word embeddings , the better the model performs ; and • noting how ranking metrics can be flawed , to provide novel insight into the prediction accuracy per relation of recent knowledge graph models , an evaluation metric we recommend in future . 2 BACKGROUND . Our work draws on knowledge graph representation and word embedding . Whilst related , these tasks differ materially in their training data . The former is restricted to datasets crafted by hand or automatically generated , the latter has the vast abundance of natural language text ( e.g . Wikipedia ) . 2.1 KNOWLEDGE GRAPH REPRESENTATION . Almost all recent knowledge graph models represent entities es , eo as vectors es , eo ∈ Rde of low dimension ( e.g . de=200 ) relative to the number of entities ne ( typically of order 104 ) , and relations as transformations in the latent space from subject entity embedding to object . These models are distinguished by their score function , which defines ( i ) the form of the relation transformation , e.g . matrix multiplication , vector addition ; and ( ii ) how “ closeness ” between the transformed subject embedding and an object embedding is evaluated , e.g . dot product , Euclidean distance . Score functions can be non-linear ( e.g . Dettmers et al . ( 2018 ) ) , or linear and sub-categorised as additive , multiplicative or both . We focus on linear models due to their simplicity and strong performance at link prediction ( including state-of-the-art ) . Table 1 shows the score functions of competitive linear models that span all linear sub-categories : TransE ( Bordes et al. , 2013 ) , DistMult ( Yang et al. , 2015 ) , TuckER ( Balažević et al. , 2019b ) and MuRE ( Balažević et al. , 2019a ) . Additive models typically use Euclidean distance and contain a relation-specific translation from a ( possibly transformed ) subject to a ( possibly transformed ) object entity embedding . A generic additive score function is given by φ ( es , r , eo ) =−‖Rses+r−Roeo‖22+bs+bo=−‖e ( r ) s +r−e ( r ) o ‖22+bs+bo . The simplest example is TransE for which Rs=Ro=I and bs=bo=0 . The score function of MuRE has Ro=I and so combines multiplicative ( Rs=R ) and additive ( r ) components . Multiplicative models have the generic score function φ ( es , r , eo ) = e > s Reo = 〈e ( r ) s , eo〉 , i.e . a bilinear product of the entity embeddings and a relation-specific matrix R. DistMult is a simple example with diagonal R and so can not model asymmetric relations ( Trouillon et al. , 2016 ) . In TuckER , each relation-specific R = W×3 r is a linear combination of dr “ prototype ” relation matrices in a core tensor W∈Rde×dr×de ( where ×n denotes tensor product along mode n ) , facilitating multi-task learning across relations . 2.2 WORD EMBEDDING . Algorithms such as Word2Vec ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) generate succinct low-rank word embeddings that perform well on downstream tasks ( Baroni et al. , 2014 ) . Such models predict the context words ( cj ) observed around each target word ( wi ) in a text corpus using shallow neural networks . Whilst recent language models ( e.g . Devlin et al . ( 2018 ) ; Peters et al . ( 2018 ) ) create impressive context-specific word embeddings , we focus on the former embeddings since knowledge graph entities have no obvious context and , more importantly , they are interpretable . Levy & Goldberg ( 2014 ) show that , for a dictionary of unique words D and embedding dimension d |D| , W2V ’ s loss function is minimised when its weight matrices W , C∈Rd×|D| ( whose columns are word embeddings wi , cj ) factorise a word co-occurrence pointwise mutual information ( PMI ) matrix , subject to a shift term ( PMI ( wi , cj ) =log P ( wi , cj ) P ( wi ) P ( cj ) ) . This connects W2V to earlier countbased embeddings and specifically to PMI , which has a long history in linguistic analysis ( Turney & Pantel , 2010 ) . From its loss function , GloVe can be seen to perform a comparable factorisation . Recent work shows why word embeddings that factorise such PMI matrix encode semantic word relationships ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) . The authors show that word embeddings can be seen as low-rank projections of high dimensional PMI vectors ( rows of the PMI matrix ) , between which the semantic relationships of similarity , relatedness , paraphrase and analogy provably manifest as linear geometric relationships ( subject to defined error terms ) , which are then preserved , under a sufficiently linear projection , between word embeddings . Thus similar words have similar embeddings , and the embeddings of analogous word pairs share a common vector offset . Specifically , the PMI vectors ( px ) of an analogy “ man is to king as woman is to queen ” satisfy pQ−pW≈pK−pM because the difference between words associated with king and man ( e.g . reign , crown ) mirrors that between queen and woman . This leads to a common difference between their co-occurrence distributions ( over all words ) , giving a common difference between their PMI vectors , which projects to a common difference between embeddings . Any discrepancy in the mirroring of word associations is shown to introduce error , weakening the analogy , as does a lack of statistical independence within certain word pairs ( see ( Allen & Hospedales , 2019 ) ) . The common difference in word co-occurrence distributions , e.g . the increased association with words { reign , crown , etc . } , can be interpreted semantically as a common change in context ( context-shift ) that transforms man to king and woman to queen by adding a royal context . Under this interpretation , context can also be subtracted , e.g . “ king is to man as queen is to woman ” ( minus royal ) ; or both , e.g . “ boy is to king as girl is to queen ” ( minus youth plus royal ) . Adding context can also be interpreted as specialisation , and subtracting context as generalisation . This establishes a correspondence between common word embedding vector offsets and semantic context-shifts . Although the projection from PMI vectors to word embeddings preserves the relative relationships , and thus the above semantic interpretability of common embedding differences , a direct interpretation of dimensions themselves is obscured , not least because any embedding matrix can be arbitrarily scaled/rotated if the other is inversely transformed . 3 RELATIONSHIPS BETWEEN EMBEDDINGS OF RELATED WORDS . Our aim is to build on the understanding of PMI-based word embeddings ( henceforth word embeddings ) , to identify what a knowledge graph relation representation needs to achieve to map all subject word embeddings to all related object word embeddings . We note that if a semantic relation between two words implies a particular geometric relationship between their embeddings , then the latter serves as a necessary quantitative condition for the former to hold ( a relation condition ) . Relation conditions implicitly define a relation-specific mapping by which all subject embeddings are mapped to all related object embedding ( s ) , allowing related entities to be identified by a proximity measure ( e.g . Euclidean distance or dot product ) . Since this is the approach of many knowledge graph representation models , their performance can be contrasted with their ability to express mappings that satisfy required relation conditions . For example , similarity and context-shift relations respectively imply closeness and a relation-specific vector offset between embeddings ( S2.2 ) . Such relation conditions can be tested for by respectively making no change to the subject entity or adding the relation-specific offset , before measuring proximity with the object . We note that since relation conditions are not necessarily sufficient , they do not guarantee a relation holds , i.e . false positives may arise . In general , the data from which knowledge graph embeddings are derived differs greatly to the co-occurrence data used for word embeddings , and the latter would not be anticipated to be learned by knowledge graph models . However , word embeddings provide a known solution ( i.e . minimise the loss function ) of any knowledge graph model able to express the required mapping ( s ) derived from relation conditions , where the loss function measures proximity between mapped entities . The relation conditions for certain relation types ( underlined ) follow readily from S2.2 : • Similarity : Semantically similar words induce similar distributions over the words they co-occur with . Thus their PMI vectors ( Fig 1a ) and word embeddings are similar . • Relatedness : The relatedness of two words can be defined in terms of the words with which both co-occur similarly ( S∈D ) , which define the nature of relatedness , e.g . milk and cheese are related by S= { dairy , breakfast , ... } ; and |S| reflects the strength of relatedness . Since PMI vector components corresponding to S are similar ( Fig 1b ) , embeddings of “ S-related ” words have similar components in the subspace VS that spans the projected PMI vector dimensions corresponding to S. The rank of VS might be expected to reflect relatedness strength . In general , relatedness is a weaker , more variable relation than similarity , its limiting case with S=D and rank ( VS ) =d . • Context-shift : In the context of word embeddings , analogy typically refers to relational similarity ( Turney , 2006 ; Gladkova et al. , 2016 ) . More specifically , the relations within analogies that give a common vector offset between word embeddings require related words to have a common difference between their distributions of co-occurring words , defined as a context-shifts ( see S2.2 ) . These relations are strictly 1-to-1 and include an aspect of relatedness due to the word associations in common ( Fig 1d ) . A specialisation relation is a context-shift in which context is only added ( Fig 1c ) . • Generalised context-shift : Context-shift relations are generalised to 1-to-many , many-to-1 and many-to-many relations by letting the fully-specified added or subtracted context be one of a ( relationspecific ) context set ( Fig 1e ) , e.g . allowing an entity to be any colour or anything blue . The potential scope and size of each context set means these relations can vary greatly . The limiting case for small context sets has a single context in each , whereby the relation is an explicit context-shift ( as above ) , and the difference between embeddings is a known vector offset . In the limiting case where context sets are large , the added/subtracted context is so loosely defined that , in effect , only the relatedness aspect of the relation and thus only the common subspace component of embeddings is known . Link to set theory : Viewing PMI vectors as sets of word associations and taking intuition from Fig 1 , the above relations can be seen to reflect set operations : similarity as set equality ; relatedness as equality of a subset ; and context-shift as the set difference equalling a relation-specific set . This highlights how the relatedness aspect of a relation reflects features that must be common , and contextshift reflects features that must differ . Whilst this mirrors an intuitive notion of “ feature vectors ” , we emphasise that this is grounded in the co-occurrence statistics of PMI-based word embeddings .

Summary: This paper proposes to provide a detailed study on the explainability of link prediction (LP) models by utilizing a recent interpretation of word embeddings. More specifically, the authors categorize the relations in KG into three categories (R, S, C) using the correlation between the semantic relation between two words and the geometric relationship between their embeddings. The authors utilize this categorization to provide a better understanding of LP models’ performance through several experiments.