Interpreting Knowledge Graph Relation Representation from Word Embeddings

1 INTRODUCTION . Knowledge graphs are large repositories of binary relations between words ( or entities ) in the form of ( subject , relation , object ) triples . Many models for representing entities and relations have been developed , so that known facts can be recalled and previously unknown facts can be inferred , a task known as link prediction . Recent link prediction models ( e.g . Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Balažević et al. , 2019b ) learn entity representations , or embeddings , of far lower dimensionality than the number of entities , by capturing latent structure in the data . Relations are typically represented as a mapping from the embedding of a subject entity to those of related object entities . Although the performance of link prediction models has steadily improved for nearly a decade , relatively little is understood of the low-rank latent structure that underpins them , which we address in this work . The outcomes of our analysis can be used to aid and direct future knowledge graph model design . We start by drawing a parallel between the entity embeddings of knowledge graphs and context-free word embeddings , e.g . as learned by Word2Vec ( W2V ) ( Mikolov et al. , 2013a ) and GloVe ( Pennington et al. , 2014 ) . Our motivating premise is that the same latent word features ( e.g . meaning ( s ) , tense , grammatical type ) give rise to the patterns found in different data sources , i.e . manifesting in word cooccurrence statistics and determining which words relate to which . Different embedding approaches may capture such structure in different ways , but if it is fundamentally the same , an understanding gained from one embedding task ( e.g . word embedding ) may benefit another ( e.g . knowledge graph representation ) . Furthermore , the relatively limited but accurate data used in knowledge graph representation differs materially from the highly abundant but statistically noisy text data used for word embeddings . As such , theoretically reconciling the two embedding methods may lead to unified and improved embeddings learned jointly from both data sources . Recent work ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) theoretically explains how semantic properties are encoded in word embeddings that ( approximately ) factorise a matrix of pointwise mutual information ( PMI ) from word co-occurrence statistics , as known for W2V ( Levy & Goldberg , 2014 ) . Semantic relationships between words , specifically similarity , relatedness , paraphrase and analogy , are proven to manifest as linear geometric relationships between rows of the PMI matrix ( subject to known error terms ) , of which word embeddings can be considered low-rank projections . This explains , for example , the observations that similar words have similar embeddings and that embeddings of analogous word pairs share a common “ vector offset ” ( e.g . Mikolov et al. , 2013b ) . ∗Equal contribution We extend this insight to identify geometric relationships between PMI-based word embeddings that correspond to other relations , i.e . those of knowledge graphs . Such relation conditions define relation-specific mappings between entity embeddings ( i.e . relation representations ) and so provide a “ blue-print ” for knowledge graph representation models . Analysing the relation representations of leading knowledge graph representation models , we find that various properties , including their relative link prediction performance , accord with predictions based on these relation conditions , supporting the premise that a common latent structure is learned by word and knowledge graph embedding models , despite the significant differences between their training data and methodology . In summary , the key contributions of this work are : • to use recent understanding of PMI-based word embeddings to derive geometric attributes of a relation representation for it to map subject word embeddings to all related object word embeddings ( relation conditions ) , which partition relations into three types ( §3 ) ; • to show that both per-relation ranking as well as classification performance of leading link prediction models corresponds to the model satisfying the appropriate relation conditions , i.e . how closely its relation representations match the geometric form derived theoretically ( §4.1 ) ; and • to show that properties of knowledge graph representation models fit predictions based on relation conditions , e.g . the strength of a relation ’ s relatedness aspect is reflected in the eigenvalues of its relation matrix ( §4.2 ) . 2 BACKGROUND . Knowledge graph representation : Recent knowledge graph models typically represent entities es , eo as vectors es , eo ∈ Rde , and relations as transformations in the latent space from subject to object entity embedding , where the dimension de is far lower ( e.g . 200 ) than the number of entities ne ( e.g . > 104 ) . Such models are distinguished by their score function , which defines ( i ) the form of the relation transformation , e.g . matrix multiplication and/or vector addition ; and ( ii ) the measure of proximity between a transformed subject embedding and an object embedding , e.g . dot product or Euclidean distance . Score functions can be non-linear ( e.g . Dettmers et al. , 2018 ) , or linear and sub-categorised as additive , multiplicative or both . We focus on linear models due to their simplicity and strong performance at link prediction ( including state-of-the-art ) . Table 1 shows the score functions of competitive linear knowledge graph embedding models spanning the sub-categories : TransE ( Bordes et al. , 2013 ) , DistMult ( Yang et al. , 2015 ) , TuckER ( Balažević et al. , 2019b ) and MuRE ( Balažević et al. , 2019a ) . Additive models apply a relation-specific translation to a subject entity embedding and typically use Euclidean distance to evaluate proximity to object embeddings . A generic additive score function is given by φ ( es , r , eo ) =−‖es+r−eo‖22+bs+bo . A simple example is TransE , where bs=bo=0 . Multiplicative models have the generic score function φ ( es , r , eo ) =e > s Reo , i.e . a bilinear product of the entity embeddings and a relation-specific matrix R. DistMult is a simple example with R diagonal and so can not model asymmetric relations ( Trouillon et al. , 2016 ) . In TuckER , each relation-specific R=W×3 r is a linear combination of dr “ prototype ” relation matrices in a core tensor W∈Rde×dr×de ( ×n denoting tensor product along mode n ) , facilitating multi-task learning across relations . Some models , e.g . MuRE , combine both multiplicative ( R ) and additive ( r ) components . Word embedding : Algorithms such as Word2Vec ( Mikolov et al. , 2013a ) and GloVe ( Pennington et al. , 2014 ) generate low-dimensional word embeddings that perform well on downstream tasks ( Baroni et al. , 2014 ) . Such models predict the context words ( cj ) observed around a target word ( wi ) in a text corpus using shallow neural networks . Whilst recent language models ( e.g . Devlin et al. , 2018 ; Peters et al. , 2018 ) achieve strong performance using contextualised word embeddings , we focus on “ context-free ” embeddings since knowledge graph entities have no obvious context and , importantly , they offer insight into embedding interpretability . Levy & Goldberg ( 2014 ) show that , for a dictionary of ne unique words and embedding dimension de ne , W2V ’ s loss function is minimised when its embeddings wi , cj form matrices W , C ∈ Rde×ne that factorise a pointwise mutual information ( PMI ) matrix of word co-occurrence statistics ( PMI ( wi , cj ) =log P ( wi , cj ) P ( wi ) P ( cj ) ) , subject to a shift term . This result relates W2V to earlier count-based embeddings and specifically PMI , which has a history in linguistic analysis ( Turney & Pantel , 2010 ) . From its loss function , GloVe can be seen to perform a related factorisation . Recent work ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) shows how the semantic relationships of similarity , relatedness , paraphrase and analogy are encoded in PMI-based word embeddings by recognising such embeddings as low-rank projections of high dimensional rows of the PMI matrix , termed PMI vectors . Those semantic relationships are described in terms of multiplicative interactions between co-occurrence probabilities ( subject to defined error terms ) , that correspond to additive interactions between ( logarithmic ) PMI statistics , and hence PMI vectors . Thus , under a sufficiently linear projection , those semantic relationships correspond to linear relationships between word embeddings . Note that although the relative geometry reflecting semantic relationships is preserved , the direct interpretability of dimensions , as in PMI vectors , is lost since the embedding matrices can be arbitrarily scaled/rotated if the other is inversely transformed . We state the relevant semantic relationships on which we build , denoting the set of unique dictionary words by E : • Paraphrase : word subsetsW , W∗⊆E are said to paraphrase if they induce similar distributions over nearby words , i.e . p ( E|W ) ≈p ( E|W∗ ) , e.g . { king } paraphrases { man , royal } . • Analogy : a common example of an analogy is “ woman is to queen as man is to king ” and can be defined as any set of word pairs { ( wi , w∗i ) } i∈I for which it is semantically meaningful to say “ wa is to w∗a as wb is to w ∗ b ” ∀a , b∈I . Where one word subset paraphrases another , the sums of their embeddings are shown to be equal ( subject to the independence of words within each set ) , e.g . wking ≈ wman+wroyal . An interesting connection is established between the two semantic relationships : a set of word pairs A= { ( wa , w∗a ) , ( wb , w∗b ) } is an analogy if { wa , w∗b } paraphrases { w∗a , wb } , in which case the embeddings satisfy wa∗−wa ≈ wb∗−wb ( “ vector offset ” ) . 3 FROM ANALOGIES TO KNOWLEDGE GRAPH RELATIONS . Analogies from the field of word embeddings are our starting point for developing a theoretical basis for representing knowledge graph relations . The relevance of analogies stems from the observation that for an analogy to hold ( see §2 ) , its word pairs , e.g { ( man , king ) , ( woman , queen ) , ( girl , princess ) } , must be related in the same way , comparably to subject-object entity pairs under a common knowledge graph relation . Our aim is to develop the understanding of PMI-based word embeddings ( henceforth word embeddings ) , to identify the mathematical properties necessary for a relation representation to map subject word embeddings to all related object word embeddings . Considering the paraphrasing word sets { king } and { man , royal } corresponding to the word embedding relationship wking≈wman+wroyal ( §2 ) , royal can be interpreted as the semantic difference between man and king , fitting intuitively with the relationship wroyal≈wking−wman . Fundamentally , this relationship holds because the difference between words that co-occur ( i.e . occur more frequently than if independent ) with king and those that co-occur with man , reflects those words that co-occur with royal . We refer to this difference in co-occurrence distribution as a “ context shift ” , from man ( subject ) to king ( object ) . Allen & Hospedales ( 2019 ) effectively show that where multiple word pairs share a common context shift , they form an analogy whose embeddings satisfy the vector offset relationship . This result seems obvious where the context shift mirrors an identifiable word , the embedding of which is approximated by the common vector offset , e.g . queen and woman are related by the same context shift , i.e . wqueen ≈ wwoman+wroyal , thus wqueen−wwoman ≈ wking−wman . However , the same result holds , i.e . an analogy is formed with a common vector offset between embeddings , for an arbitrary ( common ) context shift that may reflect no particular word . Importantly , these context shift relations evidence a case in which it is known how a relation can be represented , i.e . by an additive vector ( comparable to TransE ) if entities are represented by word embeddings . More generally , this provides an interpretable foothold into relation representation . Note that not all sets of word pairs considered analogies exhibit a clear context shift relation , e.g . in the analogy { ( car , engine ) , ( bus , seats ) } , the difference between words co-occurring with engine and car is not expected to reflect the corresponding difference between bus and seats . This illustrates how analogies are a loosely defined concept , e.g . their implicit relation may be semantic or syntactic , with several sub-categories of each ( e.g . see Gladkova et al . ( 2016 ) ) . The same is readily observed for the relations of knowledge graphs . This likely explains the observed variability in “ solving ” analogies by use of vector offset ( e.g . Köper et al. , 2015 ; Karpinska et al. , 2018 ; Gladkova et al. , 2016 ) and suggests that further consideration is required to represent relations ( or solve analogies ) in general . We have seen that the existence of a context shift relation between a subject and object word implies a ( relation-specific ) geometric relationship between word embeddings , thus the latter provides a necessary condition for the relation to hold . We refer to this as a “ relation condition ” and aim to identify relation conditions for other classes of relation . Once identified , relation conditions define a mapping from subject embeddings to all related object embeddings , by which related entities might be identified with a proximity measure ( e.g . Euclidean distance or dot product ) . This is the precise aim of a knowledge graph representation model , but loss functions are typically developed heuristically . Given the existence of many representation models , we can verify identified relation conditions by contrasting the per-relation performance of various models with the extent to which their loss function reflects the appropriate relation conditions . Note that since relation conditions are necessary rather than sufficient , they do not guarantee a relation holds , i.e . false positives may arise . Whilst we seek to establish relation conditions based on PMI word embeddings , the data used to train knowledge graph embeddings differs significantly to the text data used by word embeddings , and the relevance of conditions ultimately based on PMI statistics may seem questionable . However , where a knowledge graph representation model implements relation conditions and measures proximity between embeddings , the parameters of word embeddings necessarily provide a potential solution that minimises the loss function . Many equivalent solutions may exist due to symmetry as typical for neural network architectures . We now define relation types and identify their relation conditions ( underlined ) ; we then consider the completeness of this categorisation . • Similarity : Semantically similar words induce similar distributions over the words they co-occur with . Thus their PMI vectors and word embeddings are similar ( Fig 1a ) . • Relatedness : The relatedness of two words can be considered in terms of the words S ⊆E with which both co-occur similarly . S defines the nature of relatedness , e.g . milk and cheese are related by S= { dairy , breakfast , ... } ; and |S| reflects the strength of relatedness . Since PMI vector components corresponding to S are similar ( Fig 1b ) , embeddings of S-related words have similar components in the subspace VS that spans the projected PMI vector dimensions corresponding to S. The rank of VS is thus anticipated to reflect relatedness strength . Relatedness can be seen as a weaker and more variable generalisation of similarity , its limiting case where S=E , hence rank ( VS ) =de . • Context-shift : As discussed above , words related by a common difference between their distributions of co-occurring words , defined as context-shifts , share a common vector offset between word embeddings . Context might be considered added ( e.g . man to king ) , termed a specialisation ( Fig 1c ) , subtracted ( e.g . king to man ) or both ( Fig 1d ) . These relations are 1-to-1 ( subject to synonyms ) and include an aspect of relatedness due to the word associations in common . Note that , specialisations include hyponyms/hypernyms and context shifts include meronyms . • Generalised context-shift : Context-shift relations generalise to 1-to-many , many-to-1 and manyto-many relations where the added/subtracted context may be from a ( relation-specific ) context set ( Fig 1e ) , e.g . any city or anything bigger . The potential scope and size of context sets adds variability to these relations . The limiting case in which the context set is “ small ” reduces to a 1-to-1 context-shift ( above ) and the embedding difference is a known vector offset . In the limiting case of a “ large ” context set , the added/subtracted context is essentially unrestricted such that only the relatedness aspect of the relation , and thus a common subspace component of embeddings , is fixed . Categorisation completeness : Taking intuition from Fig 1 and considering PMI vectors as sets of word features , these relation types can be interpreted as set operations : similarity as set equality ; relatedness as subset equality ; and context-shift as a relation-specific set difference . Since for any relation each feature must either remain unchanged ( relatedness ) , change ( context shift ) or else be irrelevant , we conjecture that the above relation types give a complete partition of semantic relations .

Summary: Recent works toward the understanding of word embeddings can explain how semantic word relationships, such as similarity, analogy and paraphrasing are encoded as low-rank projections of high dimensional vectors of co-occurrence statistics (Allen et al., 2019). Thus, the semantic relationships correspond to linear relationships of word embeddings. This paper builds on this understanding of (PMI-based) word embeddings aiming at the task of understanding the latent structure of low-rank knowledge graph representations. The authors draw a parallel between the embeddings of knowledge graphs and words under the premise that fundamentally the same structure of relations is captured in different ways. Strong evidence to this premise is provided by starting at encoded semantic relations of word embeddings generalizing them to three types (R,S,C) of knowledge graph relations. The authors analyse the performance of different state-of-the-art knowledge graph models and identify the best performing model per relation type. While a multiplicative model performs best for R-relations (highly related), an additive-multiplicative model should be used for S- (specialisation) or C-type (context-shift) relations. These results correspond to the predictions made beforehand and the theoretically derived loss functions based on the respective conditions of each relation type.