Interpreting Knowledge Graph Relation Representation from Word Embeddings

1 INTRODUCTION . Knowledge graphs are large repositories of binary relations between words ( or entities ) in the form of ( subject , relation , object ) triples . Many models for representing entities and relations have been developed , so that known facts can be recalled and previously unknown facts can be inferred , a task known as link prediction . Recent link prediction models ( e.g . Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Balažević et al. , 2019b ) learn entity representations , or embeddings , of far lower dimensionality than the number of entities , by capturing latent structure in the data . Relations are typically represented as a mapping from the embedding of a subject entity to those of related object entities . Although the performance of link prediction models has steadily improved for nearly a decade , relatively little is understood of the low-rank latent structure that underpins them , which we address in this work . The outcomes of our analysis can be used to aid and direct future knowledge graph model design . We start by drawing a parallel between the entity embeddings of knowledge graphs and context-free word embeddings , e.g . as learned by Word2Vec ( W2V ) ( Mikolov et al. , 2013a ) and GloVe ( Pennington et al. , 2014 ) . Our motivating premise is that the same latent word features ( e.g . meaning ( s ) , tense , grammatical type ) give rise to the patterns found in different data sources , i.e . manifesting in word cooccurrence statistics and determining which words relate to which . Different embedding approaches may capture such structure in different ways , but if it is fundamentally the same , an understanding gained from one embedding task ( e.g . word embedding ) may benefit another ( e.g . knowledge graph representation ) . Furthermore , the relatively limited but accurate data used in knowledge graph representation differs materially from the highly abundant but statistically noisy text data used for word embeddings . As such , theoretically reconciling the two embedding methods may lead to unified and improved embeddings learned jointly from both data sources . Recent work ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) theoretically explains how semantic properties are encoded in word embeddings that ( approximately ) factorise a matrix of pointwise mutual information ( PMI ) from word co-occurrence statistics , as known for W2V ( Levy & Goldberg , 2014 ) . Semantic relationships between words , specifically similarity , relatedness , paraphrase and analogy , are proven to manifest as linear geometric relationships between rows of the PMI matrix ( subject to known error terms ) , of which word embeddings can be considered low-rank projections . This explains , for example , the observations that similar words have similar embeddings and that embeddings of analogous word pairs share a common “ vector offset ” ( e.g . Mikolov et al. , 2013b ) . ∗Equal contribution We extend this insight to identify geometric relationships between PMI-based word embeddings that correspond to other relations , i.e . those of knowledge graphs . Such relation conditions define relation-specific mappings between entity embeddings ( i.e . relation representations ) and so provide a “ blue-print ” for knowledge graph representation models . Analysing the relation representations of leading knowledge graph representation models , we find that various properties , including their relative link prediction performance , accord with predictions based on these relation conditions , supporting the premise that a common latent structure is learned by word and knowledge graph embedding models , despite the significant differences between their training data and methodology . In summary , the key contributions of this work are : • to use recent understanding of PMI-based word embeddings to derive geometric attributes of a relation representation for it to map subject word embeddings to all related object word embeddings ( relation conditions ) , which partition relations into three types ( §3 ) ; • to show that both per-relation ranking as well as classification performance of leading link prediction models corresponds to the model satisfying the appropriate relation conditions , i.e . how closely its relation representations match the geometric form derived theoretically ( §4.1 ) ; and • to show that properties of knowledge graph representation models fit predictions based on relation conditions , e.g . the strength of a relation ’ s relatedness aspect is reflected in the eigenvalues of its relation matrix ( §4.2 ) . 2 BACKGROUND . Knowledge graph representation : Recent knowledge graph models typically represent entities es , eo as vectors es , eo ∈ Rde , and relations as transformations in the latent space from subject to object entity embedding , where the dimension de is far lower ( e.g . 200 ) than the number of entities ne ( e.g . > 104 ) . Such models are distinguished by their score function , which defines ( i ) the form of the relation transformation , e.g . matrix multiplication and/or vector addition ; and ( ii ) the measure of proximity between a transformed subject embedding and an object embedding , e.g . dot product or Euclidean distance . Score functions can be non-linear ( e.g . Dettmers et al. , 2018 ) , or linear and sub-categorised as additive , multiplicative or both . We focus on linear models due to their simplicity and strong performance at link prediction ( including state-of-the-art ) . Table 1 shows the score functions of competitive linear knowledge graph embedding models spanning the sub-categories : TransE ( Bordes et al. , 2013 ) , DistMult ( Yang et al. , 2015 ) , TuckER ( Balažević et al. , 2019b ) and MuRE ( Balažević et al. , 2019a ) . Additive models apply a relation-specific translation to a subject entity embedding and typically use Euclidean distance to evaluate proximity to object embeddings . A generic additive score function is given by φ ( es , r , eo ) =−‖es+r−eo‖22+bs+bo . A simple example is TransE , where bs=bo=0 . Multiplicative models have the generic score function φ ( es , r , eo ) =e > s Reo , i.e . a bilinear product of the entity embeddings and a relation-specific matrix R. DistMult is a simple example with R diagonal and so can not model asymmetric relations ( Trouillon et al. , 2016 ) . In TuckER , each relation-specific R=W×3 r is a linear combination of dr “ prototype ” relation matrices in a core tensor W∈Rde×dr×de ( ×n denoting tensor product along mode n ) , facilitating multi-task learning across relations . Some models , e.g . MuRE , combine both multiplicative ( R ) and additive ( r ) components . Word embedding : Algorithms such as Word2Vec ( Mikolov et al. , 2013a ) and GloVe ( Pennington et al. , 2014 ) generate low-dimensional word embeddings that perform well on downstream tasks ( Baroni et al. , 2014 ) . Such models predict the context words ( cj ) observed around a target word ( wi ) in a text corpus using shallow neural networks . Whilst recent language models ( e.g . Devlin et al. , 2018 ; Peters et al. , 2018 ) achieve strong performance using contextualised word embeddings , we focus on “ context-free ” embeddings since knowledge graph entities have no obvious context and , importantly , they offer insight into embedding interpretability . Levy & Goldberg ( 2014 ) show that , for a dictionary of ne unique words and embedding dimension de ne , W2V ’ s loss function is minimised when its embeddings wi , cj form matrices W , C ∈ Rde×ne that factorise a pointwise mutual information ( PMI ) matrix of word co-occurrence statistics ( PMI ( wi , cj ) =log P ( wi , cj ) P ( wi ) P ( cj ) ) , subject to a shift term . This result relates W2V to earlier count-based embeddings and specifically PMI , which has a history in linguistic analysis ( Turney & Pantel , 2010 ) . From its loss function , GloVe can be seen to perform a related factorisation . Recent work ( Allen & Hospedales , 2019 ; Allen et al. , 2019 ) shows how the semantic relationships of similarity , relatedness , paraphrase and analogy are encoded in PMI-based word embeddings by recognising such embeddings as low-rank projections of high dimensional rows of the PMI matrix , termed PMI vectors . Those semantic relationships are described in terms of multiplicative interactions between co-occurrence probabilities ( subject to defined error terms ) , that correspond to additive interactions between ( logarithmic ) PMI statistics , and hence PMI vectors . Thus , under a sufficiently linear projection , those semantic relationships correspond to linear relationships between word embeddings . Note that although the relative geometry reflecting semantic relationships is preserved , the direct interpretability of dimensions , as in PMI vectors , is lost since the embedding matrices can be arbitrarily scaled/rotated if the other is inversely transformed . We state the relevant semantic relationships on which we build , denoting the set of unique dictionary words by E : • Paraphrase : word subsetsW , W∗⊆E are said to paraphrase if they induce similar distributions over nearby words , i.e . p ( E|W ) ≈p ( E|W∗ ) , e.g . { king } paraphrases { man , royal } . • Analogy : a common example of an analogy is “ woman is to queen as man is to king ” and can be defined as any set of word pairs { ( wi , w∗i ) } i∈I for which it is semantically meaningful to say “ wa is to w∗a as wb is to w ∗ b ” ∀a , b∈I . Where one word subset paraphrases another , the sums of their embeddings are shown to be equal ( subject to the independence of words within each set ) , e.g . wking ≈ wman+wroyal . An interesting connection is established between the two semantic relationships : a set of word pairs A= { ( wa , w∗a ) , ( wb , w∗b ) } is an analogy if { wa , w∗b } paraphrases { w∗a , wb } , in which case the embeddings satisfy wa∗−wa ≈ wb∗−wb ( “ vector offset ” ) . 3 FROM ANALOGIES TO KNOWLEDGE GRAPH RELATIONS . Analogies from the field of word embeddings are our starting point for developing a theoretical basis for representing knowledge graph relations . The relevance of analogies stems from the observation that for an analogy to hold ( see §2 ) , its word pairs , e.g { ( man , king ) , ( woman , queen ) , ( girl , princess ) } , must be related in the same way , comparably to subject-object entity pairs under a common knowledge graph relation . Our aim is to develop the understanding of PMI-based word embeddings ( henceforth word embeddings ) , to identify the mathematical properties necessary for a relation representation to map subject word embeddings to all related object word embeddings . Considering the paraphrasing word sets { king } and { man , royal } corresponding to the word embedding relationship wking≈wman+wroyal ( §2 ) , royal can be interpreted as the semantic difference between man and king , fitting intuitively with the relationship wroyal≈wking−wman . Fundamentally , this relationship holds because the difference between words that co-occur ( i.e . occur more frequently than if independent ) with king and those that co-occur with man , reflects those words that co-occur with royal . We refer to this difference in co-occurrence distribution as a “ context shift ” , from man ( subject ) to king ( object ) . Allen & Hospedales ( 2019 ) effectively show that where multiple word pairs share a common context shift , they form an analogy whose embeddings satisfy the vector offset relationship . This result seems obvious where the context shift mirrors an identifiable word , the embedding of which is approximated by the common vector offset , e.g . queen and woman are related by the same context shift , i.e . wqueen ≈ wwoman+wroyal , thus wqueen−wwoman ≈ wking−wman . However , the same result holds , i.e . an analogy is formed with a common vector offset between embeddings , for an arbitrary ( common ) context shift that may reflect no particular word . Importantly , these context shift relations evidence a case in which it is known how a relation can be represented , i.e . by an additive vector ( comparable to TransE ) if entities are represented by word embeddings . More generally , this provides an interpretable foothold into relation representation . Note that not all sets of word pairs considered analogies exhibit a clear context shift relation , e.g . in the analogy { ( car , engine ) , ( bus , seats ) } , the difference between words co-occurring with engine and car is not expected to reflect the corresponding difference between bus and seats . This illustrates how analogies are a loosely defined concept , e.g . their implicit relation may be semantic or syntactic , with several sub-categories of each ( e.g . see Gladkova et al . ( 2016 ) ) . The same is readily observed for the relations of knowledge graphs . This likely explains the observed variability in “ solving ” analogies by use of vector offset ( e.g . Köper et al. , 2015 ; Karpinska et al. , 2018 ; Gladkova et al. , 2016 ) and suggests that further consideration is required to represent relations ( or solve analogies ) in general . We have seen that the existence of a context shift relation between a subject and object word implies a ( relation-specific ) geometric relationship between word embeddings , thus the latter provides a necessary condition for the relation to hold . We refer to this as a “ relation condition ” and aim to identify relation conditions for other classes of relation . Once identified , relation conditions define a mapping from subject embeddings to all related object embeddings , by which related entities might be identified with a proximity measure ( e.g . Euclidean distance or dot product ) . This is the precise aim of a knowledge graph representation model , but loss functions are typically developed heuristically . Given the existence of many representation models , we can verify identified relation conditions by contrasting the per-relation performance of various models with the extent to which their loss function reflects the appropriate relation conditions . Note that since relation conditions are necessary rather than sufficient , they do not guarantee a relation holds , i.e . false positives may arise . Whilst we seek to establish relation conditions based on PMI word embeddings , the data used to train knowledge graph embeddings differs significantly to the text data used by word embeddings , and the relevance of conditions ultimately based on PMI statistics may seem questionable . However , where a knowledge graph representation model implements relation conditions and measures proximity between embeddings , the parameters of word embeddings necessarily provide a potential solution that minimises the loss function . Many equivalent solutions may exist due to symmetry as typical for neural network architectures . We now define relation types and identify their relation conditions ( underlined ) ; we then consider the completeness of this categorisation . • Similarity : Semantically similar words induce similar distributions over the words they co-occur with . Thus their PMI vectors and word embeddings are similar ( Fig 1a ) . • Relatedness : The relatedness of two words can be considered in terms of the words S ⊆E with which both co-occur similarly . S defines the nature of relatedness , e.g . milk and cheese are related by S= { dairy , breakfast , ... } ; and |S| reflects the strength of relatedness . Since PMI vector components corresponding to S are similar ( Fig 1b ) , embeddings of S-related words have similar components in the subspace VS that spans the projected PMI vector dimensions corresponding to S. The rank of VS is thus anticipated to reflect relatedness strength . Relatedness can be seen as a weaker and more variable generalisation of similarity , its limiting case where S=E , hence rank ( VS ) =de . • Context-shift : As discussed above , words related by a common difference between their distributions of co-occurring words , defined as context-shifts , share a common vector offset between word embeddings . Context might be considered added ( e.g . man to king ) , termed a specialisation ( Fig 1c ) , subtracted ( e.g . king to man ) or both ( Fig 1d ) . These relations are 1-to-1 ( subject to synonyms ) and include an aspect of relatedness due to the word associations in common . Note that , specialisations include hyponyms/hypernyms and context shifts include meronyms . • Generalised context-shift : Context-shift relations generalise to 1-to-many , many-to-1 and manyto-many relations where the added/subtracted context may be from a ( relation-specific ) context set ( Fig 1e ) , e.g . any city or anything bigger . The potential scope and size of context sets adds variability to these relations . The limiting case in which the context set is “ small ” reduces to a 1-to-1 context-shift ( above ) and the embedding difference is a known vector offset . In the limiting case of a “ large ” context set , the added/subtracted context is essentially unrestricted such that only the relatedness aspect of the relation , and thus a common subspace component of embeddings , is fixed . Categorisation completeness : Taking intuition from Fig 1 and considering PMI vectors as sets of word features , these relation types can be interpreted as set operations : similarity as set equality ; relatedness as subset equality ; and context-shift as a relation-specific set difference . Since for any relation each feature must either remain unchanged ( relatedness ) , change ( context shift ) or else be irrelevant , we conjecture that the above relation types give a complete partition of semantic relations .

Summary: Based on PMI word embedding, the authors categorize the knowledge graph relations into three types, which serve as the foundation of knowledge analysis. This paper is not well-motived but presents the methodology, well. However, nothing in this paper surprised me, because this seems like a ````''regular'' research in this field.