Year: 2019

Abstract:
Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred. To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping. Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained. We build on recent theoretical interpretation of word embeddings to derive an explicit structure for the representations of relations between entities. From this we explain properties and justify the relative performance of leading knowledge graph representation methods for identifiable relation types, including their often overlooked ability to make independent predictions.

heading: 1 INTRODUCTION
text: Knowledge graphs are large repositories of binary relations between words (or entities) in the form of fact triples (subject, relation, object). Many models have been developed for learning representations of entities and relations in knowledge graphs, such that known facts can be recalled and previously unknown facts can be inferred, a task known as link prediction. Recent link prediction models (e.g. Bordes et al., 2013; Trouillon et al., 2016; Balažević et al., 2019b) learn entity representations, or embeddings, of far lower dimensionality than the number of entities, by capturing latent structure in the data. Relations are typically represented as a mapping from the embedding of a subject entity to its related object entity embedding(s). Although the performance of knowledge graphlink prediction models has steadily improved for nearly a decade, relatively little is understood of the low-rank latent structure that underpins these models, which we address in this work.
We start by drawing a parallel between entity embeddings in knowledge graphs and unsupervised word embeddings, as learned by algorithms such as Word2Vec (W2V) (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). We assume that words have latent features, e.g. meaning(s), tense, grammatical type, that are innate and fixed, irrespective of what an embedding may capture (which may be only a part, subject to the embedding method and/or the data source); and that this same latent structure gives rise to patterns observed in the data, e.g. in word co-occurrence statistics and in which words are related to which. As such, an understanding of the latent structure from one embedding task (e.g. word embedding) might be useful to another (e.g. knowledge graph entity embedding).
Recent work (Allen & Hospedales, 2019; Allen et al., 2019) theoretically explains how semantic properties are encoded in word embeddings that (approximately) factorise a matrix of word cooccurrence pointwise mutual information (PMI), e.g. as is known for W2V (Levy & Goldberg, 2014). Semantic relationships between words (specifically similarity, relatedness, paraphrase and analogy) are proven to manifest as linear relationships between rows of the PMI matrix (subject to known error terms), of which word embeddings can be considered low-rank projections. This explains why similar words (e.g. synonyms) have similar embeddings; and embeddings of analogous word pairs share a common “vector offset”.
Importantly, this insight allows us to identify geometric relationships between such word embeddings necessary for other semantic relations to hold, such as those of knowledge graphs. These relation conditions describe relation-specific mappings between entity embeddings, i.e. relation representations, providing a “blue-print” against which to consider knowledge graph representation models. We find that various properties of knowledge graph representation models, including the relative
performance of leading link prediction models, accord with predictions based on these relation conditions, suggesting a commonality to the latent structure learned in word embedding models and knowledge graph representation models, despite the significant differences between their training data and methodology. In summary, the key contributions of this work are:
• to use recent understanding of PMI-based word embeddings to derive what a relation representation must achieve to map a subject word embedding to all related object word embeddings (relation conditions), based on which relations can be categorised into three types; • to show that properties of knowledge graph models fit predictions made from relation conditions,
e.g. strength of a relation’s relatedness aspect is reflected in the eigenvalues of its relation matrix; • to show that the performance per relation of leading link prediction models corresponds to
the ability of the model’s architecture to meet the relation conditions of the relation’s type, i.e. the better the architecture of a knowledge graph representation model aligns with the form theoretically derived for PMI-based word embeddings, the better the model performs; and • noting how ranking metrics can be flawed, to provide novel insight into the prediction accuracy
per relation of recent knowledge graph models, an evaluation metric we recommend in future.

heading: 2 BACKGROUND
text: Our work draws on knowledge graph representation and word embedding. Whilst related, these tasks differ materially in their training data. The former is restricted to datasets crafted by hand or automatically generated, the latter has the vast abundance of natural language text (e.g. Wikipedia).

heading: 2.1 KNOWLEDGE GRAPH REPRESENTATION
text: Almost all recent knowledge graph models represent entities es, eo as vectors es, eo ∈ Rde of low dimension (e.g. de=200) relative to the number of entities ne (typically of order 104), and relations as transformations in the latent space from subject entity embedding to object. These models are distinguished by their score function, which defines (i) the form of the relation transformation, e.g. matrix multiplication, vector addition; and (ii) how “closeness” between the transformed subject embedding and an object embedding is evaluated, e.g. dot product, Euclidean distance. Score functions can be non-linear (e.g. Dettmers et al. (2018)), or linear and sub-categorised as additive, multiplicative or both. We focus on linear models due to their simplicity and strong performance at link prediction (including state-of-the-art). Table 1 shows the score functions of competitive linear models that span all linear sub-categories: TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), TuckER (Balažević et al., 2019b) and MuRE (Balažević et al., 2019a).
Additive models typically use Euclidean distance and contain a relation-specific translation from a (possibly transformed) subject to a (possibly transformed) object entity embedding. A generic additive score function is given by φ(es, r, eo)=−‖Rses+r−Roeo‖22+bs+bo=−‖e (r) s +r−e(r)o ‖22+bs+bo. The simplest example is TransE for which Rs=Ro=I and bs=bo=0. The score function of MuRE has Ro=I and so combines multiplicative (Rs=R) and additive (r) components.
Multiplicative models have the generic score function φ(es, r, eo) = e>s Reo = 〈e (r) s , eo〉, i.e. a bilinear product of the entity embeddings and a relation-specific matrix R. DistMult is a simple example with diagonal R and so cannot model asymmetric relations (Trouillon et al., 2016). In TuckER, each relation-specific R = W×3 r is a linear combination of dr “prototype” relation matrices in a core tensor W∈Rde×dr×de (where ×n denotes tensor product along mode n), facilitating multi-task learning across relations.

heading: 2.2 WORD EMBEDDING
text: Algorithms such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) generate succinct low-rank word embeddings that perform well on downstream tasks (Baroni et al., 2014). Such models predict the context words (cj) observed around each target word (wi) in a text corpus using shallow neural networks. Whilst recent language models (e.g. Devlin et al. (2018); Peters et al. (2018)) create impressive context-specific word embeddings, we focus on the former embeddings since knowledge graph entities have no obvious context and, more importantly, they are interpretable.
Levy & Goldberg (2014) show that, for a dictionary of unique words D and embedding dimension d |D|, W2V’s loss function is minimised when its weight matrices W,C∈Rd×|D| (whose columns are word embeddings wi, cj) factorise a word co-occurrence pointwise mutual information (PMI) matrix, subject to a shift term (PMI(wi, cj)=log
P (wi,cj) P (wi)P (cj) ). This connects W2V to earlier countbased embeddings and specifically to PMI, which has a long history in linguistic analysis (Turney & Pantel, 2010). From its loss function, GloVe can be seen to perform a comparable factorisation.
Recent work shows why word embeddings that factorise such PMI matrix encode semantic word relationships (Allen & Hospedales, 2019; Allen et al., 2019). The authors show that word embeddings can be seen as low-rank projections of high dimensional PMI vectors (rows of the PMI matrix), between which the semantic relationships of similarity, relatedness, paraphrase and analogy provably manifest as linear geometric relationships (subject to defined error terms), which are then preserved, under a sufficiently linear projection, between word embeddings. Thus similar words have similar embeddings, and the embeddings of analogous word pairs share a common vector offset.
Specifically, the PMI vectors (px) of an analogy “man is to king as woman is to queen” satisfy pQ−pW≈pK−pM because the difference between words associated with king and man (e.g. reign, crown) mirrors that between queen and woman. This leads to a common difference between their co-occurrence distributions (over all words), giving a common difference between their PMI vectors, which projects to a common difference between embeddings. Any discrepancy in the mirroring of word associations is shown to introduce error, weakening the analogy, as does a lack of statistical independence within certain word pairs (see (Allen & Hospedales, 2019)). The common difference in word co-occurrence distributions, e.g. the increased association with words {reign, crown, etc.}, can be interpreted semantically as a common change in context (context-shift) that transforms man to king and woman to queen by adding a royal context. Under this interpretation, context can also be subtracted, e.g. “king is to man as queen is to woman” (minus royal); or both, e.g. “boy is to king as girl is to queen” (minus youth plus royal). Adding context can also be interpreted as specialisation, and subtracting context as generalisation. This establishes a correspondence between common word embedding vector offsets and semantic context-shifts.
Although the projection from PMI vectors to word embeddings preserves the relative relationships, and thus the above semantic interpretability of common embedding differences, a direct interpretation of dimensions themselves is obscured, not least because any embedding matrix can be arbitrarily scaled/rotated if the other is inversely transformed.

heading: 3 RELATIONSHIPS BETWEEN EMBEDDINGS OF RELATED WORDS
text: Our aim is to build on the understanding of PMI-based word embeddings (henceforth word embeddings), to identify what a knowledge graph relation representation needs to achieve to map all subject word embeddings to all related object word embeddings. We note that if a semantic relation between two words implies a particular geometric relationship between their embeddings, then the latter serves as a necessary quantitative condition for the former to hold (a relation condition). Relation conditions implicitly define a relation-specific mapping by which all subject embeddings are mapped to all related object embedding(s), allowing related entities to be identified by a proximity measure (e.g. Euclidean distance or dot product). Since this is the approach of many knowledge graph representation models, their performance can be contrasted with their ability to express mappings that satisfy required relation conditions. For example, similarity and context-shift relations respectively imply closeness and a relation-specific vector offset between embeddings (S2.2). Such relation conditions can be tested for by respectively making no change to the subject entity or adding the relation-specific offset, before measuring proximity with the object. We note that since relation conditions are not necessarily sufficient, they do not guarantee a relation holds, i.e. false positives may arise.
In general, the data from which knowledge graph embeddings are derived differs greatly to the co-occurrence data used for word embeddings, and the latter would not be anticipated to be learned by knowledge graph models. However, word embeddings provide a known solution (i.e. minimise the loss function) of any knowledge graph model able to express the required mapping(s) derived from relation conditions, where the loss function measures proximity between mapped entities.
The relation conditions for certain relation types (underlined) follow readily from S2.2: • Similarity: Semantically similar words induce similar distributions over the words they co-occur with. Thus their PMI vectors (Fig 1a) and word embeddings are similar. • Relatedness: The relatedness of two words can be defined in terms of the words with which both co-occur similarly (S∈D), which define the nature of relatedness, e.g. milk and cheese are related by S={dairy, breakfast, ...}; and |S| reflects the strength of relatedness. Since PMI vector components corresponding to S are similar (Fig 1b), embeddings of “S-related” words have similar components in the subspace VS that spans the projected PMI vector dimensions corresponding to S. The rank of VS might be expected to reflect relatedness strength. In general, relatedness is a weaker, more variable relation than similarity, its limiting case with S=D and rank(VS)=d. • Context-shift: In the context of word embeddings, analogy typically refers to relational similarity (Turney, 2006; Gladkova et al., 2016). More specifically, the relations within analogies that give a common vector offset between word embeddings require related words to have a common difference between their distributions of co-occurring words, defined as a context-shifts (see S2.2). These relations are strictly 1-to-1 and include an aspect of relatedness due to the word associations in common (Fig 1d). A specialisation relation is a context-shift in which context is only added (Fig 1c). • Generalised context-shift: Context-shift relations are generalised to 1-to-many, many-to-1 and many-to-many relations by letting the fully-specified added or subtracted context be one of a (relationspecific) context set (Fig 1e), e.g. allowing an entity to be any colour or anything blue. The potential scope and size of each context set means these relations can vary greatly. The limiting case for small context sets has a single context in each, whereby the relation is an explicit context-shift (as above), and the difference between embeddings is a known vector offset. In the limiting case where context sets are large, the added/subtracted context is so loosely defined that, in effect, only the relatedness aspect of the relation and thus only the common subspace component of embeddings is known. Link to set theory: Viewing PMI vectors as sets of word associations and taking intuition from Fig 1, the above relations can be seen to reflect set operations: similarity as set equality; relatedness as equality of a subset; and context-shift as the set difference equalling a relation-specific set. This highlights how the relatedness aspect of a relation reflects features that must be common, and contextshift reflects features that must differ. Whilst this mirrors an intuitive notion of “feature vectors”, we emphasise that this is grounded in the co-occurrence statistics of PMI-based word embeddings.

heading: 3.1 CATEGORISING KNOWLEDGE GRAPH RELATIONS
text: Analysing the relations of popular knowledge graph datasets with the above perspective, we find that they comprise (i) a relatedness aspect that reflects a common theme (e.g. both entities are animals or geographic terms); and (ii) specific word associations of the subject and/or object entities. Specifically, relations appear to fall under a hierarchy of three relation types: highly related (R); (generalised) specialisation (S); and (generalised) context-shift (C). As above, “generalised” indicates that any added/subtracted contexts can be from a set. From Fig 1, type R relations can be seen as a special case of S, which, in turn, is a special case of C. Type C is therefore a generalised case of all considered relations. Whilst there are several other ways to classify relations (e.g. by their hierarchy,
transitivity), by considering relation conditions, we delineate by the required mathematical form (and complexity) of their representation. Table 2 shows a categorisation of the relations of the WN18RR dataset (Dettmers et al., 2018), containing 11 relations between 40,943 entities.1 An explanation for this assignment is in Appx. A and that for NELL-995 (Xiong et al., 2017) is in Appx. B. A review of the FB15k-237 dataset (Toutanova et al., 2015) shows the vast majority of relations to be of type C preventing a contrast between relation types being drawn, hence we do not consider that dataset.

heading: 3.2 RELATIONS AS MAPPINGS OF EMBEDDINGS
text: Given the relation conditions of a particular relation type, we can recognise mappings that meet them and thus loss functions (that evaluate the proximity of mapped entity embeddings by dot product or Euclidean distance) able to identify relations of that type between PMI-based word embeddings. We then contrast these theoretically inspired loss functions (one per relation type) with those of knowledge graph models (Table 1) and, on the outline assumption that a common low-rank latent structure is exploited by both word embeddings and knowledge graph models, predict properties and the relative performance of different knowledge graphs models for different relation types. R: To evidence S-relatedness, both entity embeddings es, eo must be projected onto a subspace VS, where their images are compared. Projection requires multiplication by a matrix Pr ∈Rd×d and cannot be achieved additively, except in the limiting case of similarity, when Pr=I or vector r ≈ 0 is added. Comparison by dot product gives (Pres)>(Preo) = e>s P>r Preo = e>s Mreo (for a relation-specific symmetric Mr = P>r Pr). Euclidean distance gives ‖Pres−Preo‖2 = (es−eo)>Mr(es−eo) = ‖Pres‖2 − 2e>s Mreo + ‖Preo‖2. S/C: Evidencing these relations requires a test both for S-relatedness and for relation-entity-specific embeddings component(s) (vsr , v o r ). This can be achieved by (i) multiplying both entity embeddings by a relation-specific projection matrix Pr that projects onto the subspace that spans the low-rank projection of dimensions corresponding to S, vsr and vor , (which tests for S-relatedness whilst preserving any entity-specific embedding components); and (ii) adding a relation-specific vector r = vor − vsr to the transformed subject entity embeddings. Comparison of the final transformed entity embeddings by dot product equates to (Pres + r)>Preo; and by Euclidean distance to ‖Pres+r−Preo‖2=‖Pres+r‖2−2(Pres+r)>Preo+‖Preo‖2 (cf MuRE: ‖Res+r−eo‖2). Contrasting the above loss functions with those of knowledge graph models (Table 1), we make the following predictions: (P1) the ability to learn the representation of a particular relation is expected to reflect the complexity of its type (R>S>C), and whether all relation conditions (e.g. additive or multiplicative interactions) can be met under a given model; (P2) relation matrices for relatedness (type R) relations are highly symmetric; (P3) offset vectors for relatedness relations have low norm; and (P4) as a proxy to the rank of VS, the eigenvalues of a relation matrix reflect a relation’s strength of relatedness. To elaborate: P1 anticipates that additive-only models (e.g. TransE) are not suited to identifying the relatedness aspect of relations (except in limiting cases of similarity, requiring a zero vector); and multiplicative-only models (e.g. DistMult) should perform well on type R but are not suited to identifying entity-specific features of type S/C, for which an asymmetric relation matrix in TuckER may help compensate. Further, the loss function of MuRE closely resembles that derived for type C relations (which generalise all others) and is thus expected to perform best overall.
1We omit the relation “similar_to” since its instances have no discernible structure, and only 3 occur in the test set, all of which are the inverse of a training example and trivial to predict.

heading: 4 COMPARING KNOWLEDGE GRAPH MODELS
text: We test the predictions made on the basis of word embeddings by comparing the performance of competitive knowledge graph models, TransE, DistMult, TuckER and MuRE (see S2), which entail different forms of relation representation, on all WN18RR relations and a similar number of NELL995 relations (selected to represent each relation type). Since applying the logistic sigmoid to the score function of TransE does not give a probabilistic interpretation comparable to other models, we include MuREI , a constrained variant of MuRE with Rs=Ro=I , as a proxy to TransE for a fairer comparison. Implementation details are included in Appx. D. For evaluation, we generate 2ne evaluation triples for each test triple (for the number of entities ne) by fixing the subject entity es and relation r and replacing the object entity eo with all possible entities and then keeping eo and r fixed and varying es. The obtained scores are ranked to give the standard metric hits@10 (Bordes et al., 2013), i.e. the fraction of times a true triple appears in the top 10 ranked evaluation triples.

heading: 4.1 PERFORMANCE PER RELATION TYPE
text: Tables 3 and 4 report results (hits@10) for each relation and include the relation type and known confounding influences: percentage of relation instances in the training and test sets (approximately equal), number of instances in the test set, Krackhardt hierarchy score (see Appx. E) (Krackhardt, 2014; Balažević et al., 2019a) and maximum and average shortest path between any two related nodes. A further confounding effect is dependence between relations. Balažević et al. (2019b) and Lacroix et al. (2018) show that constraining the rank of relation representations benefits datasets with many relations (particularly when the number of instances per relation is low) due to multi-task learning, which is expected to benefit TuckER on the NELL-995 dataset (200 relations). Note that all models have a comparable number of free parameters.
P1: As predicted, all models tend to perform best at type R relations, with a clear performance gap to other relation types. Also, performance on type S relations appears higher in general than type C. Additive-only models (TransE, MuREI ) perform most poorly on average, in line with prediction since all relation types involve a relatedness component. They achieve their best results on type R relations, where the relation vector can be zero/small. Multiplicative-only DistMult performs well, sometimes best, on type R relations, fitting expectation as it can fully represent those relations and has no additional parameters that may overfit to noise (which may explain where MuRE performs
slightly worse). As expected, MuRE, performs best on average (particularly on WN18RR), and most strongly on S and C type relations that require both multiplicative and additive components. The comparable performance of TuckER on NELL-995 is explained by its ability for multi-task learning.
Other unexpected results also closely align with confounding factors, e.g. that all models perform poorly on the hypernym relation, despite it having type S and a relative abundance of training data (40% of all instances), might be explained by its hierarchical nature (Khs≈ 1 and long paths). The same may explain the reduced performance on type R relations also_see and agentcollaborateswithagent. As found previously, none of the models considered are well suited to modelling hierarchical structures (Balažević et al., 2019a). We also note that the percentage of training instances of a relation does not seem to correlate with its performance, as might typically be expected.
P2/P3: Table 5 shows the symmetry score (∈ [-1, 1] indicating perfect anti-symmetry to symmetry; see Appx. F) for the relation matrix of TuckER and the norm of relation vectors of TransE, MuREI and MuRE on the WN18RR dataset. As expected, type R relations have high symmetry, whereas both other relation types have lower scores, fitting the expectation that TuckER compensates for having no additive component. All additive models learn relation vectors of a noticeably lower norm for type R relations (where, in the extreme, no additive component is required) than for types S and C.
Table 5: Relation matrix symmetry score [-1.1] for TuckER; and relation vector norm for TransE, MuREI and MuRE (WN18RR).
Symmetry Score Vector Norm Relation Type TuckER TransE MuREI MuRE
verb_group R 0.52 5.65 0.76 0.89 derivationally_related_form R 0.54 2.98 0.45 0.69 also_see R 0.50 7.20 0.97 0.97 instance_hypernym S 0.13 18.26 2.98 1.88 member_of_domain_usage C 0.10 11.24 3.18 1.88 member_of_domain_region C 0.06 12.52 3.07 2.11 synset_domain_topic_of C 0.12 23.29 2.65 1.52 member_meronym C 0.12 4.97 1.91 1.97 has_part C 0.13 6.44 1.69 1.25 hypernym S 0.04 9.64 1.53 1.03
0 50 100 150 200 dimension
0.0
0.2
0.4
0.6
0.8
1.0
e ig
e n v a lu
e
R S C
Figure 2: Eigenvalue magnitudes of relation-specific matrices R for MuRE by relation type (WN18RR).
P4: Fig 2 shows eigenvalue magnitudes (scaled relative to their largest and ordered) of the relationspecific matrices R of MuRE, labelled by relation type. Predicted to reflect strength of a relation’s relatedness, they should be highest for type R relations, as observed. For relation types S and C the profiles are more varied, fitting the expectation that the relatedness of those types has greater variability in both choice and size of S, i.e. in the nature and strength of relatedness.
In summary, the results support all predictions made based on the assumption that knowledge graph models benefit from the same latent semantic structure as word embeddings and the relation conditions theoretically derived from them. Our analysis identifies the best performing model per relation type: multiplicative-only DistMult for type R, additive-multiplicative MuRE for types S/C; providing a basis for dataset-dependent model selection. The per-relation insight into where models perform poorly, e.g. on hierarchical or type C relations, can also be used to aid and direct future model design.

heading: 4.2 KNOWLEDGE GRAPH MODEL PREDICTIONS
text: Even though in practice we want to know whether a particular triple is true or false, such independent predictions are not commonly reported or evaluated. Despite many recent link prediction models being able to independently predict the truth of each triple, it is common practice to report rankingbased metrics, e.g. mean reciprocal rank, hits@k, which compare the prediction of a test triple to those of all evaluation triples (see S4). Not only is this computationally costly, the evaluation is flawed if entities are related to more than k others and does not evaluate a model’s ability to independently predict whether “a is related to b”. We address this by considering actual model predictions.
Since for each relation there are n2e possible entity-entity relationships, we sub-sample by computing predictions for all (es, r, eo) triples only for each es, r pair seen in the test set. We split positive predictions (σ(φ(es, r, eo))>0.5) between (i) training and test/validation instances (known truths); and (ii) other, the truth of which is not known. Per relation, we then compute accuracy over the training instances (train) and the test/validation instances (test); and the average number of other truths predicted per es, r pair. Table 6 shows results for MuREI , DistMult, TuckER and MuRE. All
models achieve almost perfect training accuracy. The additive-multiplicative MuRE gives best test set performance, followed (surprisingly) closely by MuREI , with multiplicative models (DistMult and TuckER) performing poorly on all but type R relations. Analysing a sample of “other” positive predictions for a relation of each type (see Appx. G), we estimate that TuckER is relatively accurate but pessimistic (∼0.3 correct of the 0.5 predictions≈ 60%), MuREI is optimistic but inaccurate (∼2.3 of 7.5≈31%), whereas MuRE is both optimistic and accurate (∼1.1 of 1.5≈73%). Fig 3 shows histograms of MuRE prediction probabilities for the same sample relations, split by known truths (training and test/validation) and other instances. There is a clear distinction between relation types: for type R, most train and test triples are classified correctly with high confidence; for types S and C, an increasing majority of incorrect test predictions are far below the decision boundary, i.e. the model is confidently incorrect. For relation types where the model is less accurate, fewer positive predictions are made overall and the prediction distribution is more peaked towards zero.
This analysis probes further into the difficulty models have representing type S/C relations. Given the additional insight provided and the benefit of standalone predictions, we recommend the inclusion of predictive performance in future link prediction work.

heading: 5 CONCLUSION
text: Many models learn low-rank representations for knowledge graph link prediction, yet little is known about the latent structure they learn. We build on recent understanding of PMI-based word embeddings to theoretically establish what a relation representation must achieve to map a word embedding to those it is related to for the relations of knowledge graphs (relation conditions). Such conditions partition relations into three types and also provide a framework to assess loss functions of knowledge graph models. Any model that satisfies a relation’s conditions can represent it if its entity embeddings are set to PMI-based word embeddings, i.e. a solution is known to exist. Whilst knowledge graph models do not learn the parameters of word embeddings, we show that the better a model’s architecture satisfies a relation’s conditions, the better its link prediction performance, fitting the premise that similar latent structure is exploited. Overall, we extend previous understanding of how semantic relations are encoded in relationships between PMI-based word embeddings – generalising from a limited set, e.g. similarity and analogy; we demonstrate commonality between the latent structure learned by PMI-based word embeddings (e.g. W2V) and knowledge graph representation models; and we provide novel insight into knowledge graph models by evaluating their predictive performance.

heading: A CATEGORISING WORDNET RELATIONS
text: Table 7 describes how each WN18RR relation was assigned to its respective category.

heading: B ANALYSING NELL-995 RELATIONS
text: NELL-995: We categorise a random subsample of 12 relations from the NELL-995 dataset (Xiong et al., 2017) containing 75,492 entities and 200 relations (a subset of NELL (Carlson et al., 2010)), which span our identified relation types (see Table 8). Explanation for the relation category assignment is shown in Table 9.

heading: C SPLITTING THE NELL-995 DATASET
text: The test set of NELL-995 created by Xiong et al. (2017) contains only 10 out of 200 relations present in the training set. To ensure a fair representation of all training set relations in the validation and test sets, we create new validation and test set splits by combining the initial validation and test sets with the training set and randomly selecting 20,000 triples each from the combined dataset.
D IMPLEMENTATION DETAILS
All algorithms are re-implemented in PyTorch with the Adam optimizer (Kingma & Ba, 2015) that minimises binary cross-entropy loss, using hyper-parameters that work well for all models (learning rate: 0.001, batch size: 128, number of negative samples: 50). Entity and relation embedding dimensionality is set to de=dr=200 for all models except TuckER, for which dr=30 (Balažević et al., 2019b).

heading: E KRACKHARDT HIERARCHY SCORE
text: The Krackhardt hierarchy score measures the proportion of node pairs (x, y) where there exists a directed path x→ y, but not y → x; and it takes a value of one for all directed acyclic graphs, and zero for cycles and cliques (Krackhardt, 2014; Balažević et al., 2019a).
Let M ∈Rn×n be the binary reachability matrix of a directed graph G with n nodes, with Mi,j = 1 if there exists a directed path from node i to node j and 0 otherwise. The Krackhardt hierarchy score of G is defined as:
KhsG =
∑n i=1 ∑n j=1 1(Mi,j == 1 ∧Mj,i == 0)∑n i=1 ∑n j=1 1(Mi,j == 1) . (1)

heading: F SYMMETRY SCORE
text: The symmetry score∈ [−1, 1] (Hubert & Baker, 1979) for a relation matrix R∈Rde×de is defined as:
s =
∑∑ i 6=j RijRji − ( ∑∑ i6=j Rij) 2
de(de−1)∑∑ i 6=j R 2 ij − ( ∑∑ i6=j Rij) 2
de(de−1)
, (2)
where 1 indicates a symmetric and -1 an anti-symmetric matrix.

heading: G “OTHER” PREDICTED FACTS
text: Tables 10 to 13 shows a sample of the unknown triples (i.e. those formed using the WN18RR entities and relations, but not present in the dataset) for the derivationally_related form (R), instance_hypernym (S) and synset_domain_topic_of (C) relations at a range of probabilities (σ(φ(es, r, eo)) ≈ {0.4, 0.6, 0.8, 1}), as predicted by each model. True triples are indicated in bold; instances where a model predicts an entity is related to itself are indicated in blue.
Ta bl
e 10
:“ O
th er
” fa
ct s
as pr
ed ic
te d
by M
uR E I
.
R el
at io
n (T
yp e)
σ (φ (e
s ,r ,e
o )) ≈
0 .4
σ (φ (e
s ,r ,e
o )) ≈
0. 6
σ (φ (e
s ,r ,e
o )) ≈
0 .8
σ (φ (e
s ,r ,e
o )) ≈
1
de riv
at io
na lly _ re la te d_ fo rm (R )
(e qu
al iz
er _N
N _2
,s et
_o ff_
V B
_5 )
(e xt
ra po
la tio
n_ N
N _1
,m at
hs _N
N _1
) (s
ew er
_N N
_2 ,s
tit ch
er _N
N _1
) (t
ra il_
V B
_2 ,t
ra il_
V B
_2 )
(c on
st el
la tio
n_ N
N _2
,s at
el lit
e_ N
N _3
) (s
pr ea
d_ V
B _5
,c ir
cu la
ri ze
_V B
_3 )
(l ar
d_ V
B _1
,v eg
et ab
le _o
il_ N
N _1
) (w
or sh
ip _V
B _1
,w or
sh ip
_V B
_1 )
(s hr
in k_
V B
_3 ,s
ub tr
ac tio
n_ N
N _2
) (fl
au nt
_N N
_1 ,s
ho w
in g_
N N
_2 )
(s nu
gg le
_N N
_1 ,d
ra w
_c lo
se _V
B _3
) (s
te er
_V B
_1 ,s
te er
_V B
_1 )
(c on
tin ue
_V B
_1 0,
pr oc
ee d_
V B
_1 )
(e xt
ra po
la te
_V B
_3 ,s
yn th
es is
_N N
_3 )
(tr ai
n_ V
B _3
,t ra
in in
g_ N
N _1
) (s
or t_
ou t_
V B
_1 ,s
or t_
ou t_
V B
_1 )
(s up
po rt
_V B
_6 ,d
ef en
d_ V
B _5
) (s
tr at
eg is
t_ N
N _1
,m ac
hi na
tio n_
N N
_1 )
(s cr
at ch
_V B
_3 ,s
ki n_
se ns
at io
n_ N
N _1
) (m
ak e_
fu ll_
V B
_1 ,m
ak e_
fu ll_
V B
_1 )
(s hu
tte r_
N N
_1 ,fi
ll_ up
_V B
_3 )
(c ru
sh _V
B _4
,g ri
nd _V
B _2
) (s
ch em
e_ N
N _5
,s ch
em at
iz at
io n_
N N
_1 )
(u til
iz e_
V B
_1 ,u
til iz
e_ V
B _1 ) (y aw ni ng _N N _1 ,p at el la r_ re fle x_ N N _1 ) (s pi ke _V B _5 ,s te ad y_ V B _2 ) (o rd ai n_ V B _3 ,v es t_ V B _1 ) (g eo lo gy _N N _1 ,g eo lo gy _N
N _1 ) (y aw _N N _1 ,s pi ra l_ V B _1 ) (l ic ki ng _N N _1 ,v an qu is h_ V B _1 ) (l ie _V B _1 ,f ro nt _e nd _N N _1 ) (z oo lo gy _N N _2 ,z oo lo gy _N N _2 ) (s tr at um _N N _2 ,s oc ia l_ gr ou p_ N N _1 ) (s yn th et ic al _J J_ 1, sy nt he si ze r_ N N _2 ) (t re ad _N N _1 ,s te p_ N N _9 ) (u ra no lo gy _N N _1 ,u ra no lo gy _N N
_1 )
(d ue
l_ V
B _1
,s cu
ffl e_
N N
_3 )
(r ea
liz at
io n_
N N
_2 ,e
m bo
di m
en t_
N N
_3 )
(r eg
is te
r_ N
N _3
,fi le
_a w
ay _V
B _1
) (t
ra ve
l_ V
B _1
,t ra
ve l_
V B
_1 )
in st
an ce _ hy pe rn ym (S )
(t ho
m as
_a qu
in as
_N N
_1 ,m
ar ty
r_ N
N _2
) (ta
iw an
_N N
_1 ,a
si an
_n at
io n_
N N
_1 )
(p ro
ph et
s_ N
N _1
,g os
pe ls
_N N
_1 )
(h el
si nk
i_ N
N _1
,u rb
an _c
en te
r_ N
N _1 ) (v ol ca no _i sl an ds _N N _1 ,v ol ca no _N N _2 ) (s t._ gr eg or y_ of _n ._ N N _1 ,c an on iz at io n_ N N _1 ) (m al co lm _x _N N _1 ,p as si ve _r es is te r_ N N _1 ) (m an nh ei m _N N _1 ,s to ck ad e_ N N _2 ) (c ap e_ ho rn _N N _1 ,u rb an _c en te r_ N N _1 ) (s t._ gr eg or y_ of _n ._ N N _1 ,s ai nt _V B _2 ) (t ai w an _N N _1 ,n at io na l_ ca pi ta l_ N N _1 ) (n ip po n_ N N _1 ,n ip po n_ N N _1 ) (b er ge n_ N N _1 ,n at io na l_ ca pi ta l_ N N _1 ) (m cc or m ic k_ N N _1 ,fi nd _V B _8 ) (t ru th _N N _5 ,a bo lit io ni sm _N N _1 ) (v ic to r_ hu go _N N _1 ,n ov el _N N _1 ) (m ar sh al l_ N N _2 ,g en er al sh ip _N N _1 ) (s t._ gr eg or y_ i_ N N _1 ,b is ho p_ N N _1 ) (th om as _a qu in as _N N _1 ,s ai nt _V B _2 ) (r eg io m on ta nu s_ N N _1 ,u ra no lo gy _N N
_1 )
(n an
se n_
N N
_1 ,v
en tu
re r_
N N
_2 )
(r ic
ha rd
_b uc
km in
st er
_f ._
N N
_1 ,t
ec hn
ol og
ic al
_J J_
2) (c
en tr
al _a
m er
ic a_
N N
_1 ,s
._ am
._ na
tio n_
N N
_1 )
(p ro
ph et
s_ N
N _1
,b oo
k_ N
N _1 0) (w is co ns in _N N _2 ,s ta te _c ap ita l_ N N _1 ) (t ho m as _a qu in as _N N _1 ,a rc hb is ho p_ N N _1 ) (d e_ m ill e_ N N _1 ,d an ce _V B _1 ) (t ho m as _a qu in as _N N _1 ,c hu rc
h_ fa
th er
_N N
_1 )
(p ru
ss ia
_N N
_1 ,s
to ck
ad e_
N N
_2 )
(m ar
sh al
l_ N
N _2
,g en
er al
_o ffi
ce r_
N N
_1 )
(s t._
gr eg
or y_
i_ N
N _1
,a po
st le
_N N
_3 )
(w oo
dy _g
ut hr
ie _N
N _1
,m in
st re
l_ V
B _1 ) (d e_ m ill e_ N N _1 ,d an ci ng -m as te r_ N N _1 ) (n ew m an _N N _2 ,p ri m at es hi p_ N N _1 ) (f er til e_ cr es ce nt _N N _1 ,a si an _n at io n_ N N _1 ) (c en tr al _a m er ic a_ N N _1 ,c ._ am ._ na tio n_
N N
_1 )
(a eg
ea n_
se a_
N N
_1 ,a
eg ea
n_ is
la nd
_N N
_1 )
(t ho
m as
_t he
_a po
st le
_N N
_1 ,s
an ct
ify _V
B _1
) (r
ob er
t_ ow
en _N
N _1
,i nd
us tr
y_ N
N _1
) (a
eg ea
n_ se
a_ N
N _1
,i sl
an d_
N N
_1 )
sy ns
et _d
om ai n_ to pi c_ of (C )
(w ri
te _V
B _8
,t ap
e_ V
B _3
) (d
ra w
_V B
_6 ,c
re at
iv e_
pe rs
on _N
N _1
) (l
ib el
_N N
_1 ,s
ul ly
_V B
_3 )
(li be
l_ N
N _1
,d is
pa ra
ge m
en t_
N N
_1 )
(i nt
ro ve
rt _N
N _1
,s ci
en tifi
c_ di
sc ip
lin e_
N N
_1 )
(s ub
or de
r_ N
N _1
,t ax
on om
ic _g
ro up
_N N
_1 )
(r el
at io
ns hi
p_ N
N _4
,r el
at io
ns hi
p_ N
N _4
) (r
ol l-o
n_ ro
llof
f_ N
N _1
,t ra
ns po
rt _N
N _1 ) (li be l_ N N _1 ,s lu r_ N N _2 ) (d ra w _V B _6 ,d ra w _V B _6 ) (e ty m ol og iz in g_ N N _1 ,l in gu is tic s_ N N _1 ) (p ra nc e_ V B _4 ,e qu es tr ia n_ sp or t_ N N _1 ) (e ty m ol og iz in g_ N N _1 ,l aw _N N _1 ) (fi rs t_ sa ck er _N N _1 ,b al lg am e_ N N _2 ) (tu rn _V B _1 2, cu lti va tio n_ N N _2 ) (li be l_ N N _1 ,t ra du ce m en t_ N N _1 ) (te m pl e_ N N _4 ,p la ce _o f_ w or sh ip _N N _1 ) (a lc he m iz e_ V B _1 ,m od ify _V B _3 ) (b ry nh ild _N N _1 ,m yt ho lo gi ze _V B _2 ) (s el l_ V B _1 ,s el lin g_ N N _1 ) (t ri al _i m pr es si on _N N _1 ,p ro of _V B _1 ) (s er m on _N N _1 ,s er m on _N N _1 ) (b ry nh ild _N N _1 ,m yt h_ N N _1 ) (t ro t_ V B _2 ,r id e_ ho rs eb ac k_ V B _1 ) (f ri en d_ of _t he _c ou rt _N N _1 ,w ar _m ac hi ne _N N _1 ) (s ai nt _V B _2 ,c at ho lic _c hu rc h_ N N _1 ) (a ss is t_ N N _2 ,a m ._ fo ot ba ll_ ga m e_ N N _1 ) (p ra nc e_ V B _4 ,r id e_ ho rs eb ac k_ V B _1 ) (m ul tiv ._ an al ys is _N N _1 ,a pp lie d_ m at h_ N N _1 ) (m al e_ JJ _1 ,m as cu lin e_ JJ _2 ) (m itz va h_ N N _2 ,h um an _a ct iv ity _N N _1 ) (g al lo p_ V B _1 ,r id e_ ho rs eb ac k_ V B _1 ) (s el l_ V B _1 ,t ra ns ac tio n_ N N _1 ) (fi re _V B _3 ,z oo lo gy _N N _2 ) (d ri ve _N N _1 2, dr iv e_ V B _8 ) (b ry nh ild _N N _1 ,m yt ho lo gy _N N _2 ) (d ra w _V B _6 ,r ep re se nt _V B _9 ) (s el l_ V B _1 ,s el l_ V B _1 ) (r el at io ns hi p_ N N _4 ,b io lo gy _N N _1 ) (d ri ve _N N _1 2, ba dm in to n_ N N _1 )
Ta bl
e 11
:“ O
th er
” fa
ct s
as pr
ed ic
te d
by D
is tM
ul t.
R el
at io
n (T
yp e)
σ (φ (e
s ,r ,e
o )) ≈
0 .4
σ (φ (e
s ,r ,e
o )) ≈
0 .6
σ (φ (e
s ,r ,e
o )) ≈
0. 8
σ (φ (e
s ,r ,e
o )) ≈
1
de riv
at io
na lly _ re la te d_ fo rm (R )
(s ta
g_ V
B _3
,u nd
er co
ve r_
w or
k_ N
N _1
) (d
is h_
N N
_2 ,s
te w
_N N
_2 )
(s hr
in k_
N N
_1 ,p
ed ol
og y_
N N
_1 )
(a lli
te ra
te _V
B _1
,v er
si fie
r_ N
N _1 ) (p ri nt _V B _4 ,p ub lis he r_ N N _2 ) (e xp os e_ V B _3 ,s ho w _N N _1 ) (fi ni sh _V B _6 ,fi ni sh in g_ N N _2 ) (g eo lo gy _N N _1 ,s tr uc tu ra l_ JJ _5 ) (c ri er _N N _3 ,p itc hm an _N N _2 ) (s ys te m _N N _9 ,o rd er lin es s_ N N _1 ) (p la y_ V B _2 6, pl ay in g_ N N _3 ) (r es ec t_ V B _1 ,a m pu ta tio n_ N N _2 ) (p la y_ V B _2 6, tu rn _N N _1 0) (s pr ea d_ N N _4 ,s tr ew _V B _1 ) (c en tr al iz at io n_ N N _1 ,u ni te _V B _6 ) (n ut ri tio n_ N N _3 ,m an _N N _4 ) (c ou nt _V B _4 ,r ec ite _V B _2 ) (t ak e_ do w n_ V B _2 ,p ut _V B _2 ) (e xi st en ce _N N _1 ,l iv in g_ N N _3 ) (s ai nt _N N _3 ,s an ct ify _V B _1 ) (v iv id ne ss _N N _2 ,i m bu e_ V B _3 ) (w re st le _V B _4 ,w re st le r_ N N _1 ) (m ou th _V B _3 ,s as si ng _N N _1 ) (r ig ht _fi el de r_ N N _1 ,l ef tfi el d_ N N
_1 )
(s ea
_m ew
_N N
_1 ,l
ar us
_N N
_1 )
(a ut
ot r._
or ga
ni sm
_N N
_1 ,e
pi ph
yt ic
_J J_
1) (c
on st
el la
tio n_
N N
_2 ,s
ta r_
N N
_1 )
(li st
_V B
_4 ,s
lo pe
_N N
_2 )
(a lk
al i_
N N
_2 ,a
ci di
fy _V
B _2
) (d
ue l_
V B
_1 ,s
lu gf
es t_
N N
_1 )
(p ri
nt _V
B _4
,p ub
lis hi
ng _h
ou se
_N N
_1 )
(l ie
ut en
an cy
_N N
_1 ,c
ap ta
in _N
N _1 ) (s ee _V B _1 7, un de rs ta nd _V B _2 ) (v oc al _N N _2 ,r oc k_ st ar _N N _1 ) (p uz zl e_ V B _2 ,s ec re t_ N N _3 ) (tr ea d_ N N _1 ,s te p_ V B _7 ) (s hu n_ V B _1 ,h ed gi ng _N N _2 ) (s m el lin g_ N N _1 ,s ce nt _V B _1 ) (u ra no lo gy _N N _1 ,t t_ N N _1 ) (e xe nt er at io n_ N N _1 ,e nu cl ea te _V
B _2
)
in st
an ce _ hy pe rn ym (S )
(w is
co ns
in _N
N _2
,u rb
an _c
en te
r_ N
N _1
) (m
is si
ss ip
pi _r
iv er
_N N
_1 ,a
m er
ic an
_s ta
te _N
N _1
) (d
ee p_
so ut
h_ N
N _1
,s ou
th _N
N _1
) (h
el si
nk i_
N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(m ar
sh al
l_ N
N _2
,l ie
ut en
an t_
ge ne
ra l_
N N
_1 )
(r ._
e. _b
yr d_
N N
_1 ,c
om m
is si
on ed
_o ffi
ce r_
N N
_1 )
(c ap
ita l_
of _g
am bi
a_ N
N _1
,u rb
an _c
en te
r_ N
N _1
) (t
he _n
az ar
en e_
N N
_1 ,s
av e_
V B
_7 )
(a bi
dj an
_N N
_1 ,c
ot e_
d’ iv
oi re
_N N
_1 )
(k ob
en ha
vn _N
N _1
,u rb
an _c
en te
r_ N
N _1
) (s
ou th
_w es
t_ af
ri ca
_N N
_1 ,a
fr ic
a_ N
N _1
) (ir
is h_
ca pi
ta l_
N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(w or
ld _w
ar _i
_N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(t he
_g am
bi a_
N N
_1 ,a
fr ic
a_ N
N _1
) (b
ra nd
en bu
rg _N
N _1
,u rb
an _c
en te
r_ N
N _1
) (r
._ e.
_b yr
d_ N
N _1
,i ns
pe ct
or _g
en er
al _N
N _1 ) (s t._ pa ul _N N _2 ,e va ng el is t_ N N _2 ) (t ir ic h_ m ir _N N _1 ,u rb an _c en te r_ N N _1 ) (s ie rr a_ ne va da _N N _1 ,u rb an _c en te r_ N N _1 ) (r ._ e. _b yr d_ N N _1 ,c hi ef _o f_ st af f_ N N _1 ) (d ee p_ so ut h_ N N _1 ,u rb an _c en te r_ N N _1 ) (r ._ e. _b yr d_ N N _1 ,m ili ta ry _a dv is or _N N _1 ) (m al co lm _x _N N _1 ,e m an ci pa tio ni st _N N _1 ) (c en tr al _a m er ic a_ N N _1 ,c ._ am ._ na tio n_ N N
_1 )
(n up
ts e_
N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(r ._
e. _b
yr d_
N N
_1 ,a
id e-
de -c
am p_
N N
_1 )
(n or
th _p
la tte
_r iv
er _N
N _1
,u rb
an _c
en te
r_ N
N _1
) (m
al co
lm _x
_N N
_1 ,e
nv ir
on m
en ta
lis t_
N N
_1 )
(t ic
in o_
N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(t am
pa _b
ay _N
N _1
,u rb
an _c
en te
r_ N
N _1
) (o
sl o_
N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(th e_
na za
re ne
_N N
_1 ,c
hr is
tia n_
JJ _1 ) (a eg ea n_ se a_ N N _1 ,a eg ea n_ is la nd _N N _1 ) (t id ew at er _r eg io n_ N N _1 ,s ou th _N N _1 ) (z ai re _r iv er _N N _1 ,u rb an _c en te r_ N N _1 ) (t ho m as _a qu in as _N N _1 ,c hu rc h_ fa th er
_N N
_1 )
(c ow
pe ns
_N N
_1 ,w
ar _o
f_ am
._ in
d. _N
N _1
) (r
._ e.
_b yr
d_ N
N _1
,e xe
cu tiv
e_ of
fic er
_N N
_1 )
(t ra
ns yl
va ni
an _a
lp s_
N N
_1 ,u
rb an
_c en
te r_
N N
_1 )
(t he
_n az
ar en
e_ N
N _1
,e l_
ni no
_N N
_2 )
sy ns
et _d
om ai n_ to pi c_ of (C )
(l im
ita tio
n_ N
N _4
,t ra
m m
el _V
B _2
) (r
ol l-o
n_ ro
llof
f_ N
N _1
,t ra
ns po
rt _N
N _1
) (e
ty m
ol og
iz in
g_ N
N _1
,e xp
la na
tio n_
N N
_1 )
(fl at
_J J_
5, m
at te
_N N
_2 )
(l ig
ht _c
ol on
el _N
N _1
,c ol
on el
_N N
_1 )
(h iz
b_ ut
-t ah
ri r_
N N
_1 ,a
si a_
N N
_1 )
(f er
ry _V
B _3
,t ra
ve l_
V B
_1 )
(e ty
m ol
og iz
in g_
N N
_1 ,d
er iv
e_ V
B _3 ) (n ur se _V B _1 ,n ur si ng _N N _1 ) (s lu gg er _N N _1 ,s of tb al l_ ga m e_ N N _1 ) (p ub lic _p ro se cu to r_ N N _1 ,p ro se cu te _V B _2 ) (h ol e_ ou t_ V B _1 ,h ol e_ N N _3 ) (s er m on _N N _1 ,p ro ph es y_ V B _2 ) (s er m on _N N _1 ,s er m on iz e_ V B _1 ) (a lc he m iz e_ V B _1 ,m od ify _V B _3 ) (r el at io ns hi p_ N N _4 ,r el at io n_ N N
_1 )
(l ib
el _N
N _1
,p ra
ct ic
e_ of
_l aw
_N N
_1 )
(d ra
w _V
B _6
,d ra
w er
_N N
_3 )
(l ib
el _N
N _1
,l ib
el _V
B _1
) (d
ri ve
_N N
_1 2,
ba dm
in to
n_ N
N _1 ) (s lu gg er _N N _1 ,b as eb al l_ pl ay er _N N _1 ) (tu rn _V B _1 2, pl ow _N N _1 ) (t ur n_ V B _1 2, til l_ V B _1 ) (e ty m ol og iz in g_ N N _1 ,e ty m ol og iz
e_ V
B _2 ) (r na _N N _1 ,c he m is tr y_ N N _1 ) (a ss is t_ N N _2 ,s of tb al l_ ga m e_ N N _1 ) (h it_ N N _1 ,h it_ V B _1 ) (m at ri x_ al ge br a_ N N _1 ,d ia go na liz at io n_ N
N _1 ) (m et ri fy _V B _1 ,v er si fy _V B _1 ) (c ou nc il_ N N _2 ,a ss em bl y_ N N _4 ) (fi re _V B _3 ,fl am in g_ N N _1 ) (c ab in et w or k_ N N _2 ,w oo dw or ki ng _N N _1 ) (t ri al _i m pr es si on _N N _1 ,p ub lis h_ V B _1 ) (th ro ug hp ut _N N _1 ,t ur no ut _N N _4 ) (r in g_ N N _4 ,c he m ic al _c ha in _N N _1 ) (c ab in et w or k_ N N _2 ,b ot to m _V B _1 ) (tu rn _V B _1 2, pl ow m an _N N _1 ) (c re am _V B _1 ,c re am _N N _2 ) (li bi di na l_ en er gy _N N _1 ,c ha rg e_ N N _9 ) (c ab in et w or k_ N N _2 ,u ph ol st er _V B _1 )
Ta bl
e 12
:“ O
th er
” fa
ct s
as pr
ed ic
te d
by Tu
ck E
R .
R el
at io
n (T
yp e)
σ (φ (e
s ,r ,e
o )) ≈
0. 4
σ (φ (e
s ,r ,e
o )) ≈
0 .6
σ (φ (e
s ,r ,e
o )) ≈
0. 8
σ (φ (e
s ,r ,e
o )) ≈
1
de riv
at io
na lly _ re la te d_ fo rm (R )
(t ym
pa ni
st _N
N _1
,g on
g_ N
N _2
) (m
as h_
N N
_2 ,m
ill _V
B _2
) (t
ak e_
ch an
ce s_
V B
_1 ,v
en tu
re _N
N _1
) (v
en tu
re r_
N N
_2 ,v
en tu
re r_
N N
_2 )
(in di
ca tio
n_ N
N _1
,s ig
na liz
e_ V
B _2
) (w
al k_
V B
_9 ,z
im m
er _f
ra m
e_ N
N _1
) (s
hu tte
r_ N
N _1
,fi ll_
up _V
B _3
) (d
yn am
iti st
_N N
_1 ,d
yn am
iti st
_N N
_1 )
(tu rn
_o ve
r_ V
B _3
,r ot
at io
n_ N
N _3
) (u
se _V
B _5
,u til
ity _N
N _2
) (e
xi t_
N N
_3 ,l
ea ve
_V B
_1 )
(lo ve
_V B
_3 ,l
ov er
_N N
_2 )
(d at
e_ V
B _5
,g eo
lo gi
ca l_
da tin
g_ N
N _1
) (m
us ic
al _i
ns tr
um en
t_ N
N _1
,w ri
te _V
B _6
) (tr
em bl
er _N
N _1
,v ib
ra te
_V B
_1 )
(s nu
gg le
_N N
_1 ,s
qu ee
ze _V
B _8 ) (s et _V B _2 3, em bl em _N N _2 ) (li ni ng _N N _3 ,w ra p_ up _V B _1 ) (m ot iv at or _N N _1 ,t ri p_ V B _4 ) (d ep ar te d_ N N _1 ,d ie _V B _2 ) (ty ro _N N _1 ,s ta rt _V B _5 ) (s cr ap _V B _2 ,s tr ug gl e_ N N _2 ) (s up po rt _V B _6 ,i nd or se m en t_ N N _1 ) (p os iti on _V B _1 ,p la ce m en t_ N
N _1 ) (id en tifi ca tio n_ N N _1 ,n am e_ V B _5 ) (ta pe _V B _3 ,t ap e_ re co rd er _N N _1 ) (f ed er at e_ V B _2 ,c on fe de ra tio n_ N N _1 ) (r ep en ta nt _J J_ 1, re pe nt an t_ JJ _1 ) (s ta bb er _N N _1 ,t hr us t_ V B _5 ) (v in di ca te _V B _2 ,j us tifi ca tio n_ N N _2 ) (ta ke _o ve r_ V B _6 ,r et ur n_ N N _7 ) (tr ea d_ N N _1 ,s te p_ V B _7 ) (j us tifi ca tio n_ N N _1 ,a po lo gy _N N _2 ) (le ac hi ng _N N _1 ,p er co la te _V B _3 ) (w ai t_ on _V B _1 ,s up po rt er _N N _3 ) (s to ck is t_ N N _1 ,s to ck is t_ N N _1 ) (m an uf ac tu re _V B _1 ,p re va ri ca tio n_ N N _1 ) (s yn ch ro ni ze _V B _2 ,s yn ch ro sc op e_ N N _1 ) (d en ot e_ V B _3 ,p ro m ul ga tio n_ N N _1 ) (p hi la nt hr op is t_ N N _1 ,p hi la nt hr op is
t_ N
N _1
)
in st
an ce _ hy pe rn ym (S )
(d ee
p_ so
ut h_
N N
_1 ,s
ou th
_N N
_1 )
(t ho
m as
_a qu
in as
_N N
_1 ,b
is ho
p_ N
N _1
) (c
ow pe
ns _N
N _1
,s ie
ge _N
N _1
) (r
._ e.
_b yr
d_ N
N _1
,s ie
ge _N
N _1 ) (s t._ pa ul _N N _2 ,o rg an is t_ N N _1 ) (ir is h_ ca pi ta l_ N N _1 ,u rb an _c en te r_ N N _1 ) (m cc or m ic k_ N N _1 ,a rm s_ m an uf ac tu re r_ N N _1 ) (s ha w _N N _3 ,w om en ’s _r ig ht
is t_
N N
_1 )
(h el
si nk
i_ N
N _1
,u rb
an _c
en te
r_ N
N _1
) (th
om as
_t he
_a po
st le
_N N
_1 ,a
po st
le _N
N _2
) (t
ho m
as _t
he _a
po st
le _N
N _1
,e va
ng el
is t_
N N
_2 )
(a eg
ea n_
se a_
N N
_1 ,a
eg ea
n_ is
la nd
_N N
_1 )
(m al
co lm
_x _N
N _1
,e m
an ci
pa tio
ni st
_N N
_1 )
(s t._
pa ul
_N N
_2 ,a
po st
le _N
N _3
) (m
cc or
m ic
k_ N
N _1
,t ec
hn ol
og is
t_ N
N _1
) (t
ho m
as _a
qu in
as _N
N _1
,c hu
rc h_
fa th
er _N
N _1 ) (t ho m as _t he _a po st le _N N _1 ,c hu rc h_ fa th er _N N _1 ) (m cc or m ic k_ N N _1 ,p ai nt er _N N _1 ) (s t._ gr eg or y_ i_ N N _1 ,c hu rc h_ fa th er _N N _1 ) (s t._ gr eg or y_ of _n ._ N N _1 ,s er m on iz er _N N _1 ) (t ho m as _t he _a po st le _N N _1 ,t ro gl od yt e_ N N _1 ) (r ob er t_ ow en _N N _1 ,m ov ie _m ak er _N N _1 ) (m cc or m ic k_ N N _1 ,e le ct ri ca l_ en gi ne er _N N _1 ) (t he re sa _N N _1 ,m on k_ N N _1 ) (m is si ss ip pi _r iv er _N N _1 ,a m er ic an _s ta te _N N _1 ) (s t._ pa ul _N N _2 ,p hi lo so ph er _N N _1 ) (i bn -r os hd _N N _1 ,p ed ag og ue _N N _1 )
sy ns
et _d
om ai n_ to pi c_ of (C )
(r ol
l-o n_
ro ll-
of f_
N N
_1 ,m
ot or
ca r_
N N
_1 )
(d ri
ve _N
N _1
2, ba
dm in
to n_
N N
_1 )
(li be
l_ N
N _1
,l eg
is la
tu re
_N N
_1 )
(r ol
l-o n_
ro ll-
of f_
N N
_1 ,p
as se
ng er
_v eh
ic le
_N N
_1 )
Ta bl
e 13
:“ O
th er
” fa
ct s
as pr
ed ic
te d
by M
uR E
.
R el
at io
n (T
yp e)
σ (φ (e
s ,r ,e
o )) ≈
0. 4
σ (φ (e
s ,r ,e
o )) ≈
0. 6
σ (φ (e
s ,r ,e
o )) ≈
0. 8
σ (φ (e
s ,r ,e
o )) ≈
1
de riv
at io
na lly _ re la te d_ fo rm (R )
(s ur
ro un
d_ V
B _1
,w al
l_ N
N _1
) (w
or d_
pi ct
ur e_
N N
_1 ,s
ke tc
h_ V
B _2
) (s
m el
lin g_
N N
_1 ,w
in d_
V B
_4 )
(s po
lia tio
n_ N
N _2
,s ac
k_ V
B _1 ) (u np le as an t_ JJ _1 ,u np al at ab le ne ss _N N _1 ) (d ev el op _V B _1 0, ad ju st m en t_ N N _4 ) (t ry _o ut _V B _1 ,s om at ic _c el l_ nu cl ea r_ tr an sp la nt at io n_ N N _1 ) (d es ir e_ N N _2 ,h op e_ V B _2 ) (lo ve _V B _3 ,e nj oy m en t_ N N _2 ) (g lo ss _V B _3 ,c om m en ta ry _N N _1 ) (li gh tin g_ N N _4 ,s et _o n_ fir e_ V B _1 ) (s nu ffl e_ V B _3 ,w hi ne _N N _1 ) (m ag ni tu de _N N _1 ,t al l_ JJ _1 ) (v io la te _V B _2 ,v io la tio n_ N N _3 ) (s ym ph al an gu s_ N N _1 ,o ne -h al f_ N N _1 ) (n as al iz at io n_ N N _1 ,s ou nd _o
ut _V
B _1 ) (te st ify _V B _2 ,i nf or m at io n_ N N _1 ) (s uf fo ca te _V B _1 ,s tr an gl er _t re e_ N N _1 ) (ju st _J J_ 3, va lid ity _N N _1 ) (tr ea d_ N N _1 ,s te p_ V B _7 ) (c on ne ct _V B _6 ,c on ve rg in g_ N N _1 ) (n um be r_ V B _3 ,p oi nt _N N _1 2) (r ep ro ve _V B _1 ,t al ki ng _t o_ N N _1 ) (y ea rn _V B _1 ,p in in g_ N N _1 ) (c on ne ct _V B _6 ,c on ne xi on _N N _4 ) (d ev el op _V B _1 0, or ga ni c_ pr oc es s_ N N _1 ) (s us ta in _V B _5 ,b ea m _N N _2 ) (u nr el ia bl en es s_ N N _1 ,a rb itr ar y_ JJ _1 ) (o pe ra te _V B _4 ,p sy op _N N _1 ) (p lic at io n_ N N _1 ,t w is t_ V B _4 ) (s pr in g_ N N _6 ,h ur dl e_ V B _1 ) (t ra ve st y_ N N _2 ,t ra ve st y_ N N _2 ) (m ar ke t_ V B _1 ,t ra de _N N _4 ) (s pl it_ up _V B _3 ,s ep ar at io n_ N N _5 ) (s pa rk _N N _1 ,s ci nt ill at e_ V B _1 ) (s pa rk _N N _1 ,s pa rk le _V B _1 ) (o pe ra te _V B _4 ,m is si on _N N _2 ) (p lic at io n_ N N _1 ,w ri nk le _V B _2 ) (u til ity _N N _2 ,f un ct io na l_ JJ _1 ) (s to ck is t_ N N _1 ,s to ck is t_ N N _1 )
in st
an ce _ hy pe rn ym (S )
(m al
co lm
_x _N
N _1
,h ip
st er
_N N
_1 )
(c en
tr al
_a m
er ic
a_ N
N _1
,c en
tr al
_a m
er ic
a_ N
N _1
) (t
he re
sa _N
N _1
,m on
k_ N
N _1 ) (t he _n az ar en e_ N N _1 ,j ud ai sm _N N _2 ) (s t._ gr eg or y_ i_ N N _1 ,c hu rc h_ fa th er _N N _1 ) (n ip po n_ N N _1 ,e ur op ea n_
na tio
n_ N
N _1 ) (o ld _l in e_ st at e_ N N _1 ,r iv er _N N _1 ) (s ou th _k or ea _N N _1 ,a fr ic an _n at io n_ N N _1 ) (g re at _l ak es _N N _1 ,r iv er _N N _1 ) (r ._ e. _b yr d_ N N _1 ,c om m is si on ed _o ffi ce r_ N N _1 ) (m al co lm _x _N N _1 ,p as si ve _r es is te r_ N N _1 ) (r ._ e. _b yr d_ N N _1 ,n on co m m is si on ed _o
ffi ce
r_ N
N _1 ) (s ou th _k or ea _N N _1 ,p en in su la _N N _1 ) (m al co lm _x _N N _1 ,b ir th -c on tr ol _r ef or m er _N N _1 ) (w or ld _w ar _i _N N _1 ,p itc he d_ ba ttl e_ N N _1 ) (s t._ gr eg or y_ of _n ._ N N _1 ,v ic ar _o f_ ch ri st _N N _1 ) (lo s_ an ge le s_ N N _1 ,p or t_ N N _1 ) (ir is h_ ca pi ta l_ N N _1 ,u rb an _c en te r_ N N _1 ) (n ip po n_ N N _1 ,i ta lia n_ re gi on _N N _1 ) (g re at _l ak es _N N _1 ,c an ad ia n_ pr ov in ce _N N _1 ) (v ol ca no _i sl an ds _N N _1 ,u rb an _c en te r_ N N _1 ) (r ob er t_ ow en _N N _1 ,t yc oo n_ N N _1 ) (t ra ns yl va ni an _a lp s_ N N _1 ,u rb an _c en te r_ N N _1 ) (n ip po n_ N N _1 ,a m er ic an _s ta te _N N _1 ) (m an da la y_ N N _1 ,n at io na l_ ca pi ta l_ N N _1 ) (g et ty sb ur g_ N N _2 ,s ie ge _N N _1 ) (h el si nk i_ N N _1 ,u rb an _c en te r_ N N _1 ) (n an _l in g_ N N _1 ,u rb an _c en te r_ N N _1 ) (w is co ns in _N N _2 ,g eo gr ap hi ca l_ re gi on _N N _1 ) (c ap ita l_ of _g am bi a_ N N _1 ,u rb an _c en te r_ N N _1 )
sy ns
et _d
om ai n_ to pi c_ of (C )
(li be
l_ N
N _1
,c ri
m in
al _l
aw _N
N _1
) (w
ri te
_V B
_8 ,t
ra ns
cr ip
tio n_
N N
_5 )
(a ss
is t_
N N
_2 ,a
m ._
fo ot
ba ll_
ga m
e_ N
N _1 ) (b ry nh ild _N N _1 ,m yt ho lo gy _N N _2 ) (t em pl e_ N N _4 ,m us lim is m _N N _2 ) (d ri ve _N N _1 2, co ur t_ ga m e_ N N _1 ) (s lu gg er _N N _1 ,s po rt _N N _1 ) (a ss is t_ N N _2 ,h oc ke y_ N N _1 ) (s el l_ V B _1 ,o ffe ns e_ N N _3 ) (s el l_ V B _1 ,l aw _N N _1 ) (r el at io ns hi p_ N N _4 ,b io lo gy _N N _1 ) (s lu gg er _N N _1 ,s of tb al l_ ga m e_ N N _1 ) (s em iti c_ de ity _N N _1 ,m yt ho lo gy _N N _1 ) (a po st le _N N _3 ,w es te rn _c hu rc h_ N N _1 ) (d ri ve _N N _1 2, ba dm in to n_ N N _1 ) (n uc le ar _d et er re nc e_ N N _1 ,l aw _N N _1 ) (a ss is t_ N N _2 ,s po rt _N N _1 ) (r ec ep tio n_ N N _5 ,b as eb al l_ ga m e_ N N _1 ) (tr ot _V B _2 ,e qu es tr ia n_ sp or t_ N N _1 ) (p ho to sy nt he si s_ N N _1 ,c he m is tr y_ N N _1 ) (r na _N N _1 ,c he m is tr y_ N N _1 ) (is ol de _N N _1 ,p ar ab le _N N _1 ) (a ss is t_ N N _2 ,s oc ce r_ N N _1 ) (a ss is t_ N N _2 ,c ou rt _g am e_ N N _1 ) (a ss is t_ N N _2 ,f oo tb al l_ ga m e_ N N _1 )

Review summaries:
This paper proposes to provide a detailed study on the explainability of link prediction (LP) models by utilizing a recent interpretation of word embeddings. More specifically, the authors categorize the relations in KG into three categories (R, S, C) using the correlation between the semantic relation between two words and the geometric relationship between their embeddings. The authors utilize this categorization to provide a better understanding of LP models’ performance through several experiments.

The paper attempts to understand the latent structure underlying knowledge graph embedding methods. The work can be seen as an extension of understanding of PMI-based word embedding methods. They categorize knowledge graph relations into three categories based on their relation conditions: Relatedness (R), Specialisation (S), and Context-shift (C). For each category, they evaluate a representative of different types of knowledge graph embedding methods. Through results, they demonstrate that a model’s ability to represent a specific relation type depends on the limitations imposed by the model architecture with respect to satisfying the necessary relation conditions.