On Understanding Knowledge Graph Representation
Authors: Carl Allen*
Ivana Balazevic*
Timothy M Hospedales
Venue:  (2020)
Keywords: knowledge graphs
word embedding
representation learning

Abstract: Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred. To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping. Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained. We build on recent theoretical interpretation of word embeddings as a basis to consider an explicit structure for representations of relations between entities. For identifiable relation types, we are able to predict properties and justify the relative performance of leading knowledge graph representation methods, including their often overlooked ability to make independent predictions.

Under review as a conference paper at ICLR 2020
ON UNDERSTANDING KNOWLEDGE GRAPH REPRESENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many methods have been developed represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred. To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space fol- lowing celation-specific mapping. Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained. We build on recent theoretical interpretation of word embed- dings to derive an explicit structure for the representations of relations between entities. From this we explain properties and justify the relative performance of leading knowledge graph representation methods for identifiable relation types, including their often overlooked ability to make independent predictions.
INTRODUCTION
Knowledge graphs are large repositories of binary relations between words (or entities) in the form of fact triples (subject, relation, object). Many models have been developed for learning representations of entities and relations in knowledge graphs, such that known facts can be recalled and previously unknown facts can be inferred, a task known as link prediction. Recent link prediction models (e.g. Bordes et al. 2013 Trouillon et al., 2016 Balažević al., 2019b) learn entity representations, or embeddings, of far lower dimensionality than the number of entities, by capturing latent structure in the data. Relations are typically represented as a mapping from the embedding of subject entity to its related object entity embedding(s). Although the performance of knowledge graphlink prediction models has steadily improved for nearly decade, relatively little understood of the low-rank latent structure that underpins these models, which address in this work.
We start by drawing a parallel between entity embeddings in knowledge graphs and unsupervised word embeddings, as learned by algorithms such as Word2Vec (W2V) (Mikolov et al. 2013 and GloVe (Pennington et al. 2014). We assume that words have latent features, e.g. meaning(s), tense, grammatical type, that innate and fixed, irrespective of what an embedding may capture (which may be only part, subject to the embedding method and/or the data source); and that this same latent structure gives rise to patterns observed in the data, e.g. in word co-occurrence statistics and in which words are related to which. As such, an understanding of the latent structure from one embedding task (e.g. word embedding) might be useful to another (c.g. knowledge graph entity embedding).
Recent work (Allen & Hospedales 2019; Allen et al. 2019) theoretically explains how semantic properties are encoded in word embeddings that (approximately) factorise matrix of word co- occurrence pointwise mutual information (PMI), e.g. as is known for W2V (Levy & Goldberg 2014). Semantic relationships between words (specifically similarity, relatedness, paraphrase and analogy) proven to manifest as linear relationships between rows of the PMI matrix (subject to known error terms), of which word embeddings can be considered low-rank projections. This explains why similar words (c.g. synonyms) have similar embeddings; and embeddings of analogous word pairs share a common "vector offset""
Importantly, this insight allows us to identify geometric relationships between such word embeddings necessary for other semantic relations to hold, such as those knowledge graphs. These relation conditions describe relation-specing mappings between entity embeddings, i.e. relation representa- tions, providing a "blue- "blue-print" against which to consider knowledge graph representation models. We find that various properties of knowledge graph representation models, including the relative

---

Under review as a conference paper at ICLR 2020
Table Score functions of representative linear link prediction models. E x and r E R are relation matrix and translation vector, W E Ie xd, xde is the core tensor and b,, bu E B are the entity biases.
Model
Linear Subcategory
Score Function
TransE:
(Brdes et al.
2013
additive
lle, -r- coll e, Re
DistMult
Yang et al.
2013
multiplicative (diagonal)
TuckER
Balzevic ei al.
2019h
multiplicative
X1 6, X2 X3 60
MuRE
Balazević al
2019a
multiphiatiive (diagonal) + additive
Re, co 18t bs
performance of leading link prediction models, accord with predictions based these relation conditions, suggesting a commonality to the latent structure learned in word embedding models and knowledge graph representation models, despite the signifieant differences between their training data and methodology. In summary, the key contributions of this work are:
to use recent understanding of PMI-based word embeddings to derive what relation represen- tation must achieve to map subject word embedding all related object word embeddings (relation conditions), based on which relations can be categorised into three types;
to show that properties of knowledge graph models fit predictions made from relation conditions, e.g. strength of relation's relatedness aspect reflected the eigenvalues of its relation matrix;
to show that the performance per relation of leading link prediction models corresponds to the ability of the model's architecture meet the relation conditions of the relation's type, i.e. the better the architecture of knowledge graph representation model aligns with the form theoretically derived for PMI-based word embeddings, the better the model performs; and noting how ranking metrics can be flawed, to provide novel insight into the prediction accuracy per relation of recent knowledge graph models, an evaluation metric we recommend in future.
2
BACKGROUND
Our work draws on knowledge graph representation and word embedding. Whilst related, these tasks differ materially in their training data. The former is restricted to datasets crafted by hand or automatically generated, the latter has the vast abundance of natural language text (e.g. Wikipedia).
2.1 KNOWLEDGE GRAPH REPRESENTATION
Almost all recent knowledge graph models represent entities Cz, c, as vectors e., e, 1d of low dimension (e.g. de 200) relative to the number of entities ne (typically of order 10*), and relations as transformations in the latent space from subject entity embedding to object. These models are distinguished by their score function, which defines (i) the form of the relation transformation, e.g. matrix multiplication, vector addition; and (ii) how "closeness" between the transformed subject embedding and object embedding is evaluated, e.g. dot product, Euclidean distance. Score functions can be non-linear (c.g. Dettmers et al.) (2018)). or linear and sub-categorised as additive, multiplicative both. We focus linear models due to their simplicity and strong performance link prediction (including state-of-the-artt.. Table T shows the score functions of competitive linear models that span all linear cub-categories: TransE (Bordes al. (013), DistMult (Yang et al.) 2015).
TuckER (Balažević et af. 2019b) and MuRE (Balažević et al. 2019a).
Additive models typically Euclidean distance and contain relation -ppccfff translation from (possibly transformed) subject to a (possibly transformed) object entity embedding. A generic additive score function is given by ó(e,, r, e.) R. +r R., || b, +b, = llee -r-c." (" 111 +b,+ +b,- The simplest example is TransE for which R, = R, - I and b, = b, =0. The score function of MuRE has R, I and combines multiplicative (R,- R.) and additive (r) components.
Multiplicative models have the generic score function ó(cs, e.) e._ Re, = (e!") eo). i.c. a bilinear product of the entity embeddings and a relation-specific matrix R. DistMult is a simple example with diagonal R and so cannot model asymmetric relations (Trouillon et al. 2016). In TuckER, each relation-specifie R = W r is linear combination of d, "prototype" relation matrices in a core tensor e dd, (where X,, denotes tensor product along mode n)) facilitating multi-tak learning across relations.
2

---

Under review as a conference paper at ICLR 2020
2.2 WORD EMBEDDING
Algorithms such as Verd2Vec (Mikolov et al. 2013) and GloVe (Pennington et al. (0144) generate succinet low-rank word embeddings that perform well on downstream tasks (Baroni et al. 2014).
Such models predict the context words (cz) observed around each target word (w;) in text corpus
using shallow neural networks. Whilst recent language models (c.g. Devlin et al. 82; Peters et al. (2018) create impressive context- -peciicc word embeddings, we focus the former embeddings since knowledge graph entities have no obvious context and, more importantly, they are interpretable.
Levy & Goldberg (2014) show that, for dictionary of unique words ID and embedding dimension d [D],, W2V's loss function is minimised when its weight matrices W, C' (Rdx[D] (whose columns are word embeddings w,, Gÿ) factorise word co-occurrence pointwise mutual information (PMI)
matrix, subject to shift term (PMI( We, Cÿ) =log P(w).c;)
P(er))P(es)> ). This connects W2V to earlier count- based embeddings and specifically to PMI, which has long history in linguistic analysis (Tunnyy &
Pantel 2010). From its loss function, Glo can be seen to perform a comparable factorisation.
Recent work shows why word embeddings that factorise such PMI matrix encode semantic word relationships (Allen & Hospedales] 2019; Allen et al. (019). The authors show that word embeddings can be seen as low-rank projections of high dimensional PMI vectors (rows of the PMI matrix), between which the semantic relationships of similarity, relatedness, paraphrase and analogy provably manifest as linear geometric relationships (subject to defined error terms), which are then preserved, under sufficiently linear projection, between word embeddings. Thus similar words have similar embeddings, and the embeddings of analogous word pairs share a common vector offset.
Specifically, the PMI vectors (P₂) of an analogy "man is to king as woman is to queen" satisfy Po-Pw PP PM because the difference between words associated with king and man (e.g. reign, crown) mirrors that between queen and woman. This leads to common difference between their co-occurrence distributions (over all words), giving a common difference between their PMI vectors, which projects common difference between embeddings. Any discrepancy in the mirroring of word associations is shown to introduce error, weakening the analogy, as does lack of statistical independence within certain word pairs (see (Allen & Hospedales) 2 The common difference in word co-occurrence distributions, e.g. the increased association with words (reign, crown, etc. ). can be interpreted semantically as a common change in context (context-shift) that transforms man to king and woman to queen by adding a royal context. Under this interpretation, context can also be subtracted, e-g. "king is to man as queen is to woman" (minus royal); or both, e.g. "boy is to king as girl is to queen" (minus youth plus royal). Adding context can also be interpreted as specialisation, and subtracting context as generalisation. This establishes a correspondence between common word embedding vector offsets and semantic context-shifts.
Although the projection from PMI vectors to word embeddings preserves the relative relationships, and thus the above semantic interpretability of common embedding differences, a direct interpretation of dimensions themselves is obscured, not least because any embedding matrix can be arbitrarily scaled/rotated if the other is inversely transformed.
3 RELATIONSHIPS BETWEEN EMBEDDINGS OF RELATED WORDS
Our aim is to build on the understanding of based word embeddings (henceforth word embed- dings), to identify what knowledge graph relation representation needs to achieve to map all subject word embeddings to all related object word embeddings. We note that if semantic relation between two words implies a particular geometric relationship between their embeddings, then the latter serves as necessary quantitative condition for the former to hold relation condition). Relation conditions implicitly define a relation-speciife mapping by which all subject embeddings are mapped to all related object embedding(s), allowing related entities to be identified by a proximity measure (e.g. Euclidean distance or dot product). Since this is the approach of many knowledge graph representa- tion models, their performance can be contrasted with their ability to express mappings that satisfy required relation conditions. For example, similarity and context-shift relations respectively imply closeness and a relation-specle specific vector offset between embeddings ($2.2). Such relation conditions can be tested for by respectively making change to the subject entity or adding the clattin--peccfii offset, before measuring proximity with the object. We note that since relation conditions are not necessarily sufficient, they do not guarantee relation holds, i.e. false positives may arise.
3

---

Under review as a conference paper at ICLR 2020
(a) Similarity
(b) Relatedness
(c) Specialisation
(d) Context-shift
(e) Gen. context-shift
Figure 1: Relationships between PMI vectors of entities under different relation types. Shaded regions indicate
strong word associations (positive PMI values); red lines indicate relatedness; black lines denote context sets.
In general. the data from which knowledge graph embeddings are derived differs greatly to the co-occurrence data used for word embeddings, and the latter would not be anticipated to be learned by knowledge graph models. However, word embeddings provide a known solution (i.e. minimise the loss function) of any knowledge graph model able express the required mapping(s) derived from relation conditions, where the loss function measures proximity between mapped entities.
The relation conditions for certain relation types (underlined) follow readily from $2.2
Similarity: Semantically similar words induce similar distributions over the words they co-occur with. Thus their PMI vectors (Fig [a) and word embeddings are similar.
Relatedness: The relatedness of two words can be defined in terms of the words with which both co-occur similarly (S ID), which define the nature of relatedness, e.g. milk and cheese are related by S = {dairy, breakfast, I..;: and [S] reflects the strength of relatedness. Since PMI vector components corresponding to S are similar (Fig lb), embeddings of "S-related"" words have similar components in the subspace V/ that spans the projected PMI vector dimensions corresponding to S. The rank of V3 might be expected to reflect relatedness strength. In general, relatedness is a weaker, more variable relation than similarity, its limiting case with = and rank(V/))
Context-shift: In the context of word embeddings, analogy typically refers to relational similarity Turney 2006 Gladkova et al., 2016). More specifically, the relations within analogies that give common vector offset between word embeddings require related words to have a common difference between their distributions of co-occurring words, defined as context-shifts (see $2.2). These relations are strictly 1-to-1 and include an aspect of relatedness due to the word associations in common (Fig Tdp. A specialisation relation is context-shift in which context is only added (Fig
Generalised context-shift: Context-shift relations are generalised to I-to-many, many-to-1 and many-to-many relations by letting the fully-specified added or subtracted context be one of (relation- specific) context set (Fig [cp. c.g. allowing an entity to be any colour or anything blue. The potential scope and size of each context set means these relations can vary greatly. The limiting case for small context sets has a single context in each, whereby the relation is an explicit context-shift (as above), and the difference between embeddings is a known vector offset. In the limiting case where context sets are large, the added/subtracted context is so loosely defined that, in effect, only the relatedness aspect of the relation and thus only the common subspace component of embeddings known.
Link to set theory: Viewing PMI vectors as sets of word associations and taking intuition from [ the above relations can be seen to reflect set operations: similarity as set equality; relatedness as equality of subset; and context-shift as the set difference equalling a relation- speciffc set. This highlights how the relatedness aspect of a relation reflects features that must be common, and context- shift reflects features that must differ. Whilst this mirrors an intuitive notion of "feature vectors", we emphasise that this is grounded in the co-occurrence statistics of PMI-based word embeddings.
3.1 CATEGORISING KNOWLEDGE GRAPH RELATIONS
Analysing the relations of popular knowledge graph datasets with the above perspective, we find that they comprise (i) relatedness aspect that reflects common theme (e.g. both entities are animals or geographic terms); and (ii) specific word associations of the subject and/or object entities. Specifically, relations appear to fall under hierarchy of three relation types: highly related (R); (generalised) specialisation (S); and (generalised) context-shifi (C). As above, "generalised" indicates that any added/subtracted contexts can be from set. From Fig I type R relations can be special case of S. which, in turn, is a special case of C. Type C is therefore a generalised case of all considered relations. Whilst there are several other ways classify relations (e.g. by their hierarchy,
4

---

Under review as a conference paper at ICLR 2020
Table 2: Categorisation of WNISRR relations.
Type
Relation
Exumples (subject entity, object entity)
verb_ weapanallllyeeeeeedeeeee
group
(trim down__ VB cut VB 35). (hatch VB incubate_ VB 2)
R
(lodge_ Birre_VB____ 4, Decommoddtionn_Ni 4). [question_NN___ inquire_VB VB_I}
VB_4,
also_sec
(clean_JJ_1 JJ_1, tidy_JJ_11 JJ_ (ram__ VB 2. screw_VB_L
S
hypermym
Semnd_refom__iV reform_NN Bapmm_NN___ 1), (prickle-weed NN_ herbaceous plant_NN NN_I]
instance hyperryym
(yellowstone yiver__evy ___I, Fiver_NN__e 1). (leipzig NN J. wawaauuabauattteeeeeeeee center NN__)
member_o of domain usage
(colloquialism NN_ figure VB 5). (ppurall form_oooooia.. _N_ Outhority_NN.2.
member_ of domain region member member_meronym has__ part
(rome (rome_NN B_te__N__L J. gladiator _N WW_ (usa, NN I. multiple voting _oting_NN_II NN_I)
C
(south_ NN 2. sunsline state_ Stat_NNV__
(genus, carya NN pecon pree _e__N BN____N_33,____ IN_ (morocco LorroccoNN_____ J. atlas mountains NN_I)
(aircraft _N_,, cabin_NN
synset domain _opiccttooppoo__p
(quark_ NN physics Mhyysiss_NN_LL__ (harmonize VB 3, music musie_NN_
transitivity), by considering relation conditions, we delineate by the required mathematical form (and complexity) of their representation. Table shows a categorisation of the relations of the WNI8RR dataset (Dettmers et al. 2018), 8), containing 11 relations between 40,943 entities | An explanation for A
this assignment is in Appx. A and that for NELL-995 [Xiong et al. 2017) is in Appx. B A review of the 5k-237 dataset (Toutanova et al., 2015) shows the vast majority of relations to be of type C
preventing contrast between relation types being drawn, hence we do not consider that dataset.
3.2 RELATIONS AS MAPPINGS OF EMBEDDINGS
Given the relation conditions of a particular relation type, we can recognise mappings that meet them and thus loss functions (that evaluate the proximity of mapped entity embeddings by dot product or Euclidean distance) able to identify relations of that type between PMI-based word embeddings. We then contrast these theoretically inspired loss functions (one per relation type) with those of knowledge graph models (Table I and, on the outline assumption that common low-rank latent structure is exploited by both word embeddings and knowledge graph models, predict properties and the relative performance of different knowledge graphs models for different relation types.
R: To evidence S-relatedness, both entity embeddings es: C. must be projected onto a subspace Vs. where their images are compared. Projection requires multiplication by a matrix P. Rdxd
and cannot be achieved additively, except in the limiting case of similarity, when P. I or vector r se 0 is added. Comparison by dot product gives (P,e,) (Pre,) = e, P. P.e. = e. M, (for a relation-specific symmetric M, = P.P P.). Euclidean distance gives P.e. P, eell" (c,-co) M, c.) = |P.c,ll2 2e, M,C. + |PP.c.l2
S/C: Evidencing these relations requires a test both for S-relatedness and for cuatiion-enttty-ppeeii embeddings component(s) (uf,, wf).. This can be achieved by (i) multiplying both entity embeddings by a relation-specifie projection matrix P. that projects onto the subspace that spans the low-rank projection of dimensions corresponding to S, of and of, (which tests for S-relatedness whilst preserving any entity-specific embedding components); and (ii) adding a relation-specific vector of e. to the transformed subject entity embeddings. Comparison of the final transformed entity embeddings by dot product equates to (P,e, + +r) TPPea: and by Fuclidean distance to [PPee, P. 12 P,ee, + 11 2(Pree, +r) P.e. |PP.eoll (ef MuRE: Re, +r- e, 13.
Contrasting the above loss functions with those of knowledge graph models (Table we make the following predictions: (P1) the ability learn the representation of a particular relation is expected to reflect the complexity of its type (R>S>C), and whether all relation conditions (e.g. additive or multiplicative interactions) can be met under given model; (P2) relation matrices for relatedness (type R) relations are highly symmetrie; (P3) offset vectors for relatedness relations have low norm; and (P4) as proxy to the rank of Vs. the eigenvalues of a relation matrix reflect a relation's 's strength of relatedness. To elaborate: P1 anticipates that additive-only models (e.g. TransE)) are not suited to identifying the relatedness aspect of relations (except in limiting cases of similarity, requiring zero vector); and multiplicative-only models (e.g. DistMult) should perform well type R but are not suited to identifying entity-specific features of type S/C, for which an asymmetric relation matrix in TuckER may help compensate. Further, the loss function of MuRE closely resembles that derived for type C relations (which generalise all others) and is thus expected to perform best overall.
We omit the relation "similar to" since its instances have discernible structure, and only 3 occur in the set, all of which are the inverse of training example and trivial to predict.
5

---

Under review as a conference paper at ICLR 2020
Table 3: Hits @ 10 per relation WN18RR.
Relation Name
Type
Khs
Max/Avg Path
TransF:
MuRE,
DistMult
TuckER
MuRE
verb group
R
1%
39
0.00
1.0
0.87
0.95
0.97
0.97
0.97
derivationally fellated_oom form
R
34%
1074
0.04
1.0
0.93
0.96
0.96
0.96
also see
R
2%
56
0.24
44
15.2
0.59
0.73
0.67
0.72
0.73
instance_ hypemyym
S
4%
122
1.00
1.0
0.22
0.52
0.47
0.53
0.54
synset domain topic_ol
C
4%
0.99
0.19
0.43
0.42
member_ov of domain usage
C
1%
24
1.00
1.0
0.42
0.42
0.48
0.38
0.50
member. Lemainn
C
1%
26
1.00
1.0
0.35
0.40
0.40
member, meronym has hart hypernym
C
8%
253
1,00
10
3.9
0.04
0.38
0.30
0.39
0.39
C
6%
172
1.00
13
2.2
0.04
0.31
0.28
0.29
0.35
0.20
all
100%
3134
0.38
0.52
0.51
0.53
0.57
Table 4: Hits@ 10 per relation on NELL-9955.
Relation Name
Type
%
Khs
Max/Avg Path
TransE
MaRE,
DistMult
TuckER
MuRE
leamplaysgainssteemm
clothingrogowithcleehh
R
1%
132
0.17
5
2.6
0.72
0.80
0.88
0.85
0.84
professionisty
7
0.37
animalistyeoffnnimall
S
1%
103
0.68
9
3.1
0.50
0.56
0,64
0.68
0.65
uthletoplaysee ssport
C
1%
113
1.00
1
0.54
0,58
0.60
chemicalistypeofcheee
S
1%
115
0.53
6
3.0
0.23
0.43
0.49
0.51
0.60
itemfoundirroo
C
2%
162
1.00
1
1.0
0.39
0.57
0.53
0.56
0.59
genncolaabouttesviuueen baddpaancoonaiinbboyyaar
R
1%
119
0.51
0.44
0.58
0.64
0.61
0.58
C
1%
103
0.60
7
3.2
0.30
0.38
0.54
0.58
0.55
atdate
10%
967
0.99
4
0.41
0.50
0.48
0.48
0.52
bocaioolocaccdiiiiillcciioo
C
2%
157
1.00
6
1.9
0.35
0.37
0.46
0.48
0.48
atlocation
0.22
100%
20000
0.36
0.48
0.51
0,52
0.52
COMPARING KNOWLEDGE GRAPH MODELS
We test the predictions made on the basis of word embeddings by comparing the performance of competitive knowledge graph models, TransE, DistMult, TuckER and MuRE (see $2). which entail different forms of relation representation, 995 relations (selected to represent each relation type). Since applying the logistic sigmoid to the score function of TransE does give probabilistic interpretation comparable other models, we include MuRE,, a constrained variant of MuRE with R, R. I, as proxy to TransE for fairer comparison. Implementation details are included Appx. D For evaluation, we generate 2n, evaluation triples for each test triple (for the number of entities n.)) by fixing the subject entity e, and relation and replacing the object entity e, with all possible entities and then keeping e, and fixed and varying ex. The obtained scores are ranked to give the standard metric hits@ 10 (Bordes) al. 2013), i.e. the fraction of times a true triple appears in the top 10 ranked evaluation triples.
all WNI8RR relations and similar number of NELL-
4.1 PERFORMANCE PER RELATION TYPE
Tables and report results (hits @ 10) for each relation and include the relation type and known confounding influences: percentage relation instances in the training and test sets (approximately equal), number of instances in the test set, Krackhardt hierarchy score (see Appx. (Krackhardt)
2014 Balazević et al.) 2019a) and maximum and average shortest path between any two related nodes. A further confounding effect is dependence between relations. [alažević al. (2019b) and
Lacroix et al. (2018) show that constraining the rank of relation representations benefits datasets
with many relations (particularly when the number of instances per relation low) due multi- task learning, which is expected to benefit TuckER on the NELL--95 dataset (200 relations). Note that all models have a comparable number free parameters.
P1: As predicted, all models tend to perform best at type R relations, with a clear performance gap to other relation types. Also, performance on type S relations appears higher general than type C. Additive-only models (TransE, MuRE,) perform most poorly on average, in line with prediction since all relation types involve a relatedness component. They achieve their best results on type R relations, where the relation vector can be zero/small. Multiplicative-nnl DistMult performs well, sometimes best, on type R relations, fitting expectation as it can fully represent those relations and has additional parameters that may overfit noise (which may explain where MuRE performs

---

Under review as a conference paper at ICLR 2020
slightly worse). As expected, MuRE, performs best average (particularly on WN18RR), and most strongly on S and C type relations that require both multiplicatiie and additive components. The comparable performance of TuckER NELL-995 is explained by its ability for multi-task learning.
Other unexpected results also closely align with confounding factors, c.g. that all models perform poorly on the hypernym relation, despite it having type S and a relative abundance of training data (40% of all instances), might be explained by its hierarchical nature (Khs = and long paths). The same may type relations also_see and agentcollaborruessihhaeenn As
explain the reduced performance found previously, none of the models considered are well suited to modelling hierarchical structures (Balažević et al. 2019a). We also note that the percentage of training instances of a relation does not seem to correlate with its performance, as might typically be expected.
P2/P3: Table 5 shows the symmetry score (E [-1, 1] indicating perfect anti-symmetry to symmetry; see Appx. for the relation matrix of TuckER and the norm of relation vectors of TransE, MuRE, and MuRE on the WNISRR dataset. As expected, type relations have high symmetry, whereas both other relation types have lower scores, fitting the expectation that TuckER compensates for having no additive component. All additive models learn relation vectors of noticeably lower norm for type relations (where, in the extreme, additive component is required) than for types S and C.
Table 5: Relation matrix symmetry score [-1.1] for TuckER; and relation vector norm for TransE, MuRE, and MuRE (WN18RR).
1.0
Symmetry Score TuckER
Vector Norm MuRE,
Relation
Type
TransE
MuRE
06
verb, group
R
0.52
5.65
0.76
0.89
derivationally peraainnllyyrellteedd.oooee
2.98
also_scc
R
0,50
7.20
0.97
0.97
instance_ instance_hypernyy hypernym
S
0.13
18.26
2.98
1.88
member of domain, usage
C
0.10
11.24
3,18
1.88
member_o of dommin region
C
0.06
12.52
3,07
2.11
100
150
synset synseL_domain topic_of member mmrrooym has, part hypemny
0.12
23.29
2.65
1,52
dimension
0.12
4.97
1.91
1.97
6,44
Eigenvalue magnitudes of relation-specific matrices R for MuRE by relation type (WN18RR).
Figure 2:
S
0.04
9.64
1.53
1.03
P4: Fig ssowss eigenvalue magnitudes (scaled relative to their largest and ordered) of the relation- specific matrices of MuRE, labelled by relation type. Predicted to reflect strength of a relation's relatedness, they should be highest for type R relations, as observed. For relation types S and C the profiles are more varied, fitting the expectation that the relatedness of those types has greater variability in both choice and size of S, i.e. in the nature and strength of relatedness.
In summary, the results support all predictions made based on the assumption that knowledge graph models benefit from the same latent semantic structure word embeddings and the relation conditions theoretically derived from them. Our analysis identifies the best performing model per relation type: multiplicative-only DistMult for type R, aaditive-multiplicatte basis for dataset-depender model selection. The per-relation insight into where models perform poorly, e.g. on hierarchical type C relations, can also be used to aid and direct future model design.
MuRE for types S/C; providing
4.2 KNOWLEDGE GRAPH MODEL PREDICTIONS
Even though in practice we want to know whether a particular triple is true or false, such independent predictions are not commonly reported or evaluated. Despite many recent link prediction models being able to independently predict the truth of each triple, it is common practice to report ranking- based metrics, e.g. mean reciprocal rank, hits which compare the prediction of a test triple to those of all evaluation triples (see $4) Not only is this computationally costly, the evaluation is flawed entities are related to more than others and does not evaluate model's ability to independently predict whether "a is related to b". We address this by considering actual model predictions.
Since for each relation there are nı" possible entity-entity relationships, we sub-sample by computing predictions for all (e,, T.) triples only for each es, pair seen in the test set. We split positive predictions (o(o(e,, eo)) > 0.5) between (i) training and test/validation instances (known truths); and (ii) other, the truth of which is not known. Per relation, then compute accuracy over the training instances (train) and the test/validation instances (test); and the average number other
truths predicted per es; pair. Table shows results for MuRE,, DistMult, TuckER and MuRE. All

---

Under review as a conference paper at ICLR 2020
Table 6: Per relation prediction accuracy for MuRE, (M,). (D)istMult, (T)uckER and (M)RE (WNI8RR).
Accuracy (train)
Accuracy (test)
Other "True"
Relation Name
#ensis
#Frut
M,
D
T
M
M,
D
T
M
M₁
D
T
M
verb_groop
15
1.00
1.000
1.00
1.00
0.97
0.97
0.97
0.97
8.3
1.7
0.9
2.7
derivatiionall__elaaee form
1714
1127
1.00
1.00
1.00
1.00
0.96
0.94
0.95
0.95
8.8
0.5
0.6
1.7
also_scc
95
61
1.00
1.00
1.00
1.00
0.64
0.64
0.61
0.59
7.9
1.6
0.9
1.9
instance_ hypernym
instance_hypermyn
122
1.00
1.00
1.00
1.00
0.32
0.32
0.23
0.43
1.3
0.3
0.9
member wwwr__o_ddmmmimnoeeeee
545
43
0.98
1.00
1.00
1.00
0.02
0.00
0.02
0.00
1.5
0.6
0.0
0.3
member of domain nomain_regioon
543
42
0.88
0.89
1.00
0.93
0.02
0.02
0.00
0.02
1.0
0.4
0.8
0.7
synset_ doaset_domain_aee coomain_topic_oo
115
1.00
1.00
1.00
1.00
0.42
0.10
0.14
0.47
0.1
0.2
nember__ernny mermnym
1402
307
1.00
1.00
1.00
1.00
0.22
0.02
0.01
0.22
7.9
34
1.5
5.6
part
848
196
1.00
1.00
1.00
1.00
0.24
0.05
0.09
0.22
2.4
1.3
3.9
hypemym
57
1254
1.00
1.00
1.00
1.00
0.15
0.02
0.02
0.22
3.7
1.2
0.0
1.7
all
5284
3306
0.99
0.99
1.00
0.99
0.47
0.37
0.37
0.50
5.9
1.2
0.5
2.1
750
40
40-
8c0
other
70
train test
other test
5c0
test
00
420
3c0
2:0
1:0
10
82
80
83
(a) Derivationalll_relatee ffrrmddfffooooooooo (R)
(b) mistance_hypemyn (S)
(c) pannet_domain_togg catomnaintttic_ottt (C)
Figure 3: Histograms of MuRE predictions for an example WNISRR relation of each type, split into true training, true test/validation and other instances.
models achieve almost perfect training accuracy. The cultiiie-m MuRE gives best test set performance, followed (surprisingly) closely by MuRE,, with multiplicative models (DistMult and TuckER) performing poorly on all but type R relations. Analysing a sample of "other" positive predictions for a relation of each type (see Appx. GG. we estimate that TuckER is relatively accurate but pessimistic (~00.3 correct of the 0.5 predictions 60%), MuRE, is optimistic but inaccurate (~2.3 of 7.5 1)), whereas MuRE is both optimistic and accurate (~~1.1 of 1.5 73%).
Fig shows histograms of MuRE prediction probabilities for the same sample relations, split by known truths (training and test/validation) and other instances. There is clear distinction between relation types: for type R, most train and test triples are classified correctly with high confidence; for types S and C, an increasing majority of incorrect test predictions are far below the decision boundary, ie. the model is confidently incorrect. For relation types where the model less accurate, fewer positive predictions are made overall and the prediction distribution is more peaked towards zero.
This analysis probes further into the difficulty models have representing type S/C relations. Given the additional insight provided and the benefit of standalone predictions, we recommend the inclusion of predictive performance in future link prediction work.
CONCLUSION
Many models leam low-rank representations for knowledge graph link prediction, yet little is known about the latent structure they learn. We build on recent understanding of PMI-based word embeddings theoretically establish what relation representation must achieve to map word embedding to those it is related to for the relations of knowledge graphs (relation conditions). Such conditions partition relations into three types and also provide framework to assess loss functions of knowledge graph models. Any model that satisfies a relation's 's conditions can represent it if its entity embeddings are set to PMI-based word embeddings, i.e. a solution is known to exist. Whilst knowledge graph models do not learn the parameters of word embeddings, we show that the better model's architecture satisfies a relation's 's conditions, the better its link prediction performance, fitting the premise that similar latent structure is exploited. Overall, we extend previous understanding of how semantic relations are encoded in relationships between PMI-based word embeddings generalising from a limited set, e.g. similarity and analogy; demonstrate commonality between the latent structure learned by PMI-based word embeddings (c.g. W2V) and knowledge graph representation models; and we provide novel insight into knowledge graph models by evaluating their predictive performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Carl Allen and Timothy Hospedales. Analogies Explained: Towards Understanding Word Embed- dings. In International Conference on Machine Learning, 2019.
Carl Allen, Ivana Balažević, and Timothy Hospedales. What the Vec? Towards Probabilistically Grounded Embeddings. In Advances in Neural Information Processing Systems, 2019.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. Multi-relational Poincaré Graph Embed- dings. In Advances in Neural Information Processing Systems, 2019a.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. TuckER:
Tensor Factorization for Knowl- edge Graph Completion. In Empirical Methods in Natural Language Processing, 2019b.
Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't Count, Predict! Systematic Comparison of Context-Counting Context-Prediiting Semantic Vectors. In Association for Computational Linguistics, 2014.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. Advances in Neural Information Processing Systems, 2013.
Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam Hruschka, and Tom M Mitchell. Toward an Architecture for Never-ending Language Learning. In Association for the Advancement of Artificial Intelligence, 2010.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D Knowledge Graph Embeddings. In Association for the Advancement of Artificial Intelligence, 2018.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics, 2018
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based Detection of Morphological and Semantic Relations With Word Embeddings: What Works and What Doesn't. In NAACL Student Research Workshop, 2016.
Lawrence Hubert and Frank B Baker. Evaluating the Symmetry of Proximity Matrix. Quality & Quantity, 13(1):77-84, 1979.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
David Krackhardt. Graph Theoretical Dimensions of Informal Organizations. In Computational Organization Theory. Psychology Press, 2014.
Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical Tensor Decomposition for
Knowledge Base Completion. In International Conference on Machine Learning, 2018.
Omer Levy and Yoav Goldberg. Neural Word embedding as Implicit Matrix Factorization. In Advances in Neural Information Processing Systems, 2014.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality, In Advances Neural Information Processing Systems, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In North American Chapter of the Association for Computational Linguistics, 2018.

---

Under review as a conference paper at ICLR 2020
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing Text for Joint Embedding of Text and Knowledge Bases. In Empirical Methods in Natural Language Processing, 2015.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex Embeddings for Simple Link Prediction. In International Conference Machine Learning, 2016.
Peter D Turney. Similarity of Semantic Relations. Computational Linguistics, 32(3):379-416, 2006.
Peter D Tumey and Patrick Pantel. From Frequency Mcaning: Vector Space Models of Semantics.
Journal of Artificial Intelligence Research, 37:141-188, 2010.
Wenhan Xiong, Thien Hoang, and William Yang Wang. DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning, Empirical Methods in Natural Language Processing, 2017.
Bishan Yang. Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding Entities and Relations for Learning and Inference Knowledge Bases. In International Conference on Learning Representations, 2015.
10

---

Under review as a conference paper at ICLR 2020
A
CATEGORISING WORDNET RELATIONS
Table 7 describes how each WN18RR relation was assigned to its respective category.
Table 7: Explanation for the WN18RR relation category assignment.
Type
Relation
Relatedness
Subject Specifics
Object Specifies
verb_group
both verbs; similar in meaning
R
derivationally relatedd_fomm
different syntactic categories: semantically related semantically similar
also
S
hypernym in
semantically similar
instance of the object instance object
instance hypemym
semantically similar
member of domain usage member of rogiin
loosely semantically related (word usage features)
usage deseriptor
broad feature set broad feature set part of the subject part of the subject domain descriptor
loosely semantically related (rogional features)
rogion descriptor
C
member merouym
semantiallyy related
collection of objects collection of objects broad feature
has part synset_ doooddddmminoopp
semantically related
semantically related
B ANALYSING NELL-995 RELATIONS
NELL-999 9-95:: We categorise a random subsample of 12 relations from the NELL-995 dataset (Xiong et al. 2017) containing 75,492 entities and 200 relations (a subset of NELL (Carlson al. 2010)), which span our identified relation types (see Table 8 Explanation for the relation category assignment shown in Table g
Table Categorisation of NELL-995 relations.
Type
Relation
Examples (subject entiry, object entity)
tcamplay saamplaysgainstttaann
sagainsttcam clothingtogowithel oowoothhcloohhngg
(sportateam paattee, sportsteum porrsteam_marinerrs, moriners), (sportsteam (sportsteam_phhiliee,, phillies, pportsteamjammpee bay__byyyaeeee (ays) (clothing shirss, clothing trosser)) (clothing shoes, clathung black, shin) (musicartist, white whitt_trrppss,e, stripes, musicartist janccartist_jack... white), (politicion aapiiiiiin_phomma, phoma, politician hillicy
R
black,,
agentcollahoratess aaloaaaeswwthhgge ithugent
pooproasooooooooaaiiiioon unimalistypnn mrimalistppooaania
(profession Jrial Jawyers, profession attomnys), (profession pprofssion_engineers. profession _esperts)
S
(mammul cats, mammal small animals), (hird chickens, agriculunn/pradet livestook) (chemical moisture, chemical gar). (chemical oxide, chemical materials))
chemicalisty setnecalisttyefchmmieee.
alyyeeeplayspoot
(athlete joe smith, sport buseball), (athlete chris cooley, sport, foothall) (bedroomitem bed. room renj_ (hallwayttem refrigerator, visualizablething hitcieg___ttettteeeeeee jitchen_area) (bodypart systesssssssms braiatissue eyesi, (bodyport blood, bodypart Jeft yentricle)
meemooondinooo
bodypartonntainsboyyye
C
atdate
(currency scotland date ,2009), (goveromentorgn ganizzatinn wio, date ,2003) (city medelin,, country colombiaj, (city jackson, stateorprvvnnce wyoming) (city ogunquin, stateorprovince maine). (city_palmer Jake, butterppeoinceeceelooeee celorado)
atlocation
Table 9; Explanation for the NELL-995 relation category assignment.
Type
Relation
Relatedness
Subject Specifics
Object Specifics
ecamplaysagainsttee
both sport learns
R
clothingtogow anggwithhloohhnn
hoth items of clothing that together both people or companies; related industries
agentcollaben worattee withesettt
semantically related (both profession types) semantically related (both animuls)
instance of the object
S
animmaiittpeeoaaimaall
instance of the object instance of the object
chemicalisty peoccheect
semantically related (both chemicals)
athleteplaysspoet
semantically related (sports features)
athlete descriptor
sport descriptor room descriptor part of the subject date descriptor collection of objects collection of abjects
itenfoundinroom
semantically related (room/furniture features)
item descriptor
emantically related (specific body part features)
collection of objects
aldate
lousely semantically related (start date leatures)
broad feature set
semantically related (geographical features)
of the subject
atlocation
semantically related (geographical features))
part of the subject
C SPLITTING THE NELL-995 DATASET
The test set of NELL-995 created by Xiong et al. (017) contains only 10 out of 200 relations present in the training set. To ensure a fair representation of all training set relations in the validation and test sets, we create new validation and test set splits by combining the initial validation and test sets with the training set and randomly selecting 20,000 triples each from the combined dataset.
11

---

Under review as a conference paper at ICLR 2020
D IMPLEMENTATION DETAILS
All algorithms are re-implemented in PyTorch with the Adam optimizer (Kingma & Ba 2015) that minimises binary cross-entropy loss, using hyper-parameters that work well for all models (learning rate: 0.001, batch size: 128, number of negative samples: 50). Entity and relation embedding dimensionality is set dg d, 200 for all models except TuckER, for which 30 (Balažević et al. 2019b).
E KRACKHARDT HIERARCHY SCORE
The Krackhardt hierarchy score measures the proportion of node pairs (x, 3) where there exists directed path x y. but not y - æ; and it takes a value of one for all directed acyclic graphs, and zero for cycles and cliques (Krackhardi, 2014 Balažević et al. 2019a).
Let M RR* be the binary reachability matrix of directed graph with n nodes, with M.,, = there exists a directed path from node to node and 0 otherwise. The Krackhardt hierarchy score of G is defined as:
Σ Li=1
Σ j=1
1(M
== 1A M == 0)
Khsg
(I)
2i=1 1(M = 1)
Li=1
F SYMMETRY SCORE
The symmetry score ecoree[-1e 1
Hubert Baker 1979) for a relation matrix RERdd defined as:
ΣΣ R. R₁: E R.))
de(d.
(2)
(E R.
ΣE R2j d,(de
where indicates a symmetric and -1 an anti-symmetric matrix.
G "OTHER" PREDICTED FACTS
Tables 10 to 13 shows sample of the unknown triples (i.e. those formed using the WN18RR entities and relations, but not present in the dataset) for the derivationally_ee frclttedeeeeeddd form (R), instance hhypeeeyyypmmmmmm (S) and synset_ domain coppniin_tppi____ of (C) relations at range of probabilities (a ((((e,, {0.4, 0.6, 0.8, 1)), as predicted by each model. True triples are indicated in bold; instances where a model predicts an entity is related to itself are indicated in blue.
12

---

Under review as a conference paper at ICLR 2020
Iikamas aquines NN. J. church futher NN 1) (wondy gathii
foratral voeatrl on NN J.C. an mation NN (J) (argean NN J. island NN I)
tworship VB. J. worship. VB. I)
(sort out VB. I..I. sort out VB JI
Imake Juil JMi_BB VB. J. make full V8
Igeology_ _AN I. geology NN 1)
Eoooiog _N 2 woology NN
tunnology NW J. aranolog NN. I)
travel VB. [._. travel VB. VB_J) J]
(helsinki J. J.rronn sonerr NN I))
(manbeein NY J. stncaade NN 2)
(nippon NY nippon AN 11
fuctor hudtor hugo NN 1. novel NN 1)
legioomnntunn monturu NN urunology NN. I)
(prophes NN J, 1.book, NN NV_ 10)
m.insteed, VV. 1)
dile NN I. disperagemntt NN I)
(roll-on roll-aff NN J. tronsport LN J)
(prance 4, equeittion sport, NN
dibee J. trrouucement_ NN_
foell Vn. 1, i,llling NN 1)
(ire, 28.2.id ride horsebuc Bepehook__...... VBB.)
(pronce VR. 4. ride borsebick VR ()
(gallop V/0 1. ride Jonchas
drywhild_MM NN W.J, mytholer BooooooNN"
NN 22
(drine NN 12, baddinton NN J)
VB. 2. vn 2)
VB J]
J. unitee VR
af (o [s)
VB
junlice
(mallaa x NN I passne renister NN !)
(contal omerten NN J. om. Jusiion NN
yertile crooeet NN 1. t.aim nation NN I)
(onviz NN 2, mm. foothal boothllggameee. gome NN 1)
(omugode NN drow close VB 3)
(seratch VR. V8-3, skin senvation NN D (acheme
S, schomatiation NN MN
(raiwan J. matiioaal capital [N_
(hommr aquinar NN J. saint VB 2)
gregorr i NNN apontic NW 3)
(moeeri owen NN J. induery NN 1)
rellatonsiip NN d. relattorsiip NN 4)
(etmmatggiitgg NN I, lllggiisiis NN
(bryniildd NN. myyhhlopicc. VB. 2)
(ndicovoi NY 2. hummm acrvity NN 1i
(lard VB. J. vegelable oil NN 1)
(registe NN 3.fle Ailee_noe away VB II
(truth NN 5, abolitomism mm NN 1)
(furn, VB VB_I2, f2, culiivation NN 2)
(relationchpp AN 4, biology NN
(irain VB. 30.3. triining NN N. 1)
(prophets I. gospels NN
(de multe NN J. dance VB. 11
(Orzahild L NN J. myth myth_NN IN
[srwer 2, mitehor NN
fordain VB, 3 vest 12 J;
AN 1)
(tread J. mep NW 9)
(driive Mivve_NN MN 12. drive V8 81
fiibel (iibeLN LN sully. snlly_VV VB
Table 10: "Other" facts predicted by MuRE
V8
trichand Nockmmnnter f. (_NN J. techmologion tool (1.2)
fit. gregory of n. AN 1. cononication NY 1)
thoomas aquinas NN 1. orciiiishop NN NN II
(ihomay the aponte AN J. nancelfy vB JI
Idrow VB 6. ereative aareaaieepeooee peronn NN_.) (suborider NN J. taxomomd group NN 1))
(extrapolare VD. 3, synthonis NN 3)
urrategitt NN J. mechinatoon NW I) ferush
(yyutheticat 11 I. synterizer AN 2)
rreaiicttoon NN 2, embeedimen JN 3)
(tabwnn NN J. aalan nation NN D)
gregury LNN J. t.bisoup NN 1)
(marsholl, NN 2. genoni offfeer NN
faramum NN 2. primoteship hip N) ()
(dros vn 6, J J. ballgame NN
Gaint VR 2, eatholic church NN I)
NN
Exprend, 1B 5. circularize VB 3)
(Houn NN showingg NN 2)
grind 21
rspile, VB 5 steady VB 2)
fliking NY vongguih V8 J)
Inccomulek NN IN fuul VB 8)
(atchemice Vo. J. modiff VR 31
fremmon Leemoo_NNLL I. sermon Nermoo_NN_ NN
(male. 1. musculine 11.2)
thee 3, zoolegy_NN, NN_2)
gregory nf NN.
(sell, VD sill VB. 11
oooe, c.)) =0 0.6
(extrapolation NN
Uirst (irit_ucker jucker AN
(frend of the the court NN J. war mucoiine NN II (mulii. anatyds NN J. applied math NN
fnnroveer AN J. achinifff dicoppine NN .1)
(de mille NY 1, Sancing-maatte NN 1)
(argean NN J. argean argeon__ssaam island AN .))
Davning NN J. putelfer refier, NN NNJI
fthomux agninos NN I. morryy 2) 2)
folcann idands NY 1. volcamo NN 2)
feape IN_ urbay conter NY [)
(begen NN J. national capizal NN 17
wutermmii NN 2. state rapital NN 1)
(tomple NN 4 place wansipp NN
coonstellationmo AN 2, satellite NN_5 3)
(ohrink VB subrrrccion NN 2)
(comminue 170 10, pooced VB..) I)
fatratin NW.2, 2, social social-graup group NY_J)
(marshhil NY gonerutdipp NN I)
(monsen NN I. venturer NN 2)
(prosia NN J. stockade 3N 2)
terymologizing puiiinn_NV NNL.. low low__N NN Ii
(trial Itria_inno iupressioon seissso NN_ J.L.Ippuot paiof VB
fsell VB J. trunswcttion aonntiinn_ NN I)
(equolizer NN 2, tot off Via
taugport VB 6. defend VB. 3)
(oluttr MN I.I.8. I.fil up 3)
VR J)
rdree V8_1 I. soufle 33
J, slar 2)
(dro. VB, & represent V0
(wrlie VB 8 cope aope_VB.
mil., (r)) ss 0.4
(yow NN
Relation (Type)
synse, donain
derivationall. relnnd form (R)
hypenym (5)
irstamce
topic
(C)
13

---

Under review as a conference paper at ICLR 2020
(irth. capinl LN J. urben senter AN (r. c. hynt NN 1. iimpector general NN I) fr. hynt NN J. chief of stuff AN D. icontrll icontrui_meen comerica
Baerria_N NN c. am. pation NNL maaloom x AN J. enviromenntio (th nacarene NN 1. christiim H.. I.
rfhomex riguinar NY 1. charch fucher. NN JI.
imutnis imatni__ogooma, algebna NW J.J. diugonddizziinn NN J)) fembinetwenk NN NN_2, woodwarking NN N.
NN_ !J
fetymelogiiing AN J. etymologice VB 2)
().2)
(alliterate VB. J. versiffer NN J]
(gcology NN J, structural J1S)
(2
fright iright fielier NN J. I.leffield NN I)
fleutemncy Maneey_-ii NN J. cappidie NN 1)
(helsighi NN J. arrhnn center NN
the juzzrene NN save sove_WW VW D)
NN
(fat JJ S, matie NN 2) (ctymologiingg NN dorive VB.
relation NV 1)
(drive NN 12, budmintton NN J]
foabinetwork, 2, botom, VB. 1)
frabinetwork AN 2, uphoitter VB. 1)
fnurtiion, NN 3. man AN 4)
sanctify VB. 1)
(exentenatio NN J. ennncletee
thole out VR. J. holeee NN_
(resect Vn J, aoputttion
(lit 4, slape NN 2)
(tread NN J. s,stp. (5
nucareme NN J. ..
trelationship NY
aidis (.
(saint
NN I. south. AN (copiaal of gombia NN J. urbbnn center (NL()
(north plane river NY J. urkan genter JN II
(oolo NY I, urbon contr NY I) Canire sivr NY J. urhan conter NN JI (munsyihanion plpp NN J. uurhon senter NN []
(brandentury NN 1. wubann center NW I)
(sternu, nemud NN J. wrbon, center AN 1)
(mmlcolmm_ NNN. umuncipationnll LN IN
letymologicing NN J. explamation NN I)
Mator__NN NN J. promecate VV_2) 2) (alchemice. VH I. modily VB
(print VB 4, publiching houre AN J)
(souti, pefiaafiic MV J. ofreca NN WN
(Biddina energy NN 1, chharge NN 9)
(centrulization NN J, unice VD. 6)
NV 1)
tring NN 4, chemical chain NN
(abriak NN J. pedology AN JI
(existence UN NN_ J. L.iinng NNN
(Baish VB f.iisiiing NN
Goly Vn 26, playing Sagiin-NN... NN-3) 3)
(mooth VB. 3, (consellaation sorsing NN 1)
(pucale VB. 2, sccret A.. 33
(aranology. NN J. n.. NNN _N
(fre VR. 3 J.famingg NN I)
Table 11: "Other" facts as predicted by DistMulit
afle,, (u)) 0.8
2,
(oum. VB. VB_12. 12. ail VB 11
(bii (Ni_8.. N881. hil VB 11
(ferry 3, irvel
(pubiic (oublic_ppoeee prowettrr
(Dikel, NN J. lihell
(deep
(misiisnigot river NN american state I) (r (ree._ bord NN J. (koheuham AN commissionne aasstoneddddi officer JN [I
(wrestle. VB 4, wrestler NN 1) (autorr. organism NN 1, epiphytii H
(r.s. byrd NN J. millaryy antvior NN
(r.s. byrd NN J. execuuive officer NN
L AN
AN II
(re mee-uu bun NN J. olde-de-canpp caee--copppp II (rampe boy NN J. urhan center NV
(ideewterr region AN 1. soath NN 1)
(rallonn rill- AN 1, munggort NN (hick nt sahrir
Glugzer NN J. seffball aaftbal_gomeee gome NN_I) (sermon__V) NV
(assivt NN_ 2. softball game .N. II) (council
(hroughput NV J. t,runnu Bernoo__NN... NV 4)
(yarom NW 9, andertienss NN D)
(the gambia NY 1. affica NN 1) (tinch mür NN uhhnn
asia II
No NN 2. assembll NN 4)
(expore, VB show NW JI
(spreuad NN 4. wteww VB. F)
(take down VB. 10.2.pm. pio VB. 2)
(duel VB 1, slugfat NY. 1)
(vocal 2. sarr NN 1)
(smelling. NN J. scent 1)
sermonize VR_)
(dow VB 6, drawer ddawer_NN- NY.3)
(cream VB 1, cream NN 2)
(tura VB 12, plow NN 1)
(dish_ NN 2. stew stew_NN NN 21
(ol (o)) R 0.6
(marshall BeosshllLNNNWN NN 2. Mleutemnnt Seant_ggeee general Leneral_NN NN J)
(world war NN 1. whan center NN 1) (Mt__
(Niino NV .,urhon contr NN 1) (nggoan, sea NW 1. negoan istend NN 11
(stg VB_5. 3, undercover work NN.J )
J)
d'woirr NN I]
(deep (desp_woult south swouh_AWW AW- I. urban arrar_ccmtte center_ (NL))
(mipie NN J. urbon center NN 1)
NNN J. proctice of lav law_NN NN J1
(slugger NN J. J. baschall player NN 1))
(rial, impression NN publish VB. 1)
Himutation AN 4. mammel VB.2 2)
(lght. (Ughtt colomel NN J. colonel NN
(print VB publisher NN 2N
(vriez NN 3. pizelman NN 2)
NN 10)
(count_ 48.4. reciee 28
(vividncss NN 2, lindue VB.3 3)
(ooa mew NN N. 1,la NW 1)
(albait NW_ 2. acitiiyy VB. 2)
(see Via understond VB 2)
(shan, V8 J. hedging NN 2)
(wibconsin NN 2. urbon sontr
NW 2. enungelie AN
(compemo NN_. wur of con
(murse VB J. nursing Nossin_NNN ()
(srmon_ MN__NN_ prophery, VR 2)
chemistry IN
(metrify _o__y__V_____ J. versfyy V8 11
(urm VB 12. plowman NN 1)
df F, n2)) 0.4
(obidjon NV. J.
(play Vn 26,
(rna Irma_N JV_., J.
Relation (Tppe)
synset domain lopic, of
darivationnlly. related Jomu
Instance
hypennym (8)
14

---

Under review as a conference paper at ICLR 2020
(argoon se0 NN J. Lsgeon joland NN JI (thomas aguinas MN church fother NN J)
J. phatemmmmpiic J)
ldynanitiir .N dynamitis NN _1
(position VB. 1, 1,placement NN B
AW women's righiigh NN
(remurer N. wenuuer NN
(amugele. NN J. sgnece VB 8)
(repentame I. I. 11.1)
(atokke NN 1. stocllt NN 11 (phitanthmpiir
(r. e bynd NN J. singe AN 1)
(low: 3, lover. NN
JNNJ.dde VB
(trend J. see. VB 7)
(deported
arns manufacturer NN 1I
(hooes the aponie NN J. enongelli NN 2)
(M. gregory iNN J. church father NV
(unccormick AY 1. techhologiit NN F)
(federate VB. 2, confederation NN. 1)
faaee chwooooooss...... VB. J. voutvee NN
(auporr VB 6, (.indoreeent NN I)
(donote VB 3, promol guttigu NW. I)
(wait on VB J. supporroo NN 3)
(brembher N. J. vibruie VB 1)
Make over 6, return (atuun__N_) NN 7)
[shuntr__ Moluter_AW... AN. J. lul up. VB 3)
(compons NN. J. sege NN I) imccormick
(extt lovee Vn 1)
(mottwttr__gg -N_NNN J,rri VB
a(dl.. 0.8
Table 12: "Other" facts predicted TuckER.
(mississippi river NN J. americam store stire_N NN J2
(thomas the apoote NN J. reiglddye AN /)
center JJ
(thmas the aposie NN NN J. aponte NN NY_2)
deceiiao engineer NN
(musical aomnical_jnntttnn fnstrument NN J. write V.. 6)
trindicate. 2, /ustifttationn (N_2)
lgynchrunize. VB 2, symchroncopp.. AN
Iolommnt nigninas AN J. hishop AN 1) firish capiaal NN
bwoik V8 9, simuner frome NN. II
recurder NW 1]
deechhngg NN J. percaalar VR 3)
Imcourmick NN I. painter NN M.Nter-NN ()
(drive NN f2, badminton NY !)
(emmp VB 2, struggle AN 2)
NN aporile NN 31
(liiing NN wmp up VB
(mush 2, mill VB 2)
(ave. 3, uiltty NW
r. 0.6
(tppe VB 3,
imcommic
(malcolm NN omancipationie AN 1) (hooaax the sponiie NN J. church, fatheer jatter NN 1I
(rolbon, ralboll NN J. pasenger rehicle NN JI
ist. gregory of n. NN J. sermonizer NN 1I movie
NW J)
frolbon roll-aff NN J. matorcar NN ()
date va 5, geological dating AN 1)
(st__ pow.poohhh NY I. J.w. urben center NN 1)
(indication NN J. signalize VR 2)
(turn ower 3, rotation (N
(ddenificatio fetiiontiioo_.. NN I. nume VB 5)
justification NN. 1. appologg NN 2)
(st. poul NN 2. phitosapherr AN 1)
1
(yympuiist J. gong NW 21 21
(staMeer NN J. thrust V8.5) 5)
(deop. south NN. J. touth, NN 11
(Lihel NN 1, lagioleuure AN D)
(theresa NN I. monk Y.monk NN f)
iibn-roddd NN J. polagogue
V8 18_22, emblem_NN
(yyo NN 1, strr VB 5)
organist
a role..t.e.)) 8. 0.1
famuufacture Vn. J.
(helsikii NN
(nabert own,
Relation (Ppe))
detivationaaly.
hypermyya
instunce.
rolated,
synvet topic (C)
(R)
15

---

Under review as a conference paper at ICLR 2020
(nasaiiaation NN sound nnt VR. 1)
(unredanteeey NN 1,aritarry II.I)
tepolintion AN 2. sack VB
(destre NN 2, 2.hop whine whine_NN NN_
2)
(tread NN J. seep VB 78
(yeam.. V2. J. pining NN D)
InnvennNN NN 22, iraveor
(spark AN J. 1.sporte VR II
(stookkot MN M,okkinn JN
(amufli fee VW
JI
(ioy not J. somatic. coll mut transpplantttion
offerr NN I)
(copital of gombia AN J. urbon senter NN II
jolond, NW urbon urdunneemerr senter NN_11
mier AN Or e. hynd NN J, noncommiasion (wordd wur LN J. pitched buiile NN 1)
J, urhon conter NN
NN
(spopphulanuu. NA J. 1.nn--oufj NN 1)
mmppyoo AW J. eeropenn nation NN_) []
puppon NN american mote NN 1)
(hebriah NN arbon center NN D.
(alogger v.ffball gume NN_I)
flghhing NN A. set on fire 18. 33
reppoeee. 18. J. salking 90 NN. 1)
(ourt AN J. sciniilat VH. 1)
(ubiliy NN 2. funcliounl 11.1)
(eviist NN 2, football (drire NN court gome IN
(drive NN 12, hadmintom NN 1)
(amelling NN J. wind VB 41
(oustain V8. 5, &heam N. 2)
(spring NN 6. hurolll VB.
(heress NN J. monk NN 11
(jut 11.3 validity CN IN ()
foril, VW. offence NN 31
Table 13: "Other" facts as predicted by MoRE.
(ole, [. ful) 0.8
ürish capiral
Delcano poliono__jioon
Igroat
tcenral america NN J. conrrol americo NN JI
gregory LNN J. church, father NN 1I tomoht Joma
fmmiicalm VV. J. hhrch-commnn reformer NN I)
fgreat Jakes NN J. conerdion grevinceeeeeeee Mo NN. 1I trmmriinatio.
NN
(wicconii NN 2. geographical region NN. JI
(muiicalm 1. NN. 1. pusiive resiver NY II
AN.I)
XN.1
(qundde AN 3, wodtrn charch NV D)
fword picture NN sketch, VB 2)
feloze VIL3, commentury NN [1)
strangler free. NN
toplit up VB J. separation AN 5)
(plication NN wrinkle 2)
flos angolor AN J. poport NN 1)
NN. urban
(gettalorz 2. slege NN I)
(wilee VB transcrpptonn NN
tomple 4 musiimisn Mastiiimm__a NW
/retatiinshpp 4, biology NY
(rot 18.22 equatriam sport NN I)
2. foathall gome 1)
(developp 10, adjurment
friaaate VW 2, viinattion NN
tommber VB. 3. point NN 12)
(plication NN. J. twit VB.4 40
lastee NN 2. hockey NN I)
(ron NN J. chemioryy NN II
fassid NN 2. succer NN I]
african
fasiit 2. spor NN (JI)
(develip VB Hh. arganic (ufffoate
nlel r.t.l 0.6
fgreat
Ny NN. J. commet uioved officer NN 13
J. vicar of chrri 1
junpleatont Jl I. iopalammionn (N_I) II)
NNN NN J. mational Leciioooo_oppii compitll 11
fxmmitic deity NN 1, myyholog NN 1)
yutheesy AN J. chemisry NV_
feetiiy 18 2, information NN II
(conncet VB 6, converging NN 1)
(connecr VB commetion NN 4)
moaicalm x. LNN 1. hipuuer NN J)
the nazurene NN J. judaii Modiim_NN_ NN
11
fr e Jynd bonth hovath_hora howa_WW NV penismla NN 1)
ist gregory of iialionrriion region NN 11 Inippon
NX J. tycon, NN 1)
fnon Fing NN J. Larban center NN 17
flibel NN J. criminal low NN J)
(brynhild NN J. mythollgg NV.2)
fomelear deeerence NN J. law lov_NN NN J) treception
bovehall game NN
faurzound_ VB... J. wall AN 1)
(ooee VB. 3 enjoyment Imugnitud: NN J. till II.))
2)
toporatte VB 18.4,pm pxyo I)
(morket VB 1, brade AW 4)
(operate VB 4, mivsion NN
(dhegger NN J. 1. sport NN 1)
fännld NN purable NN 1)
court gome NV
Inberr NW_. italion
mieee To full 0.4
foid stote_ NN.
boell_ NoW
Imundalay
Relation (Type)
synset domain topicc of
derivationally related, form (R)
hypernym (S)
instanee
16


Reviews:
[{"review_id": "rkgCydwkcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThere has been a large family of knowledge base models developed in the recent years, aiming to encode both entities and relations in a latent space, so the entities can be “linked” via a relation-specific mapping.\n\nThe paper focuses on understanding these entity embeddings (and geometric embedding relationships), built on top of the connections with PMI-based word embeddings.\n\nThe paper categorizes all the relations into 3 types (1) related, (2) specialisation, and (3) context shift, and examines some relations in WordNet and NELL, and then empirically evaluates the performance of different types of models and draws the correlation of the results and intuitive understanding of different types of relations.\n\nTo me, this paper is more like providing some intuitive explanations of existing KG embeddings methods and their performance (not really theoretical justifications). It was an interesting read and I appreciate the authors trying to understanding the latent structure that has been encoded in these models. However, I am just not that sure how many take-aways we can get from this study. \n\nI am wondering how loose this categorization is , esp. for the important relations in practice. I’d be also interested in seeing more results on Freebase (and possibly Wikidata) as those KG embeddings are usually more useful. As indicated in the Appendix, the paper mentiosns most of FB15k-237 datasets are in type C, so I am just not sure how many R/S relations are actually there.\n\nAlso, according to Table 3 and Table 4, I am not sure if there are any surprising findings from there. It seems that there is some randomness/noise, but MuRE generally works better than the others. It is true that DistMult works well on the R-type relations but it is not consistent between WN and NELL.\n\nIt’d be useful to show results on more relations (and aggregated results in each category). \n\nIt'd be really great if the paper actually provides some insights on we can further improve these entity embeddings according to this categorization. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lsoiXVKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper attempts to understand the latent structure underlying knowledge graph embedding methods. The work can be seen as an extension of understanding of PMI-based word embedding methods. They categorize knowledge graph relations into three categories based on their relation conditions: Relatedness (R), Specialisation (S), and Context-shift (C). For each category, they evaluate a representative of different types of knowledge graph embedding methods. Through results, they demonstrate that a model’s ability to represent a specific relation type depends on the limitations imposed by the model architecture with respect to satisfying the necessary relation conditions.\n\nQuestions:\n1. The results in Tables 3 and 4 demonstrate that MuRE is the most effective method for handling different types of relations but then how come its performance on FB15k-237 (.336 MRR) is significantly lower than other methods like TuckER (.358 MRR). Can you provide an explanation?\n\n2. In Section 3.2, the authors list 4 predictions (P1-4). It would be great if authors could provide some more reasoning behind coming with these predictions. \n\n3. In Section 4.2, it is stated that “ranking based metrics like MRR and hits@k are flawed if entities are related to more than k others”. It would be great if the authors could give an example to make it more clear. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1guFCaotH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to provide a detailed study on the explainability of link prediction (LP) models by utilizing a recent interpretation of word embeddings. More specifically, the authors categorize the relations in KG into three categories (R, S, C) using the correlation between the semantic relation between two words and the geometric relationship between their embeddings. The authors utilize this categorization to provide a better understanding of LP models’ performance through several experiments.\n\nThis paper reads well and the results appear sound. I personally believe that works on better understanding KGC models are a very essential direction which is mostly ignored in this field of study. Moreover, the provided experiments support the authors’ intuition and arguments.\n\nAs for the drawbacks, I find the technical novelty of the paper is somewhat limited, as the proposed method consists of a mostly straightforward combination of existing methods. Further, I believe this work needs more experimental results and decisive conclusions identifying future directions to achieve better performance on link prediction. My concerns are as follows:\n\n•    I am wondering about the reason for omitting Max/Avg path for two of the relations in WN18RR? Further, the average of 15.2 for the shortest path between entities with “also_see” relation appears to be a mistake?\n•    Was there any specific reason in choosing WN18RR and NELL-995 KGs for the experiments?\n•    It would be interesting to see the length of paths between entities for train and test data separately. \n•    I suggest providing a statistical significance evaluation for each experiment to better validate the conclusions.\n•    I find the provided study in section 4.2 very similar to the triple classification task in KGs. Can you elaborate on the differences and potential advantages of your setting?\n•    I am wondering how you identified the “Other True” triples for WN18RR KG in section 4.2 experiments?\n\nOn overall, although I find the proposed study very interesting and enlightening, I believe that the paper needs more experimental results and decisive conclusions.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a set of conditions that enable a mapping from word embeddings to relation embeddings in knowledge graphs. Then, using recent results about pointwise mutual information word embeddings, the paper provides insights to the latent space of relations, enabling a categorization of relations of entities in a knowledge graph. Empirical experiments on recent knowledge graph models (TransE, DistMult, TuckER and MuRE) are interpreted in light of the predictions coming from the proposed set of conditions.

The authors responded to reviewer comments well, providing significant updates during the discussion period. Unfortunately, the reviewers did not engage further after their original reviews, and so it is hard to tell whether they agreed that the changes resolved all their questions.

Overall, the paper provides much needed analysis for understanding of the latent space of relations on knowledge graphs. Unfortunately, the original submission did not clearly present the ideas, and it is unclear whether the updated version addresses all the concerns. The paper in its current state is therefore not yet suitable for publication at ICLR.
Decision: Reject