Interpreting Knowledge Graph Relation Representation from Word Embeddings
Authors: Carl Allen
Ivana Balazevic
Timothy Hospedales
Venue: ICLR 2021 Poster (2021)
Keywords: knowledge graphs
word embedding
representation learning

Abstract: Many models learn representations of knowledge graph data by exploiting its low-rank latent structure, encoding known relations between entities and enabling unknown facts to be inferred. To predict whether a relation holds between entities, embeddings are typically compared in the latent space following a relation-specific mapping. Whilst their predictive performance has steadily improved, how such models capture the underlying latent structure of semantic information remains unexplained. Building on recent theoretical understanding of word embeddings, we categorise knowledge graph relations into three types and for each derive explicit requirements of their representations. We show that empirical properties of relation representations and the relative performance of leading knowledge graph representation methods are justified by our analysis.

Published as a conference paper at ICLR 2021
INTERPRETING KNOWLEDGE GRAPH RELATION REPRESENTATION FROM WORD EMBEDDINGS
Carl Allen' Ivana Balažević & Timothy Hospedales
1 University of Edinburgh, UK Samsung Al Centre, Cambridge, UK (carl. allen, ayana.balazevig h.hospedales) @ed. ac. uk
ABSTRACT
Many models learn representations of knowledge graph data by exploiting its low-rank latent structure, encoding known relations between entities and enabling unknown facts to be inferred. To predict whether a relation holds between entities, embeddings are typically compared in the latent space following a relation-specific mapping. Whilst their predictive performance has steadily improved, how such models capture the underlying latent structure of semantic information remains unexplained. Building on recent theoretical understanding of word embeddings, we categorise knowledge graph relations into three types and for each derive explicit requirements of their representations. We show that empirical properties relation representations and the relative performance of leading knowledge graph representation methods are justified by our analysis.
INTRODUCTION
Knowledge graphs are large repositories of binary relations between words (or entities) in the form of (subject, relation, object) triples. Many models for representing entities and relations have been developed, so that known facts can be recalled and previously unknown facts can be inferred, a task known link prediction. Recent link prediction models (e.g. [Brdds et al. 2013) Trouillon al.] 2016 Balažević et al.] 2019b) learn entity representations, or embeddings, of far lower dimensionality than the number of entities, by capturing latent structure in the data. Relations are typically represented as mapping from the embedding of a subject entity to those of related object entities. Although the performance of link prediction models has steadily improved for nearly decade, relatively little is understood of the low-rank latent structure that underpins them, which we address in this work. The outcomes of our analysis can be used aid and direct future knowledge graph model design.
We start by drawing parallel between the entity embeddings of knowledge graphs and context- free word embeddings, e.g. as learned by Word2 (W2V) (Mikolov et et. 2013a) and GloVe (Pennington) al., 22144). Our motivating premise is that the same latent word features (e.g. meaning(s), tense, grammatical type) give rise to the patterns found different data sources, i.e. manifesting in word co- occurrence statistics and determining which words relate to which. Different embedding approaches may capture such structure in different ways, but if it is fundamentally the same, an understanding gained from one embedding task (c.g. word embedding) may benefit another (e.g. knowledge graph representation). Furthermore, the relatively limited but accurate data used in knowledge graph representation differs materially from the highly abundant but statistically noisy text data used for
word embeddings. As such, theoretically reconciling the two embedding methods may lead to unified and improved embeddings learned jointly from both data sources.
Recent work (Allen & Hospedales 2019 Allen et al. 2019) theoretically explains how semantic properties are encoded in word embeddings that (approximately) factorise a matrix of pointwise mutual information (PMI) from word co-occurrence statistics, as known for W2V (Levy & Goldberg 2014). Semantic relationships between words, specifically similarity. relatedness, paraphrase and analogy, are proven to manifest as linear geometric relationships between rows of the PMI matrix (subject to known error terms), of which word embeddings can be considered low-rank projections. This explains, for example, the observations that similar words have similar embeddings and that embeddings of analogous word pairs share a common "vector offset" (e.g. Mikolov et al. 2013b).
"Equal contribution

---

Published as a conference paper at ICLR 2021
Table 1: Score functions of representative linear link prediction models. R E Rd and r E R are the relation matrix and translation vector, W E IR xd, xde is the core tensor and b,, by E B are the entity biases.
Model
Linear Subcategory
Score Function
TransE
Bordes et al.
2013
additive
lle. +r.+r-ee,
DistMult
Yang cl al. 2015
multiplicatiie (diagonal)
e. Re.
TuckER
Balazevic et al.
2019b
multiplicative
W X2 X3 Eo
MuRE
Balažević et al.
2019a
multiplicative (diagonal) additive
Re, +r e
b.
We extend this insight to identify geometric relationships between PMI-based word embeddings
that correspond to other relations, i.e. those of knowledge graphs. Such relation conditions define relation-specine mappings between entity embeddings (i.e. relation representations) and so provide "blue-print" for knowledge graph representation models. Analysing the relation representations of leading knowledge graph representation models, we find that various properties, including their relative link prediction performance, accord with predictions based on these relation conditions, supporting the premise that a common latent structure is learned by word and knowledge graph emhedding models, despite the significant differences between their training data and methodology.
In summary, the key contributions of this work are:
to use recent understanding of PMI-based word embeddings to derive geometric attributes of a relation representation for it map subject word embeddings to all related object word embeddings (relation conditions), which partition relations into three rypes (33);
show that both per-relation ranking as well as classification performance of leading link prediction models corresponds to the model satisfying the appropriate relation conditions, i.e. how closely its relation representations match the geometric form derived theoretically $ and to show that properties of knowledge graph representation models fit predictions based on relation conditions, e.g. the strength of relation's 's relatedness aspect is reflected in the eigenvalues of its relation matrix (4.2).
BACKGROUND
Knowledge graph representation: Recent knowledge graph models typically represent entities es 6, as vectors Cs2 C. E TRd and relations as transformations in the latent space from subject to object entity embedding, where the dimension de is far lower (e.g. 200) than the number of entities ne (e.g. >> 104).. Such models are distinguished by their score function, which defines (i) the form of the relation transformation, e.g. matrix multiplication and/or vector addition; and (ii) the measure of proximity between a transformed subject embedding and an object embedding, e.g. dot product or Euclidean distance. Score functions can be non- non-linear (e.g. Dettmers et al. 2018). or linear and sub-categorised as additive, multiplicative or both. We focus on linear models due to their simplicity and strong performance
link prediction (including otate-of-the-art.. Table [ shows the score functions of competitive linear knowledge graph embedding models spanning the sub-categories: TransE (Bordes et al., 2013), DistMult (Yang et al. 2015), TuckER (Balažević al. 2019b) and MuRE BBlazević et al. 2019a).
Additive models apply relation-specific translation to a subject entity embedding and typically use Euclidean distance to evaluate proximity to object embeddings. A generic additive score function is A simple example TransE, where b, =b, =0.
given by ő(es, r, =- + e,.12
- + b..bbb......
Multiplicative models have the generic score function ó(e,, r, e.) = Re,, i.c. a bilinear product of the entity embeddings and a relation-specifi. matrix R. DistMult a simple example with R diagonal and so cannot model asymmetric relations (Trouillon et al., 2016). In TuckER, each relation-specife R- X3 r is linear combination of d, "prototype" relation matrices in a core tensor We IR xad, (x (nn denoting tensor product along mode nt), facilitating multi-task learning across relations.
Some models, e.g. MuRE, combine both multiplicative (R) and additive (r) components.
Algorithms such as Word2Vec (Mikolov et al. 2013a) and Glo (Pennington ct al. 2014) generate low-dimensional word embeddings that perform well on downstream tasks (Baroni et al. 2014). Such models predict the context words (c;) observed around target word (w:) in a text corpus using shallow neural networks. Whilst recent language models (e.g. Devlin et al.) 2018, Peters et al.) 2018) achieve strong performance using contextullised word embeddings, focus on "context-free embeddings since knowledge graph entities have no obvious context and, importantly, they offer insight into embedding interpretability.
Word embedding:
2

---

Published as a conference paper at ICLR 2021
Levy & Goldberg (014)) show that, for a dictionary of ne unique words and embedding dimension de & ne. W2V's s loss function is minimised when its embeddings w;, сj form matrices W. C Rd X that factorise a pointwise mutual information (PMI) matrix of word co-occurrence statistics P(w,,;;) ), subject to a shift term. This result relates W2V to earlier count-based
(PMI(u;,, Cy) = log P(ws)
embeddings and specifically PMI, which has a history in linguistie analysis (Turney & Pantel 2010).
From its loss function, Glo'e can be seen to perform a related factorisation.
Recent work (Allen & Hospedales 2019 Allen et al. 2019) shows how the semantic relationships of similarity, relatedness, paraphrase and analogy are encoded in PMI-based word embeddings by recognising such embeddings as low-rank projections of high dimensional rows of the PMI matrix, termed PMI vectors. Those semantic relationships are described in terms of multiplicatiive interactions between co-occurrence probabilities (subject to defined error terms), that correspond to additive interactions between (logarithmic)) PMI statistics, and hence PMI vectors. Thus, under a sufficiently linear projection, those semantic relationships correspond to linear relationships between word embeddings. Note that although the relative geometry reflecting semantic relationships is preserved, the direct interpretability of dimensions, as in PMI vectors, is lost since the embedding matrices can be arbitrarily scaled/rotated if the other is inversely transformed. We state the relevant semantic relationships on which we build, denoting the set of unique dictionary words by E:
word subsets W, W CE are said to paraphrase if they induce similar distributions over nearby words, i.e. p(E[W) =p(E)W") e.g. {king) paraphrases {man, royal).
Paraphrase:
Analogy: a common example of an analogy is "woman is to queen as man is to king" and can be defined any set of word pairs {(w,, w hez for which it is semantically meaningful to say "wo is to w as Wi is to " Ya, bEI.
Where one word subset paraphrases another, the sums of their embeddings are shown to be equal (subject to the independence of words within each set), e.g. Wking R Wman + Wroyat- An in- teresting connection is established between the two semantic relationships: a set of word pairs A= {(wr, wH), (Wp, ) is an analogy if (wa: wx)) } paraphrases (wi W, in which case the em- beddings satisfy W.. Wa & wh. Wb ("vector offset").
FROM ANALOGIES TO KNOWLEDGE GRAPH RELATIONS
Analogies from the field of word embeddings are our starting point for developing theoretical basis for representing knowledge graph relations. The relevance of analogies stems from the observation that for an analogy to hold (see (2). its word pairs, e.g [(man, king), (woman, queen), (girl, princess)), must be related the same way, comparably to subject-object entity pairs under a common knowledge graph relation. Our aim is to develop the understanding of PMI-based word embeddings (henceforth word embeddings), to identify the mathematical properties necessary for a relation representation to map subject word embeddings to all related object word embeddings.
Considering the paraphrasing word sets (king) and (man, royal) corresponding to the word embed- ding relationship Wking Rs Wman + Wroya! (22). royal can be interpreted as the semantic difference between man and king. fitting intuitively with the relationship Wroyal R3 Wking - Wman- Fundamen- tally, this relationship holds because the difference between words that co-occur (i.e. occur more frequently than if independent) with king and those that co-occur with man, reflects those words that co-occur with royal. We refer to this difference in co-occurrence distribution as "context shift", from man (subject) to king (object). Allen & Hospedales (2019) effectively show that where multiple word pairs share a common context shift, they form an analogy whose embeddings satisfy the vector offset relationship. This result seems obvious where the context shift mirrors an identifiable word, the embedding of which is approximated by the common vector offset, e.g. queen and woman are related by the same context shift, i.e. Wqucen RJ Wwoman + royal, thus Wqusen - Wyooman R Whing- wman- However, the same result holds, i.e. an analogy is formed with a common vector offset between embeddings, for arbitrary (common) context shift that may reflect no particular word. Importantly, these context shift relations evidence a case in which is known how a relation can be represented, Le. by an additive vector (comparable to TransE) if entities represented by word embeddings. More generally, this provides an interpretable foothold into relation representation.
Note that not all sets of word pairs considered analogies exhibit clear context shift relation, e.g. in the analogy I (caar,engine). (bus,seats)), the difference between words co-occurring with engine and car is not expected to reflect the corresponding difference between bus and seats. This illustrates how
3

---

Published as a conference paper at ICLR 2021
22
(a) Similarity
(b) Relatedness
(c) Specialisation (d) Context-shift (e) Gen. context-shift
Figure 1: Relationships between PMI vectors (black rectangles) of subject/object words for different relation types. PMI vectors capture co-occurrence with every dictionary word: strong associations (PMI> 0) are shaded (blue detine the relation, grey are random other associations); red dash = relatedness; black dash = context sets.
analogies are loosely defined concept, e.g. their implicit relation may be semantic or syntactie, with several sub-categories of each (e.g. see Gllakooa et al.] (2016). The same is readily observed for the relations of knowledge graphs. This likely explains the observed variability in "solving" analogies by use of vector offset (e.g. Köper et al. ; Karpinska et al., 2018 Gladkova et al. 2016) and suggests that further consideration required to represent relations (or solve analogies) in general.
We have seen that the existence of a context shift relation between a subject and object word implies (relation-specifie) geometric relationship between word embeddings, thus the latter provides a necessary condition for the relation to hold. We refer to this as a "relation condition" and aim to identify relation conditions for other classes of relation. Once identified, relation conditions define mapping from subject embeddings to all related object embeddings, by which related entities might be identified with a proximity measure (e.g. Euclidean distance or dot product). This is the precise aim of knowledge graph representation model, but loss functions are typically developed heuristically. Given the existence of many representation models, we can verify identified relation conditions by contrasting the per-relation performance of various models with the extent to which their loss function reflects the appropriate relation conditions. Note that since relation conditions are necessary rather than sufficient, they do not guarantee relation holds, i.e. false positives may arise.
Whilst we seek to establish relation conditions based on PMI word embeddings, the data used to train knowledge graph embeddings differs significantly to the text data used by word embeddings, and the relevance of conditions ultimately based on PMI statistics may seem questionable. However, where knowledge graph representation model implements relation conditions and measures proximity between embeddings, the parameters of word embeddings necessarily provide a potential solution that minimises the loss function. Many equivalent solutions may exist due to symmetry as typical for neural network architectures. We now define relation types and identify their relation conditions (underlined); we then consider the completeness of this categorisation.
Similarity: Semantically similar words induce similar distributions over the words they co-occur with. Thus their PMI vectors and word embeddings are similar (Fig Ta).
The relatedness of two words can be considered in terms of the words S E with which both co-oceur similarly. S defines the nature relatedness, e.g. milk and cheese are related by S == {dairy, breakfast, ...); and [S] reflects the strength of relatedness. Since PMI vector components corresponding to S are similar (Fig lb). embeddings of S-related words have similar components in the subspace Vs that spans the projected PMI vector dimensions corresponding to S. The rank of Vs is thus anticipated to reflect relatedness strength. Relatedness can be seen as a weaker and more variable generalisation of similarity, its limiting case where S E,
Relatedness:
E, hence rank () dg.
Context-shift: As discussed above, words related by a common difference between their distribu- tions of co-occurring words, defined as context-shifts, share common vector offset between word embeddings, Context might be considered added (e.g. man to king). termed a specialisation (Fig Ic subtracted (e.g. king to man) or both (Fig ld]. These relations are 1-to-1 (subject to synonyms) and include an aspect of relatedness due to the word associations in common. Note that, specialisations include hyponyms/hypernyms and context shifts include meronyms.
Generalised context-shift: Context-shift relations generalise to 1 1-to-many, many-to-1 and many- to-many relations where the added/subtracted context may be from a (relation-specifii) context set (Fig Ie). e.g. any city or anything bigger. The potential scope and size of context sets adds variability to these relations. The limiting case in which the context set is "small" reduces to a 1-to-1 context-shift (above) and the embedding difference is known vector offset. In the limiting case of a "large" context set, the added/subtracted context is essentially unrestricted such that only the relatedness aspect of the relation, and thus common subspace component of embeddings. is fixed.
4

---

Published as a conference paper at ICLR 2021
Categorisation completeness: Taking intuition from Fig I and considering PMI vectors as sets of word features, these relation types can be interpreted as set operations: similarity as set equality; relatedness as subset equality; and context-shiff as a relation-speciffe set difference. Since for any relation each feature must either remain unchanged (relatedness), change (context shift) or else be irrelevant, we conjecture that the above relation types give a complete partition of semantic relations.
Table 2: Categorisation of WN18RR relations.
Type
Relation
Exumples (subject entity, object entity)
verb group
(trim down VB
VB 35). (hatch_ VB_ incubate_ VB
R
derivationalll_ recaatele_ddeee
(lodge_ Birige__V______ VB_4, accommodaatoon L_NN_NN___ _N_4), (question_NN) Booooo_NN___.__ inquire_VB VB_I}
also_see
(clean_JJ_1, 1, tidy (I_ (ram_ VB 2, screw_VB_J
hypermym
honddreffow__ae Lerform_N__L II, reform_ L_NN__N__L 1), (prickle-weed NN_ herbaceous plant_NN NN_I)
S
instance hhpaemmymmmyymmmmmmmmmmmm
(yellowstone
yiver_ witperstteeevver___eee
yiver_NN_______ J. river_NN__I 1). (leipzig NN_ wrbba_ceettrr____a
member_ of _ooominnnuu usage
(colloquialism__ _N__ figure VB VB_S), (plurol_ form_ooooo__leeeee ___ outhority_NN_22
member_ of reggion
(rome Lim___N___ J, gladiator_Ni I. B_____NN___I [W_]i, (usa Musa_NN_L J, I,multiple__ooi [______N_
C
member_ mereooyrmer has_part part
(south NN 2. sunshine _stttestttteeeeeeeee Boite_NNN_LL 1), (genus, caass_aara____aa NN J. pecan_iree Sree_reeee tree_NN__)
(aircraft NN_I, I__, cabin_NN
B____NN33,,__ _3, (merocco Caprocco_NN_L_ atlas mountains _N_]]
(quark_NN Mookk_NN_LLL physics NN_iis_NNN_ (harmonize VB_3, (usii_NN_4)
3.1 CATEGORISING REAL KNOWLEDGE GRAPH RELATIONS
Analysing the relations of popular knowledge graph datasets, we observe that they indeed imply (i) relatedness aspect reflecting a common theme (e.g. both entities are animals or geographic terms); and (ii) contextual themes specific to the subject and/or object entities. Further, relations fall under hierarchy of three relation lypes: highly related (R); generalised specialisation (S); and generalised context-shift (C). As above, "generalised" indicates that context differences are not restricted to be 1-1. From Fig [ it can be seen that type R relations are a special case of S, which are special case of C. Thus type C encompasses all considered relations. Whilst there are many ways to classify relations, e.g. by hierarchy, transitivity, the proposed relation conditions delineate relations by the required mathematical form (and complexity) of their representation. Table 2
shows a categorisation of the relations of the WN18RR dataset (Dettmers al. 2018) 18 comprising II relations and 40,943 entities T An explanation for the category assignment is in Appx. A Analysing the commonly used FB 1-3-237 dataset (Totanova et al., 2013) reveals relations to be almost exclusively of type C, precluding contrast of performance per relation type and hence that dataset is omitted from our analysis. Instead, categorise random subsample of 12 relations from the NELL-995 dataset (Xiong et al. 2017). containing 75,492 entities and 200 relations (see Tables 8 and 9 in Appx. B
3.2 RELATIONS AS MAPPINGS BETWEEN EMBEDDINGS
Given the relation conditions of a relation type, we now consider mappings that satisfy them and thereby loss functions able to identify relations of cach type, evaluating proximity between mapped entity embeddings by dot product or Euclidean distance. We then contrast our theoretically derived loss functions, specific relation type, with those of several knowledge graph models (Table I to predict identifiable properties and the relative performance of different knowledge graph models for each relation type.
R: Identifying S-relatedness requires testing both entity embeddings e.,, e, for common subspace component Vs. which can be achieved by projecting both embeddings onto Vs and comparing their images. Projection requires multiplication by matrix P, d and cannot be achieved additively, except in the trivial limiting case of similarity (P, = I) when x= can be added.
Comparison by dot product gives (P,e.)((P,ee)) = P.T. P.e. e M, (for relation-specific symmetric M.= P P,). Euclidean distance gives P.e.- P.e.o12 = (c.-e.) M, -c,) [[P,e_l| - 2e," M,C. + ?P
S/C: The relation conditions require testing for both S-relatedness and relation-specife entity component(s) (v,,, v.). This is achieved by (i) multiplying both entity embeddings by a relation- specific projection matrix P. that projects onto the subspace that spans the low-rank projection of dimensions corresponding to S, oe and of (i.e. testing for S-relatedness while preserving relation- specific entity components); and (ii) adding a relation-specifee vector r = De v. to the transformed subject entity embeddings.
We omit the relation "similar_to" to" since its instances have discernible structure, and only 3 occur in the set, all of which are the inverse of training example and trivial to predict.
5

---

Published as a conference paper at ICLR 2021
Comparing the transformed entity embeddings by dot product equates to (P.e, + r)r p. and by Euclidean distance to P. + P,eol1==11 P,e, + r|| 2(Pre, - P.e. + P, (cf MuRE:
Re, + r -
Contrasting these theoretically derived loss functions with those of knowledge graph models (Table D. make the following predictions:
Pl: The ability to learn the representation of relation is expected to reflect:
(a) the complexity of its type (R<S<C independently of model choice; and
(b) whether relation conditions (e.g. additive/multilicative interactions) are met by the model.
P2: Knowledge graph relation representations reflect the following type-specific properties:
(a) relation matrices for relatedness (type R) relations are highly symmetric;
(b) offset vectors for relatedness relations have low norm; and
(c) as a proxy to the rank of Vs, the eigenvalues of relation matrix reflect relatedness strength.
To elaborate, core prediction (1(b) anticipates that: (i) additive-only models (e.g. TransE) are not suited to identifying the relatedness aspect of relations, except in limiting cases of similarity, requiring
vector); (ii) multiplicative-only models (e.g. DistMult) should perform well on type R relations, but are not suited to identifying entity-specific features of type S/C (an asymmetric relation matrix in TuckER may help compensate); and (iii) the loss function of MuRE closely resembles that derived for type C relations, which generalise all others, and is thus expected to perform best overall.
4 EVIDENCE LINKING KNOWLEDGE GRAPH AND WORD EMBEDDINGS
We test whether the predictions P1 and P2, made on the basis of word embeddings, apply to knowledge graph relations by analysing the performance and properties of competitive knowledge graph models. We compare TransE, DistMult, TuckER and MuRE, which entail different forms of relation representation, on all WNISRR relations and a similar number of NELL-995 relations (spanning all relation types). All models have a comparable number of free parameters.
Since for TransE, the logistic sigmoid cannot be applied to the score function to give a probabilistic interpretation comparable other models, for fair comparison we include MuRE,, a constrained variant of MuRE with R. = R. = I, as a proxy to TransE Implementation details are included in Appx. D For evaluation, we generate 2n, evaluation triples for each test triple (n, = El denoting the number of entities) by fixing the subject entity e, and relation and replacing the object entity e, with each entity in turn and then keeping e, and r fixed and varying Eg. Each model's scores for the evaluation triples ranked to give the standard metric Hits @ 10 (Bordes al. 2013), i.e. the fraction of times a true triple appears in the top 10 ranked evaluation triples.
4.1 P1: JUSTIFYING THE RELATIVE PERFORMANCE OF KNOWLEDGE GRAPH MODELS
Ranking performance: Tables 3 and 4 report Hits @ 10 for each relation and include the relation type as well as known confounding influences: percentage of relation instances in the training and test sets (approximately equal), the actual number of instances in the test set (causing some results to be highly granular). Krackhardt hierarchy score (see Appx. et al. (019a) and maximum and average shortest path between any two related nodes. A further confounding effect is dependence between relations: [acroix et al. (0118) and Balažević et al. (2019h) independently show that constraining the rank of relation representations is beneficial for datasets with many relations due to multi-task learning, particularly when the number of instances per relation low. This is expected to benefit TuckER on the NELL-995 dataset (200 relations).
(Krackhardt) 2014 Balažević
As predicted in (a), all models tend to perform best at type R relations, with a clear performance gap to other relation types. Also, performance on type relations appears higher in general than type C. In accordance with Pl(b). additive-only models (TransE, MuRE,) perform worst on average since all relation types involve (multiplicative) relatedness. Best performance is achieved on type R relations, which can be represented by small/zero additive vector. Multiplicative-onk, DistMult performs well, sometimes best, on type R relations, fitting expectation as it can represent those relations and has no inessential parameters, e.g. that may overfit to noise, which may explain instances where MuRE performs slightly worse. As expected, MuRE, performs best overall (particularly on WN18RR), and most strongly S and C type relations, predicted to require both multiplicative and additive components. Comparable performance of TuckER on NELL-995 may be explained by its multi-task learning ability.

---

Published as a conference paper at ICLR 2021
Table 3: Hits @ 10 per relation WN18RR.
Relation Name
Type
%-
Khs
Max/Avg Path
TransF:
MuRE;
DistMult
TuckER
MuRE
verh verh_group
R
1%
39
(1.00
0.87
0.95
0.97
0.97
0.97
derivationally pwwlateeeddeeeee form
R
34%
1074
0.04
0.93
0.96
0.97
also_see see
R
2%
56
0.24
44
15.2
0.59
0.73
0.67
0.72
0.73
instance hypemym
S
4%
1.00
1.0
0.22
synset_ synset_domain _opiii_o_________ of
C
4%
114
0.99
1.1
0.19
0.43
0.42
0.45
0.53
member_ of domain usage
C
1%
24
1.00
1.0
0.42
0.42
0.48
0.38
0.50
member domain regnainnneeeonnooo region
C
1%
26
1.00
1.0
0.35
0.40
0,40
0.35
0.46
member_mee meronym has part hypennm
C
8%
253
1.00
10
3.9
0.04
0.38
0,30
0.39
0.39
C
6%
1.00
13
2.2
0.04
S
40%
1251
0.99
18
4.5
0.02
0.20
0.19
0.20
0.28
all
100%
3134
0.38
0.52
0,51
0.53
0.37
Table Hits 10 per relation on NEL.I.995.
Relation Name
Type
%
Khs
Max/Avg Path
TransE
MuRE,
DistMult
TackER
MuRE
camplayysgniintteem
R
2%
243
0.11
10
3.5
0.76
0184
0.90
0.89
0.89
Hlotingtogooithbllohhnng
R
1%
132
0.17
2.6
0.72
0.88
8
1%
143
0.38
7
2.5
0.37
0.55
0.62
0.65
0.66
animalistypoofinimal
1%
103
3.1
0.50
0.64
0.68
athleteplayssper
C
1%
113
1.00
1
1.0
0.54
0.58
0,58
0.60
0.64
chemicalislyy
poammicclissppoocheennie peofchemical
S
1%
115
0.53
6
3.0
0.23
0.43
0.49
0.51
0.60
iemfoundraroom
C
2%
162
1.00
1.0
0.39
0.57
0.53
0,56
0.59
agenicollabrrateewillagggr
R
1%
119
0.51
4.7
0.44
0,58
0.64
0.61
0.58
hodypartoonzannshodyyynn
C
1%
103
0.60
3.2
0.30
0,38
0.54
0,58
0.55
atdate
10%
967
0.99
1.1
0.41
0.50
0.48
0.48
0.52
acatioolocatedwwttinooc..
157
0.46
0.48
atloration
1%
294
0.99
6
1.4
0.22
0.33
0.39
0.43
0.44
20000
0.36
0.51
Other anomalous results also closely align with confounding factors. For example, all models perform poorly on the hypernym relation, despite having relative abundance of training data (40% of all instances), which may be explained by its hierarchical nature (Khs x= and long paths). The same may explain the reduced performance on relations also see and aggntcollaborrtessvitug found previously (Balažević et al. (019a),, none of the models considered are well suited to modelling hierarchical structures. We also note that the percentage of training instances of relation is not dominant factor on performance, as would be expected if all relations could be equally represented.
As
Classification performance: We further evaluate whether holds when comparing knowledge graph models by classification accuracy on WN18RR. Independent predictions of whether given triple is true or false are not commonly evaluated, instead metrics such as mean reciprocal rank and Hits @ k are reported that compare the prediction of test triple against all evaluation triples. Not only is this computationally costly, the evaluation is flawed if an entity is related to /> & others (k: is often 1 or 3). A correct prediction validly falling within the top / but not the top k would appear incorrect under the metric. Some recent works also note the importance of standalone predictions (Speranskaya al. 2020 Pezeshkpour et al. 2020)..
Since for each relation there are n possible entity-entity relationships, we sub-sample by computing predictions only for those (e,; r; ea) triples for which the es, pairs appear in the test We split positive predictions (a(o(e,, r, e.))) > 0.5) between (i) known truths - training or test/validation instances; and (ii) other, the truth of which is not known. We then compute per-relation accuracy over the true training instances ("Irain"") and true test/validation instances ("test"); and the average number of "other" triples predicted true per c,, r' pair. Fable results for MuRE,, DistMult, TuckER and MuRE. All models achieve near perfect training accuracy. The additive additive- multiplicative MuRE gives best test set performance, followed (surprisingly) closely by MuRE,, with multiplicative models (DistMult and TuckER) performing poorly on all but type R relations in line with (b),,, with near-zero performance on most type S/C relations. Since the ground truth labels for "other" triples predicted to be true are not in the dataset,
analyse sample of "other" true predictions for one relation of each type (see Appx. G From this, we estimate that TuckER relatively accurate but pessimistic (~0.3 correct of the 0.5 predictions= 60%%. MuRE, is optimistic but inaccurate (~2.3 of 7.5x31%) 31%), whereas MuRE is both optimistic and accurate (~1.1 of 1.5 73%).
Summary: Our analysis identifies the best performing model per relation type predicted by P1(b): multiplicative-nnjy DistMult for type R. additive-miln mutitiivemuuiplicattiee MuRE for types S/C; providing a basis for detaset-dependent model selection. The per-relation insight into where models perform
7

---

Published as a conference paper at ICLR 2021
Table 5: Per relation prediction accuracy for MuRE, (M,). (D)istMult, (T)cckER and (M)RR (WNI8RR).
Accuracy (train)
Accuracy (test)
Other "Truc"
Relation Name
Type
#trriin
# teat
D
verb verb_group
R
155
39
1.00
1,00
1.00
1.00
0.97
0.97
0.97
0.97
8.3
1.7
0.9
2.7
derivationally form
1127
1.00
1.00
0.94
0.95
1.7
alsi__sed see
R
95
61
1.00
1,00
1.00
1.00
0.64
0.64
0.61
0.59
7.9
1.6
0.9
1.9
52
1.00
1.00
1.00
0.32
0.43
0.9
member_co of domain usage
C
545
43
0.98
1.00
1.00
1.00
0.02
0.00
0.02
0.00
1.5
0.6
0.0
0.3
member_en of domain region
543
42
0.88
0.89
1.00
0.93
0.02
0.02
0.00
0.02
1.0
0.4
0.7
synsel synsei domain Sommain_tppic_oo of member meronym has part
1.3
115
1.00
1.00
1.00
1.00
0.42
0.10
0.14
0.47
0.7
0.6
0.1
0.2
1402
307
1.00
1.00
1.00
1.00
0.22
0.02
0.01
0.22
7.9
3.4
1.5
5.6
848
196
1.00
1.00
1.00
1.00
0.24
0.05
0.09
0.22
7.1
2.4
1.3
3.9
hypernym
57
1254
1.00
1.00
1,00
1.00
0.15
0.02
0.02
0.22
1.2
1.7
5284
3306
0.99
0.99
1.00
0.99
0.47
0.37
0.37
0.50
5.9
1.2
0.5
2.1
poorly, e.g. hierarchical or type C relations, can be used to aid and direct future model design. Analysis of the classification performance: (i) shows that MuRE is the most reliable fact prediction model; and (ii) emphasises the poorer ability of multiplicative-onn -nlllliccavveellyyyy models to represent S/C relations.
4.2 P2: PROPERTIES OF RELATION REPRESENTATION
P2(a)-(b): Table the symmetry score (E [-1, 1] indicating perfect anti-symmetry symmetry; see Appx. F for the relation matrix of TuckER and the norm of relation vectors of TransE, MuRE, and MuRE on the WNI8RR dataset. As expected, type R relations have materially higher symmetry than both other relation types, fitting the prediction of how TuckER compensates for having no additive component. All additive models learn relation vectors of noticeably lower norm for type R relations, which in the limiting case (similarity) require no additive component, than for types S or C.
P2(c): Fig D eigenvalue magnitudes (scaled relative to the largest and ordered) of relation- specific matrices R of MuRE, labelled by relation type, as predicted to reflect the strength of relation's relatedness aspect. As expected, values are highest for type R relations. For relation types S and C the profiles are more varied, supporting the understanding that relatedness of such relations highly variable, both in its nature (components of S) and strength (cardinality S).
Table 6: Relation matrix symmctry score [-1.1] for TuckER; and relation vector norm for TransE, MuRE, and MuRE (WN18RR).
0.8
Symmerry Score TuckER
Vector Norm MuRE,
Relation
Type
TransE:
MuRE
0.6
verb_ verb_group group
R
0.52
5.65
0.76
0.89
derivationally related__ form
R
0.54
2.98
0.45
0.69
0.4
also__ses see
R
0.50
7.20
0.97
0.97
sedlacce_bypeyyn member of domain usage
S
0.13
18.26
2.98
1.88
02
0.10
11.24
3,18
1.88
member of domain region
C
0.06
12.52
3.07
2.11
0.0
synset domain topic member meronym has part
C
0.12
4.97
1.91
1.97
6.44
Figure 2: Eigenvalue magnitudes of relation-specilic matrices R for MuRE by relation type (WN18RR).
hypernym
0.04
9.64
1.53
1.03
5 CONCLUSION
Many low-rank knowledge graph representation models have been developed, yet little is known of the latent structure they learn. We build on recent understanding of PMI-based word embeddings to theoretically establish set of geometric properties of relation representations (relation conditions) required to map PMI-based word embeddings of subject entities to related object entities under knowledge graph relations. These conditions partition relations into three types and provide basis to consider the loss functions of existing knowledge graph models. Models that satisfy the relation conditions of particular type have a known set of model parameters that minimise the loss function, i.e. the parameters of PMI embeddings, together with potentially many equivalent solutions. We show that the better a model's 's architecture satisfies a relation's is conditions, the better its performance at link prediction, evaluated under both rank-based metrics and accuracy. Overall, we generalise recent theoretical understanding of how particular semantic relations, e.g. similarity and analogy, encoded between -as-based word embeddings the general relations of knowledge graphs. In doing so, we provide evidence in support of our initial premise: that common latent structure is exploited by both PMI-based word embeddings (e.g. W2V) and knowledge graph representation.
8

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
Carl Allen and Ivana Balažević were supported by the Centre for Doctoral Training in Data Science, funded by EPSRC (grant EP/L016427/I) and the University of Edinburgh.
REFERENCES
Carl Allen and Timothy Hospedales. Analogies Explained: Towards Understanding Word Embed- dings, In International Conference on Machine Learning, 2019.
Carl Allen, Ivana Balažević, and Timothy Hospedales. What the Vec? Towards Probabilistically Grounded Embeddings. In Advances in Neural Information Processing Systems, 2019.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. Multi-relational Poincaré Graph Embed- dings. In Advances Neural Information Processing Systems, 2019a.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. TuckER: edge Graph Completion. In Empirical Methods in Natural Language Processing, 2019b.
Tensor Factorization for Knowl-
Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't Count, Predict! Systematic Comparison of Context-Countee vs. Context-Predicting Semantic Vectors. In Association for Computational Linguistics, 2014.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. Advances in Neural Information Processing Systems, 2013.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D Knowledge Graph Embeddings. In Association for the Advancement of Artificial Intelligence, 2018.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics, 2018.
Anna Gladkova, Alcksandr Drozd, and Satoshi Matsuoka. Analogy-based Detection of Morphological and Semantic Relations With Word Embeddings: What Works and What Doesn't. In NAACL Student Research Workshop, 2016.
Lawrence Hubert and Frank B Baker. Evaluating the Symmetry of Proximity Matrix. Quality & Quantity, 13(1):77-84, 1979.
Marzena Karpinska, Bofang Li, Anna Rogers, and Aleksandr Drozd. Subcharacter Information in Japanese Embeddings: When Is It Worth It? In Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP, 2018.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference Learning Representations, 2015.
Maximilian Köper, Christian Scheible, and Sabine Schulte im Walde. Multilingual reliability and "semantic" structure of continuous word spaces. In International Conference Computational Semantics, 2015.
David Krackhardt. Graph Theoretical Dimensions of Informal Organizations. In Computational Organization Theory. Psychology Press, 2014.
Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical Tensor Decomposition for Knowledge Base Completion. In International Conference Machine Learning, 2018.
Omer Levy and Yoav Goldberg. Neural Word embedding as Implicit Matrix Factorization. In Advances in Neural Information Processing Systems, 2014.

---

Published as a conference paper at ICLR 2021
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, 2013a.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig, Linguistic regularities in continuous space word representations. In North American Chapter of the Association for Computational Linguistics, 2013b.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In North American Chapter of the Association for Computational Linguistics, 2018.
Pouya Pezeshkpour, Yifan Tian, and Sameer Singh. Revisiting Evaluation of Knowledge Base Completion Models. In Automated Knowledge Base Construction, 2020.
Marina Speranskaya, Martin Schmitt, and Benjamin Roth. Ranking vs. Classifying: Measuring Knowledge Base Completion Quality. In Automated Knowledge Base Construction, 2020.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing Text for Joint Embedding of Text and Knowledge Bases. In Empirical Methods in Natural Language Processing, 2015.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex Embeddings for Simple Link Prediction. In International Conference on Machine Learning, 2016.
Peter D Turney and Patrick Pantel. From Frequency to Meaning: Vector Space Models of Semantics.
Journal of Artificial Intelligence Research, 37:141-188, 2010.
Wenhan Xiong, Thien Hoang, and William Yang Wang. DeepPath: Reinforcement Learning
Method for Knowledge Graph Reasoning. In Empirical Methods in Natural Language Processing, 2017.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In International Conference Learning Representations, 2015.
10

---

Published as a conference paper at ICLR 2021
A
CATEGORISING WN18RR RELATIONS
Table 7 describes how each WN18RR relation was assigned to its respective category.
Table 7: Explanation the WN18RR relation category assignment.
Type
Relation
Relateduess
Subject Specifics
Object Specifies
verb_group
both verbs; similar in meaning
R
derivationally relltted_foont
different syntactic categories; semantically related semuntically similar
also
S
instance hypemym
mypermym
semantically similar
semantically sunilar
instance of the object instance object
member of domain usage member, rrogion
loosely semantically related (word usage features)
usage descriptor
broad feature set broad feature set of the subject part of the subject domain descriplor
loosely semantically related (regional features)
region descriptor
C
member nemmerr_meronym
semanticalyy related
collection of objects collection of objects broad feature
has_pamt part
semantically related
synset comainn topic_pi
semuntically related
B CATEGORISING NELL-995 RELATIONS
Categorisation of NELL-995 relations and the explanation for the category assignment of are shown in Tables and respectively.
Table 8: Categorisation of NELL-995 relations.
Type
Relation
Examples (subject entiry, object entity)
leamplaysaguinstteum
(rangers, mariners), (phillies, tampa tampa_bay__a bay_aays (shirts, trousers), (shoes, black shirt)
R
clothingtogooithcllohng
agentcollaborates withagent
(white_striper jack_white), _obamaa hillary_clitton)
wpwreiittttttooosssssssssoe unimalistypeufaiinall
(trial lawyers, attorneys), (engineers, experts)
S
(cats, aanimalss,,)) (chickens, livestock)
chemicalistyypoffccmiiaa
(moisture, gas), (oxide, materials)
athleteplayssport
(joe_ smith, baseball). (chris cooley, football) (bed, den). (refrigerator, kitchen _rea)) (system002, eyes), (blood, left left_ ventricle) (scotland, n2009). (wto, n2003)
itemfoundinroom
boddyputtontainssodyyarr
C
atdate
Incatiolloatedvviiiinooaal
(medellin, colombia), (juckson, wyoming) (ogunquin, maine), (palmer Jake, colorado)
atlocation
Table 9: Explanation for the NELL-995 relation category assignment.
Type
Relation
Relatedness
Subject Specifics
Object Specifies
keamplaysagainstteem
sport teams
R
clathinglogon www.thckthiingggg
both iiems of clothing that gu together people companies; related industries
agentcollaboratese tthgeen
woofosscoiitypeoorrfffsss
semantically related (both profession types) semantically related (both animals)
instance of the object instance object
S
chemicalsstypoofccmmic
semantically related (both chemicals)
instance of the object
athleteplaysspot
semantically related (sports features)
athlete descriptor item descriptor
sport descriptor deseriptor
itemfoundinroom
semantically related (room/fumiture fentures)
waulypartcomlainsbuuyy
emantically related (specific body part features)
collection objects
part of the subject
C
atdate
loosely semantically related (start date fentures)
braad feature
date descriptor
Incationooatedviiiillloccc
semantically related (geographical features)
part of the subject
collection objects
arlocation
semantically related (pcographical features)
part of the subject
collection objects
C SPLITTING THE NELL-995 DATASET
The test set of NELL-995 created by Xiong et al. (201)) contains only 10 out of 200 relations present in the training set. To ensure a fair representation of all training set relations in the validation and test sets, we create new validation and test set splits by combining the initial validation and test sets with the training set and randomly selecting 10,000 triples each from the combined dataset.

---

Published as a conference paper at ICLR 2021
D IMPLEMENTATION DETAILS
All algorithms are re-implemented in PyTorch with the Adam optimizer (Kingma & Ba 2015) that minimises binary cross-entropy loss, using hyper-parameters that work well for all models (learning rate: 0.001, batch size: 128, number of negative samples: 50). Entity and relation embedding dimensionality is set dg = d, 200 for all models except TuckER, for which 30 (Balažević et al. 2019b).
E KRACKHARDT HIERARCHY SCORE
The Krackhardt hierarchy score measures the proportion of node pairs (x, 3) where there exists directed path x y. but not y - æ; and it takes a value of one for all directed acyclic graphs, and zero for cycles and cliques (Krackhardi, 2014 Balažević et al. 2019a).
Let M RR* be the binary reachability matrix of directed graph with n nodes, with M.,, = there exists a directed path from node to node and 0 otherwise. The Krackhardt hierarchy score of G is defined as:
Σ1 1(M = IA M₁ = 0)
Khsg
Ei 1(M,, = 1)
SYMMETRY SCORE
The symmetry score E 1]
Hubertt Baker 1979 for a relation matrix R RERd is defined as:
ΣEi R. Rys (EEi R.,
deld.
(2)
ΣEfz R Eig )
(d)
where indicates symmetric and -1 an anti-symmetric matrix.
G "OTHER" PREDICTED FACTS
Tables 10 to 13 shows sample the unknown triples (i.e. those formed using the WNISRR entities and relations, but not present in the dataset) for the derivationally related form (R), instance _hypereyyppymmmm (S) and synset domain topic_o of (C) relations at range of probabilities domain_topptt
(a (o(o(e,, r.o)) (0.4, 0.6. 0.8, 1)), predicted by each model. True triples indicated in bold; instances where a model predicts an entity is related to itself are indicated in blue.
12

---

Published as a conference paper at ICLR 2021
NN NN. NN J.C.c. am. mation NN JJ)
(thomas eeuunagnnn NN J. church, Juther -NN
unmology. JN J)
(prance VB. 4, equestiin spor, sport.y (N_
tworsshpp VB. P. worship. V8. I)
(awn Low VB 18_1. sorr oot VB. JI
Imake full VR J. make full VR 1) funlize, VB. J. wiliie VB. 11
feeolngy NN J. geology NN F)
towningy MN 2, voology NN 2)
tunnnolog monnoooge...oo NW_ aranolog JN I)
(helsinl NN J. Jrrhon center NN I)
Imuenhein NV stochale NessaaleNN NN (2
trictor teittr__uuee Jugo NN I. novel novel (NL))
(prpphex NN 1.book, NW NV_10) 10)
I. minsrnll VR 11
(orgemn NN J. Isllmd MN_.
(llbe NN J. dlsprrageeett NN 1)
trollon oflloff NN J, tranpprrt AN
dihed NN J. 1.trnducement NN J)
frrot V8 2. ride ride_hhooee Jorsebuck VB. J]
(prence V8. ride. Jarsebaxt Vi. I)
VB !) igaliop VB_,, ride Jursehack VB_11
(hryahid NN 1, mynhology NN 2)
puppon NN J. nippon AN N1.
troned Vi m.mw VW 11
(trail VB. 2, t.aill VB 2)
VR 13
tsell s,lliing NNN
NN 12, hadminton
NN
(central Icnntrllam Lame nearica
tregiomentonus
twondy gamnee tooody_gamnee
(cmmrol aamrri__maa america NN J.S. cm. pation NN Jj
(kovarc VD dein sentation NY 1)
(acleme NW _5, schomaticaion NN 11
(maicolm LNN passive resiseer NN !!
(talwon NN mational capital JN (JI
(thoma aquinas NN JN_. saint Vie 2)
(errile. cresent Bancenr_Mii NN J. astin mation NX
frelatinning NN relenionsiip NW 4)
coomotogiiigg..o NN J. linguisti (NJI
foothall game NN 1)
(miccani, N 2 human actiity NN_ 1i
Vn 3)
(or guroryy M. L.N J. aportie NW 3)
(robent Lowen. .NV J. indherry MNNNN
(brvoiiid J. mythhloggice VB VB_2) (bryahild
(land V8. J. vegetuble oi NN f)
(prophets AN J, gospels NN 1)
N. NN. Ji
(hirm VB 12, culivvation NN 21
(relationship NN 4. binlogy NN
(vewerr NNN sitcher NW JI
(Iraia VB. 18,3, training J)
VR
lde mile MN MN J. dance VB ()
_N NN J. myyth NN J)
(drive NN 12, drive VR 8)
(ordoin VR. 3. vot V8 J;
(tread MN J. step NW 9)
(truhh NN US abulitioniom
(libel ANN J. suill. V8 3)
drow
(rgiister NN fllele
Table 10: "Other" facts predicted by MuRE,-
(avvis NN 2,
[reed__N
(oonggge
(ricuan Inchmmester f. JNN J. Aichmologicnl 11.2)
sregurr nf AN cononicaiion NN !I
fihomuu agginos NN orciteishop NN 1I
(thomas sporte NN sonctffy vB__) !I
(marshail NN 2. genent office NN
(tuborder NN teconomic group AN
(extropllatin NN J. mutli NN MalliaNN... 1I
textropolate VB.5 3,.3, syathesie NN 3) (omatngii NN feranh VB. A. grind 2B_2)
muchinstion NN /
(syntheticll I1 synthesizeer AN 21
(realignaann NN. 2, enthedimmm NN .3)
(uivan NN J. asian nution NN D)
(ot grogory of NN J. sninn vB_.. 22_2)
greger LNN LNN_ bichhp... NN 1))
fnemman, AN 2. primutesiig ship IN
(drow VB 6, creative preathe__eesoe person NN_) D)
(frr sacher NN J, haligume. NN.2. 2)
(waint V7 2, catholic church NN 1)
Espread. V8 5, circwlarize V8. 3)
Mo AN. funl VB 5)
(remon NN I senmon NN 11
s.shniing NN
Eypile Va S. seeuyy Vi 2)
drow VB 6)
fatchenize V._.. modiff VB
11_2
(malle J1 muscaline 11.2)
(ffre 188 3, zoology NN 2)
1. vanquish
(sell, J. VB. J)
olo(e...ee.) t: 0.6
Inceommict
CiBo_VBB4 VB
(flam
flicking
idran
(trial impression, sasssssoooN NN J. puvol VB Ji (friend the court NN I. war muchine NN. J)
(muliin andtyss NV.. 1. applied math IN (sell VR 1. mammmctio NN 1)
finrrveet MY I. actentific discppine AN. .1)
baraning NN J. patellor rnfixx NN II)
(thomass thooass_agiion aigiina NN 1. monryr NW 2)
frolcamo bland Ny J. volcun AW 2)
reape horn NN_ I. arbon, center NY J) (bergen MergenNN_L.. NN. J. national comaaraap supital NN 17
dd mille NY J. dancinng moister NN J)
NW_). ())
(temple NNN place of warship AND
Hooorelllttion NN 2. sutellite NN 3)
fatruuum NN 22 social vocial_roopp group NN !) (dued Vi scufle AN
(maraihall _N__ N._2. generatdip NN_!)
iuiscomain NN 2, state ccppiaal NN (prustia
requalizer NN 2, set off VB.5 5;
(shrink YB 3, subtrrction NN
(contiieee_ VB 10, praceed VB. !!
(mamsn NN J. vonturer NN 2)
1. stockade NN
rntymologiing NN 1. law NN P7
fupport VB 6 defend Va 5)
Dihuttrr NN NN MNI..fill fill up VB 3)
(jow NN spiral yn J)
(oegean Jeggenn_ro___ NN J. arggon
(wrlie VB & tope VB 3)
(libel J. slur_NN 22 2)
(drow. 6. represent VB.
of ts
Relation (Tpp))
derivationally related form (R)
symet domain topic of (C)
iestance hy hypenym (8)
13

---

Published as a conference paper at ICLR 2021
Ir. r.e. hyd NN J. iingectorr general NN J)
fr tr bynd NN chief of sall [add LNNNN (central Laeerica NN I. l.c. aam Jation NN 1)
(malcoio x NN invionmentaai NN 3)
rthomas thhmmss_ggiiee Lignina NN I. charch father JNN II
(morr Imomik_lggen alsemm NN J. d.aggnnolezonn Aotoootzztoion... NN ()) (cabinetwokk NW 2, woodwarknng NN J)
(etymologiing NN exmologice, V8. 2)
(alliterate VB. J, vrrifier NN. J)
(goologs J, strructurll 11
fresect VB. J. comptattoon NN_2) 2)
(righht ficlier NV tofgaold NN I)
fllcutenane _NW W. capuair NN I)
(exeatvation NN I, enunleat Vn. 2)
(helsinhl NN J. uronn conter NN_D
the mecarene NN J. Save V8. 71
u.rhhnn center. AN
thhe nuturene /N J. christion HIII)
the munearenee NN W. d..e_in Lino NW
ietymologiinng NN derive VB.3).
(relationship 4, relation NN
(drive ddriv__NN_ ANN 12, badminton NN (]
frahinetwork NN 2, hottom VB. I1
fcahinenwor JW 2. apholteer VB. 11_1
(mariiion NN_3, mon NN 4)
(aiint NN sauctify VB 11
(hole out Vn. 1. h.hle NN 3)
(bat VB 4. slope 2)
(tread AN J. stop, (6.))
(fat Jl U.S. matte NN 2)
at nlo(c.. 62)
(irth capinl
NoW (copina of gambia NN J. urbon center NN 1)
NV 1)
Orunsyhaation plpp alpp__NN NN J. urbon urron_ssonnn senter NN_I)
(brundebburg NN J. .urbon semier NN I)
ne NN_. wrboon senter NN () (morth river NN urhan (oole I, urbun center NN I)
(mnlcolm X_NN J. emaneipaaionn imanccpationtt AN I)
(etymologtcin NN J. expllamation NN JI
(ferry 3B_3, 3333,oww movel (public prosentor NN J. prosecue VD. 2)
(print VB 4, publishing houve AN I
(lthdiial energy NN charge NN 9)
(centrolizaaion NN NN_ unite VB. 6)
(oouth went africa NV offina NN
Viver-V__. urben, senter AN
(ring NN 4, chemical rhain NN 11
(dep sowph NN J, south [N_ I)
(abrink NN J. pedology NN J)
(faishh 6, finishing 2)
J. H.iinng NN 33 31
(consellation NN stor NV_
modify. VB
(oley VB 26, playing AN-3)
sarsing NN
(pucale VB 2,2,,000 sesrer NN 33
(oranndogy NN J. at N. 1)
1)
(re Vn 3, S.Maming NN 1)
(Bibet LNN J. lihel VB. J1
Table 11: "Other" facts as predicted by DistMult.
(trn V8 12_ 12, nil Vs.
thic Oii_AN NN 1. Miz VB 11
afole, fu)) ss
(exisronce NN NN
(ounckk VB
(alchemice.
Cauire
(misstouppl river NN american stute NN_)) is a. byrd NN J. commitsionns office NN 11 (koheuhawn NN urhan conter NN
(antorr. orgunism I. I. epipppppicccccccc II
center AW II
NNN NN J. mitlaryy advicor NY JII
(r .. bynd a. bynd NN (tampa buy NN J. urhan center NN 1)
nidir de NV F)
(idewater region NN J. south NN JI
(r a. byrd NN J. execurive officer MMNeer_N_. NN I)
(roll-on roll-off AN J. transport NN II (hib hich-ur-uile ur-aahrir NW I. asia NN !I
(slugger NN J. soffball _offbaalggomee gome NN I) (sermon
(asist_NN NN 2, softball game NN JI)
I, turann NN d)
(system AW 9, onderlinss NN_ J)
the gombia NN. J. africa NN J)
J. s.rroniie yn 3)
(councit NV 2.assembly NN 4)
(espose VB. 3, show NN 1)
(take takee_dom down VB. 2. pio V8. 2)
(wreadle. Vn wrester NN
(dnel VB J, slugfest NN 1)
(vocal NN 2. rook sar NN. 1) (omeliing NN J. wemn VB. 1)
(drwv VB 6, drawer drawer_NN NN. NY_3) 3)
(cream, V8 J. cream NN 2)
NN 11
(dioh, NN 2. sene MY 21
(spread NN strew
(thriughput NY J,
(ol fa)) ss
(iiiio mir NN 1.
(turn, V/
(marshall NN 2, licurmmnnt general NN D)
(Win VR.5 3, undercover work NN /)
(wiaconsin NN 2. urbon urbon_goonee goner I) 1)
(world war j.r...W JNN J. urbann center SN_1)
(st_pon pomppo__ KNy 22 enungliss NN (deep (depp_soull south NW J. J. barba center NN J) (mupite NN
(ocyean sea NV J. argean argean_istand intand Mand_NLL NN III
Lowpens_MM NNJ. wor wor_ of om Ind ind_DN IN_I) 1)
(slugger NN IN beseball player NN I)
(riaa impression NN J. pubbishh VB. f)
urhan conter NN 1)
(iiino NV. J. urbon conter. NN 11
(ight (ight_colone colonel NNJ V.L...ooomu calomel NN.I)
(HibeLAN J. practice of law law_NN NN 11
(print 4, publisher NN 2)
(erier_ NN 3. picchmm NN 2)
(vividners NN inbuue Vn. 3)
(sea New_NN NN_.. taraa NW
(alhait_NN NN 2. acidify V8. 2)
2)
(shin VR. 1, hedginng NN 2)
(ahidjon NN cot d'iwnire NN
(limtation NN 4, trammel LomeeLV V8.2)
(murse. VB J. nursing NN 1)
(temon MM J. proghery. 2)
(mn NV J. J. chemistr NN 1)
(metrify VB wriff, VB 11
(lurn. VB 12, plowman NN_D) ()
(cownt VB 4. recie VB 2)
(see Lee_VB VB. 17, indersstnnd
(play VB tum
aidlfu. f2))
(Type)
symsel, domainn topse of
decivantioally relined Form
instance
hyponmpm (S)
(C)
14

---

Published as a conference paper at ICLR 2021
(phionntropir. NNN J. philontmopii, JNL XN_I)
NN.
(chow NN 3. women's 2. rightint I. (argoin sou NN J. argoon ishne NN J1. aqainos NN
(dpnamizier NN dynamizar NN.1) 11
(position VB. J. 1.plccement NN I)
venturer NN 21
(omuggle NN J. squeeee VB 8)
(atocilist NN steciir NN 11
(c bynd NN J. viege NN 1)
(repenione 11 1. I.repenton (tread Nre NN J, strp VR. 2..7.
(love VB lover_NN
(doparted NN 1, 1,die
(venturer Wenntver_NN223 MN
(thomas the aponte NN I. evangelic NN 2) (mccarmick NN technologal (M gregory NN J. charch father NN_
Imccormick NN 1. arms manufacture NN
(okee Tanke_ohongg choncceene..... VW J. venture NN J) AN J. hi up. VB 3)
1,mip VR (uppprrt VB. 6, "inddrsement NN I)
Moderate VB 1, confederatin, NV_)) (matiiator
(denote VB. 1B_3, commilaation.... NN
(wait V8 J. supporter NN 3)
(trembeer NN. J. vibvate VB 1)
(ake over V/. 6, rxturrn NN 7)
(cowpens NN J. siege NN 1)
fealt ANI 3. leave. V8 1)
Table 12: "Other" facts predicted by TuckER.
pacpippp_iivereee fiver NN J. americcn, sare NN f
N NNL electrical Cnngricul_nnginer engineer_ Nonieer_NNN.. I)
(musiccal Instrument, NN J. write VB. 6)
tynncrooncze 2, symchruscopp NV_D)
firich papital NN J. urhon center NN I)
(thoms: the apocile NN J. aponde NN. 2)
AN 1. moggldddee NN
NN.
(indicate. VB 2, iuttiftcuioo. SoouuoooNNN NN_2)
bwalt Vn. timoner frome NN. II
(leaching NN J. perculare VBB.).
(thamas aigiina NN J. hishop NN
takanas NN I. pointer JNN 1)
budwinton I)
terop VB 2. Sttruggle_V. NN 2)
recorder Ni
(st_poodd apocile NN.3; 31
(munh NNN 2. mill VB 2)
(uve 5, utility NNW 2N
3, wrp up
mnple (. ss
(micormick
(mcorrmick
(missisiipp))
(tope VB
(üning
Oibemnx, the spentee NN J. church, church-foane fother ANN J gregory
(roll-on roll-off NN J. passuger yehicle AN
NN J, sermonizer NN I)
J. emancipationsl AN D)
(manafacture_ VB.J J. preverication. NN_D)
(rabert Lowen, ower, NN_ movie moler NW_J) !)
(ralbon rollouff NN J. motorcar IN
date VB 5, geologleat dating NN 1)
thelennki NN J. urhan conter NN 1)
(indicattion NN sigmalite Vn 2)
(rurn over VB 3. rotation NN-3)
iuniiiictionn NN 1. apology NN 2)
(iber roshd, NN J. polagogne NN f
trypponist NY J. gone NW 2)
(ddetiiiation, NN nome VB
(aabber NN J. ththent VB.S. 5)
ddepp nouth NN J. south NN 1)
(st poud NN 2. organis NN 11
(beeess NN J. monk IN f)
for paul NN 2. philoapher AN
libel LNN J. legislature, Ma NN_
(yyo NN J.J.so sort 5)
(uet eublem
(malcolm
symvet, symeet_dommin domain
derivationally. related, Form (R)
Relation
lnsturce hypernym
topic of
(C)
15

---

Published as a conference paper at ICLR 2021
iurriialeneee AN J. arbitrary JI.I)
inmsalzzaiion JNJ. J. jont VB
V& 1
Hrawway NN. 2 naveary NNV 21
(toufffe VR 3, whine Maino_NN NN 1)
(varm VB V.J. pining NN D)
fxprrk NNN J. L.parkle V28 11
(derire NN 2, hope VB (122
(Draa NN 1. step Vn 3)
fypoliation NN 2.
J.
Iin out VB J. somutic coll suclear tronuplanttiion
(r. e. by_byrd NN. J. nomcommiissoond officer NN. I)
(ouptaa of gombia NN J. urbon conter NN 1)
(volcano islande NW J. urbon conter NN 11
(wovld wwr.... JNN. nitched batll NN 1)
(rish cupitel N._ J. arrun center IN
(avier 2, am. fonthell gome NN 1)
oon--bulf NNN_1
(helsinki NN J. urbun conter NN 1.
(aluggrr NN J. s.ffball gome NN_D)
dighiing NN 4, set on fire VB. 1) (symphuiongn.
fepprove V88 J. talking jo NN I)
(gark. AN. 1. scimiillte VB. I)
(ubllit NN 2. funcciion J1 1)
NN
(great Jakes NN. J. mivr mevr_WW JJ J)
americon Moree NN.
(drive NN f2 court gome NN 1)
(drvee NN f2, bodminton NN_1. 11
(omelling NN wind Vi 4)
(gpring NN 6, haralle VR. 18.1)
(hheremn NN J. moot NN 1]
(jud J1.3 vabidtyy NN 1)
AN
focil, Vo. J. o.ffouee NN 3)
Table 13: "Other" facts as predicted by MuRE.
(olee.. (lc...et (6)) to 0.8
(antain VB
(nipproc
(nippre
(conrral america NA. J. central aomerilmmmeric amvrrcon NN JI
b.bthheeoonnn reformer NN
ferot laker NN 1. conndin province NN 1I tromsylaaiion
urban. center NN_.) (gettaborg NY 2 sieg NN I)
(wisconsin NNN souguphicol reglon NN J)
AN I)
gregozy NN church father NN
tonuth tsnul__oree Jorca NN J. affrican pation (N
mmaicalin x NN. J. paszive pasiive periser NN J1
lopodle_ AN_ weserm church NY I) faxsss NY
(word pleture NN J. sketch Vi
(develop VD 10. adjustment NN 40
Jeloxs VR 3. commenttry NN ()
frielate_ vinlation AN
(uffiorate Yn J. strungler free NN
foumber VB. 3, point NN (2)
arganic (plication NN J. n.itt VB 4)
Apolit up VB 3, seporation AN 51
(lox angeler. NN. J. part I)
VB trnncription temple NW mus mnniiimiom. NW
5)
fassis NN 2. hockey NN 1)
relationsip NN 4, bology NN I)
(Init (nrit VE 1 equestiin sport NN
trna tmna NN 1. chemistry JI
(assir NN 2, soccer NN 1I
(asiid 2, fouthell gome NN II
(plicatiion NN wrinkle
NN
atei r.ta)) te 0.6
(derolop
fmolcalm
tst gegery of J. vicur of chrisa NN. [) tnggonn AN 1. nalian region NN 13
fhue bynt_NW NN.. J. committions committiond.coo cofeelfffffcc officer NN JI bynd fomai howe AN penioonaa
(unplemant JI J. unpalanne supulanbeennn.n.. XNLI) I)
imundaie: N.V J. nunional aotiinal_supppcn cuporal NN. 1; NMV
touclear deverence NN 1. law NN freception NN 5, bowball game NN 1)
pphooryuhess AN chemisrry NN. II)
fremitic__ deiry AN. J. miwhology NN.
(testify 1n 2, Information NN 1)
(comece VH 6, comvrrging NN /)
imulcodm Matioommu_A M.N.I. J. hipuer AN J)
naurene NN judintom NN
Siner_NNJ Ji
pycon,, !Y
fnon Hine NN. J. arbon center NN
NN 1)
(brynhild NN J. mythology. NN 2)
(asin NN coart gome NN_
fourround VB. wall NN 1)
(love VB 3, enfoyment NN 2)
imagnimdee. JN_ [
(conncet 6, connction
(opoatee 18 4 p5 NN 1)
(murker_ VB J. imuie NN 4)
2)
NN 1)
fhodle NN parable NN
fold JoidJine Line state NN NN_NN.N. I.
(1 (e 0.4 0.4
(ihee Ny criminal
fsell J. low NN
(dugger NN I. I.
loportee Vn 4,
(mobert
Relation (Type)
synset domain topic of
dativationally related, Jonm (R)
hyperrym (S)
instance
16


Reviews:
[{"review_id": "fQsJIjmExaO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary and Contributions\n\nThe authors study the latent semantic properties of word representation models by categorising relations between entities.  The goal is to show that word embeddings and knowledge graph representations learn a common latent structure even if both types of models have different learning objectives.  The main contributions are the mapping of relations between subjects to object word embeddings, categorisation of such relations, and evaluation of the state-of-the-art knowledge graph representations. The study shows that knowledge representation models follow the defined relation conditions.\n\nStrengths\n\n- Clear description of background knowledge needed to understand the proposed study.  \n- The authors perform a comprehensive comparison across different knowledge graph representations.\n- The findings show that there is a connection in the lower dimensional space between word and graph representations.\n\nWeaknesses\n\n- It is not clear which particular framework is used to define the hierarchy of knowledge graph relations.\n- It is not discussed the possible benefit of the learned latent structure of knowledge graph models for the performance on downstream tasks, e.g.  text classification, or natural language inference. \n\nQuestions to the Authors\n\n- How the main contributions relate to the experiments P1 and P2?\n- Could you elaborate on how you define the hierarchy knowledge graph relations.\n- Which is the relation between the used knowledge graph categorisation and the related work?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_stedc535wj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recent works toward the understanding of word embeddings can explain how semantic word relationships, such as similarity, analogy and paraphrasing are encoded as low-rank projections of high dimensional vectors of co-occurrence statistics (Allen et al., 2019). Thus, the semantic relationships correspond to linear relationships of word embeddings. This paper builds on this understanding of (PMI-based) word embeddings aiming at the task of understanding the latent structure of low-rank knowledge graph representations. The authors draw a parallel between the embeddings of knowledge graphs and words under the premise that fundamentally the same structure of relations is captured in different ways. Strong evidence to this premise is provided by starting at encoded semantic relations of word embeddings generalizing them to three types (R,S,C) of knowledge graph relations. The authors analyse the performance of different state-of-the-art knowledge graph models and identify the best performing model per relation type. While a multiplicative model performs best for R-relations (highly related), an additive-multiplicative model should be used for S- (specialisation) or C-type (context-shift) relations. These results correspond to the predictions made beforehand and the theoretically derived loss functions based on the respective conditions of each relation type.\n\nPros:\n•\tThe paper is technically sound, well written and organized and free of typographical errors\n•\tIt focusses on the timely and interesting problem of understanding the latent structure of knowledge graph representations\n•\tThe key strength of the paper is the idea to categorize relations between entities based on the geometric properties of relation representations of word embeddings\n•\tThe theoretically derived loss functions are a strong evidence to why different kind of knowledge graph models perform unequally on the defined relation types\n\nMinor comments:\n•\tThe relation types in the text (four bullet points, p. 4) should perhaps mirror one-to-one the types in figure 1\n•\tContext-shift relations are explained as \"subject to synonyms\", a specialisation should perhaps be explained as hypernyms and context shift e.g. as meronyms\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rb0isLk9cVD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to establish a theoretical basis for geometric properties of knowledge graph relations and embedded entities by comparing knowledge graph embeddings with word embeddings. Using the insight that the semantic properties of PMI-based word embeddings manifest as linear geometric relationships, they view and compare the relationship embeddings derived from different knowledge graph embedding schemes in this way. \nThe analysis claims to show that when the KG architecture conforms to the presented relation types and conditions (divided into similarity, relatedness, and context shift types), it has better performance of link prediction for that embedding scheme. \n\nThe empirical evaluation focuses on comparison of 4 embedding schemes that have linear transformation score functions (additive, multiplicative, both) on WN18RR and NELL-995 relations for link prediction on several examples of the relation types.\n\nOverall, the paper is well-motivated, cites relevant literature in the theory behind word embeddings, and is generally clearly written. It has a useful proposal for the types and conditions of the three relation types and clear hypothesis for the performance of knowledge graph relation transformations that have certain properties.\n\nHowever, the empirical evaluation does not seem to completely support the claims. TransE is obviously lower performing across relations, but MureI seems quite close in most cases to models that involve multiplacative relatedness, so it’s not obvious to me that MureI performs worse. Further, the summery suggests that DistMult is preferable for type R, but MuRe appears to do equally well or better on most cases, thus it’s not clear under what circumstances (what dataset dependent factor) would point to not choosing MuRe. \nI would expect to see a starker contrast between the performance of the different models per claim type to support the dataset dependent statement. Perhaps another experimental setting, like comparison on non-linear transformation, or other examples, would help support that claim. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7pgcuislNmb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Based on PMI word embedding, the authors categorize the knowledge graph relations into three types, which serve as the foundation of knowledge analysis. This paper is not well-motived but presents the methodology, well. However, nothing in this paper surprised me, because this seems like a ````''regular'' research in this field.\n\nMajor Concerns:\n1. The motivation is not crystally clear. I suggest the authors explain the motivation in detail, e.g., why different embedding are fundamentally the same (strong claim but no evidence).  I suppose the author would like to align the structure of linguistic semantics and knowledge semantics. However, there is no sufficient evidence for me to accept this alignment. Can you give me some examples or strong evidence to claim the joint learning is not incremental. \n\n2. This paper is indeed fine, except for motivation. Good presentation, the clear methodology, and promising results. However, there are plenty of researches to join the textual info and knowledge info together, which makes this paper not very novel. Besides, based on PMI embedding and relation categorization are still very regular in the field. Thus, I doubt the novelty of this paper. However, compared to these papers published in this field, I think this paper can be accepted. This is the reason why I rate 7 but I still got unsatisfactory about the novelty.\n\nMinor Concerns:\n1. Fig.1: Please specify the legend clearly. \n2. Table 2: For more beginning readers, please revise the examples to human-readable.\n\nDiscussion:\n\nI think the categorization shall benefit the performance most when the difficulties between relation categories are balanced. Does this sound like a correct claim? because in your experiment, I found R is the easiest while S is hardest, and the difficulty different for the two types is large. I don't judge this question. I just want to provide a new idea for your paper to improve.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the recent theoretical understanding on geometric properties for word embeddings to relations and entities of knowledge graph. It categorizes relations into different types and derive requirements for their representations. Empirically they experiment several graph embedding approaches and show that when the loss function is aligned with the requirement of the relation type, we can achieve better performance.  The reviewers generally find the paper to be solid, well executed and provides useful insights. The authors are encouraged to strengthen the discussion of the motivation of this work, and improve the presentation based on reviewers' comments. 
Decision: Accept (Poster)