Toward Understanding The Effect of Loss Function on The Performance of Knowledge Graph Embedding
Authors: Mojtaba Nayyeri
Chengjin Xu
Yadollah Yaghoobzadeh
Hamed Shariat Yazdi
Jens Lehmann
Venue:  (2020)
Keywords: Knowledge graph embedding
Translation based embedding
loss function
relation pattern

Abstract: Knowledge graphs (KGs) represent world's facts in structured forms.  KG completion exploits the existing facts in a KG to discover new ones. Translation-based embedding model (TransE) is a prominent formulation to do KG completion. 
Despite the efficiency of TransE in memory and time, it suffers from several limitations in encoding relation patterns such as  symmetric, reflexive etc. To resolve this problem, most of the attempts have circled around the revision of the score function of TransE i.e., proposing a more complicated score function such as Trans(A, D, G, H, R, etc) to mitigate the limitations.  In this paper, we tackle this problem from a different perspective. We show that existing theories corresponding to the limitations of TransE are inaccurate because they ignore the effect of loss function. Accordingly, we pose theoretical investigations of the main limitations of TransE in the light of loss function. To the best of our knowledge, this has not been investigated so far comprehensively. We show that by a proper selection of the loss function for training the TransE model, the main limitations of the model are mitigated. This is explained by setting upper-bound for the scores of positive samples, showing the region of truth (i.e., the region that a triple is considered positive by the model).
Our theoretical proofs with experimental results fill the gap between the capability of translation-based class of embedding models and the loss function. The theories emphasis the importance of the selection of the loss functions for training the models. Our experimental evaluations on different loss functions used for training the models justify our theoretical proofs and confirm the importance of the loss functions on the performance.



Under review as a conference paper at ICLR 2020
TOWARDS UNDERSTANDING THE EFFECT OF Loss FUNCTION ON THE PERFORMANCE OF KNOWLEDGE GRAPH EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge graphs (KGs) represent world's facts in structured forms. KG com- pletion exploits the existing facts in a KG to discover new ones. Translation-ban embedding model (TransE) is prominent formulation to do KG completion. De- spite the efficiency of TransE in memory and time, it is claimed that TransE suffers from several limitations in encoding relation patterns (such as symmetrie, reflex- ive relations etc). To solve that, most of the attempts have circled around the revision of the TransE score function, resulting more complicated score functions (such as TransA/D/G/Hin) etc). These attempts have totally disregarded effect of loss functions to this end. We show that loss functions are key factors in this re- gard and disregarding them results in conclusions which are inaccurate or even wrong. More concretely, we show that the claimed limitations are inaccurate, as the effect of the loss was ignored. In this regard, we pose theoretical investiga- tions of the main limitations of TransE in the light of loss function. To the best of far, this has not been comprehensively investigated. We show
our knowledge, that by a proper selection of the loss function for training the TransE model, the mitigated. That is achieved by setting upper-hound for the
main limitations scores of positive samples, defining the region of truth (i.e. the region that a triple is considered positive by the model). Our theoretical proofs with experimental results fill the gap in understanding of the limitation of translation-base class of embedding models and confirm the importance of the selection of loss functions for training the models and on their performance.
INTRODUCTION
Knowledge is considered as commonsense facts and other information accumulated from different sources. A Knowledge Graph (KG) is collection of facts and is usually represented set of triples (h, r) where h, are entities and r is a relation, e.g. (iphone, hyponym, smartphone). Entities and relations nodes and edges in the graph, respectively. As KGs are inherently incomplete, making prediction of missing links is a fundamental task in knowlege graph analyses. Among different approaches used for KG completion, KG Embedding (KGE) has recently received growing attentions. KGE embeds entities and relations as low dimensional vectors known as embeddings. To measure the degree of plausibility of triple, a scoring function is defined over the embeddings.
Embedding model, (Bordes et al. 2013) is one of the most widely used for every positive triple r,, f) where h, r, t E Idd are embedding vectors of head (h), relation (r)) and tail (t) respectively. TransE and its many variants like TransH (Wang et al., (014)) and TransR (Lin al.) (0015)), under- perform greatly compared to the current ofate-of-the-an art models. That is reported to be due to the limitations of their scoring functions. For instance, (Wang et al., 2018) reports that TransE cannot encode a relation pattern which is neither reflexive nor irreflexive.
TransE, Translation-base
KGE models. The original assumption of TransE is hold: h +
In most of these works the effect of the loss function is ignored and the provided proof' are based on the assumptions that are not fulfilled by the associated loss functions. For instance (Sun et al. 2019) proves that TransE is incapable of encoding symmetric relation. To this end the loss function must enforce the distance of | r t|| to zero, but this is never fulfilled (or even approximated) by the employed loss function. Similarly, (Wang et al.) 2018) reports that TransE cannot encode a relation pattern which is neither reflexive nor irreflexive and (Wang et al. 2014) adds that TransE cannot

---

Under review as a conference paper at ICLR 2020
properly encode reflexive, one-to-many, many-to and many-to-man relations. However, as mentioned earlier, such reported limitations are not accurate and the problem is not fully investigated due to the effect of the loss function.
In this regards, although TransH, TransR and TransD (Wang et al. 2014 Lin et al. 2015b Di et al, 2013) addressed the reported problem of TransE in me-to--many many-to- soly-ttonnn one, many-to to-many and reflexive etc, they were misled by the assumption (enforeing ||h + tll to be zero) that was not fulfilled by the employed loss function. Considering the same assumption, (Kazemi & Poole 2018) investigated three additional limitations of TransE, FTransE (Feng et al. 2016), STransE (N (Nguyen al. 2016), TransH and TransR models: (i) if the models encode reflexive relation r, they au- tomatically encode symmetric, (ii) if the models encode a reflexive relation r, they automatically encode transitive and, (iii) if entity 6 has relation r' with every entity in A E & and entity ez has relation r with one of entities in A, then 62 must have the relation with every entity in A.
Assuming that the loss function enforces the norm to be zero, the aforementioned works have in- vestigated these limitations by focusing the capability of scoring functions in encoding relation patterns. However, we prove that the selection of loss function affects the boundary of score func- tions; consequently, the selection of loss functions significantly affects the limitations. Therefore, the above mentioned theories corresponding to the limitations of translation- els in encoding relation patterns are inaccurate. We pose new theories about the limitations of TransX(X=H,,,R. etc) models considering the loss functions. To the best of our knowledge, it is the first time that the effect of loss function is investigated to prove theories corresponding to the limitations of translation-bant based models.
embedding mod-
In a nutshell, the key contributions of this paper is as follows. (i) We show that different loss functions enforce different upper-bounds and lower-bounds for the scores of positive and negative samples respectively. This implies that existing theories corresponding the limitation of TransX models are inaccurate because the effect of loss function is ignored. We introduce new theories ac- cordingly and prove that the proper selection of loss functions mitigates the main limitations. (ii) We reformulate the existing loss functions and their optimization problems as an standard constrained optimization problem. This makes perfectly clear that how each of the loss functions affect on the boundary of triples scores and consequently ability of relation pattern encoding. (iii) Using symmet- ric relation patterns, we obtain proper upper-bound of positive triples score to enable encoding of symmetric patterns. (iv) We additionally investigate the theoretical capability of translation based embedding model when translation is applied in the complex space (TransComplEx). We show that TransComplFxx is more powerful embedding model with fewer theoretical limitations in encoding different relation patterns such as symmetric while it is efficient in memory and time.
2 RELATED WORKS
Most of the previous work have investigated the capability of translation- class of embedding models considering solely the formulation of the score function. Accordingly, in this section, we review the score functions of TransE and some of its variants together with their capabilities. Then, embedding models emphasized in
in the next section the existing limitations of Translation recent works are reviewed. These limitations will be reinvestigated in the light of score and loss functions in the section 4
The score of TransE (Bordes et al.
2013 is defined as: (h, | - t/. TransH (Wang
al. 2014) projects each entity (e) to the relation space (el e w,ew, ). The score function is defined as [- t) = l + r tt TransH can encode reflexive, one-to-many, many-to-one and many-to-many relations. However, recent theories (Kazemi & Poole 2018) prove that encoding reflexive results in encoding the both symmetric and transitive which is undesired. TransR (Lin ct al. 2015b) projects each entity (e) to the relation space by using a matrix provided for each relation (e] eM,,, M, xd.). TransR uses the same scoring function TransH. TransD (Ji et al. 2015 provides two vectors for each individual entities and relations (h, h r, rp. t, tp). Head and tail entities are projected by using the following matrices: M, = r
- h2 + mnx M, = r mmxn
The score function of TransD is similar to the score function of TransH.
RotatE (Sun et al. 2019) rotates the head to the tail entity by using relation. RotatE embeds entities and relations in the Complex space. By inclusion of constraints on the norm of entity vectors, the
2

---

Under review as a conference paper at ICLR 2020
model would be degenerated to TransE. The scoring function of RotatE is f+(h,t) t) || | t|| where h, r, t C², and is element-wise product. RotatE obtains the state-of-the-an art results using very big embedding dimension (1000) and lot of negative samples (1000). TorusE (Ebisu & Ichise 2018) fixes the problem of regularization in TransE by applying translation on a compact Lie group. The model has several variants including mapping from torus to Complex space. In this case, the model is regarded as a very special case of RotatE Sun et al. (2019) that applies rotation instead of translation the target the Complex space, According to Sun et al. (2019), TorusE is not defined on the entire Complex space. Therefore, it has less representation capacity. TorusE needs very big embedding dimension (10000 as reported in [nbbisu Ichise (2018) which is a limitation.
THE MAIN LIMITATIONS OF TRANSLATION--ASSE EMBEDDING MODELS
We review six limitations of translation based embedding models in encoding relation patterns (e.g. reflexive, symmetric) mentioned in the literature (Wang al. 2014, Kazemi
Poole 2018 Wang
et al. 2018 Sun et al. 2019).
Limitation L.1. TransE cannot encode reflexive relations when the relation vector is non-zero Wang et al. 2014).
Limitation L2. TransE cannot encode a relation which is neither reflexive nor irreflexive. To see that, if TransE encodes a relation which is neither reflexive nor irreflexive we have h, + h and h2 + r + h2, resulting
0, r + which is contradiction (Wang et al. 2018).
Limitation L.3. TransE cannot properly encode symmetric relation when of 0. To see that (Sun jet al. 2019), r is symmetrie, then we have: r = t and + h. Therefore, entities appeared in head tail parts of training triples will have the same embedding vectors.
and so all
The following limitations hold for TransE, FTransE, STransE, TransH and TransR (Feng et al. 2016
Nguyen al. 2016 Kazemi & Poole 2018):
Limitation L4. I reflexive on E E, where the set of all entities in the KG, then must also be symmetric.
Limitation L5. I is reflexive on A E E, must also be transitive.
Limitation L6. If entity ej has relation with every entity in A E & and entity e2 has relation with one of entities in A, then 62 must have the relation r with every entity in A.
OUR MODEL
TransE and its variants underperform compared to other embedding models due to their limitations we iterated in Section 3 In this section, we reinvestigate the limitations. We show that the cor- responding theoretical proofs
inaccurate because the effect of loss function is ignored. So propose new theories and prove that each of the limitations of TransE are resolved by revising either the scoring function the loss. In this regard, consider several loss functions and their effects on the boundary of the TransE scoring function. For each of the loss functions, we pose theories corresponding to the limitations. We additionally investigate the limitations of TransE using each of the loss functions while translation is performed in the complex space and show that by this new approach the aforementioned limitations are lifted. Our new model, TransComplEx, with proper selection of loss function addresses the above problems.
4.1 TRANSCOMPLEX: TRANSLATIONAL EMBEDDING MODEL IN THE COMPLEX SPACE
TransComplEx translates head entity vector the conjugate of the tail vector using relation in the complex space (Trouillon et al. 2016). Assuming h, r. E Cd be complex vectors of dimension d, the score function defined as follows:
(h, h
Advantages of TransComplEx: We highlight four advantages of using the above formulation. (i) Comparing to TransE and its variants, TransComplEx has less limitations in encoding different re- lation patterns. The theories and proofs provided in the next part. (ii) Using conjugate of tail
3

---

Under review as a conference paper at ICLR 2020
vector in the formulation enables the model make difference between the role of an entity sub- ject or object. This cannot be properly captured by TransE and its variants. (iii) Given the example (A, Like, J Jwentus), (J Jwennus, hosPlayer, C.Ronaldo). that Conolaldo plays for J Juventus may affect the person to like the team. This type of information cannot he properly captured by models such as CP decomposition (Hitchcock) (927) where two independent vectors are provided Kazemi & Poole 2018) for Juventus (for subject and object). In contrast, our model uses same real and imaginary vectors for Juventus when it is used as subject or object. Therefore, TransCom- plEx can properly capture dependency between the two triples with the same entity used as subject and object. And finally, (iiii) ComplEx (Trouillon et al., 2016) has much more computational com- plexity comparing to TransComplEx because it needs to compute eight vector multiplications to obtain score of a triple while our model only needs to do four vector sunmation/subrractions the experiment section, we show that TransComplEE outperforms ComplEx on various dataset.
4.2 REINVESTIGATION OF THE LIMITATIONS OF TRANSLATION- MODELS
The aim of this part to analyze the limitations of Translation based embedding models (including TransE and TransComplEx) by considering the effect of both score and loss functions. Different loss functions provide different upper-bound and lower-bound for positive and negative triples scores, respectively. Therefore, the loss functions affect the limitations of the models to encode relation patterns. In this regard, the existing works consider a positive triple (h, r, and a negative triple of (h', r, (') to satisfy some assumptions in score function. For instance in TransE where f- - ||h + r t|l. it is expected that f- = 0 and fr > O. Unfortunately, as we show later, this not be fulfilled (or even approximated) by using the proposed loss functions (e.g. margin ranking loss and RotatE loss).
To investigate and address the limitations, we propose four conditions (Table-D for taking a triple as positive or negative by the score function. This is done by defining upper-bound and lower-bound for the scores. We show that these conditions can be approximated by proper loss functions, and in this regards we propose four losses that each will handle one of the condition.
Table 1: Region of truth and falsity.
Condition
Positive
Negative
Y1, 72 E R
(a)
(r 1) n,
f,(h', (()) 1 72
m = 0, 72 0
(b) r, t)
(.(htt)) 72
72 > 7/1 > 0
(c) (h, Sm
f((hi(() 72
72 > 71 > 0
(d) (h, S (1chee)))
fr(h',t' 2 (2(h.rst)
72(har)) > 71(h..))) >
To better comprehend Table T we have visualized the conditions in Figure 1 The condition (a) indicates a triple is positive if + r = holds. It means that the length of residual vector i.e. is zero. It is the most strict condition that expresses being positive. Authors in (Sun et al. 2019 Kazemi & Poole (018) consider this condition to prove their theories as well as limitation of TransE in encoding of symmetric relations. However, the employed loss cannot approximate (a), rather it fulfills (c), resulting the reported limitation to be void in that setting.
Condition (b) considers triple be positive if its residual vector lies on a hyper-sphere with radius n. It is less restrictive than (a) which considers point to express being positive. The optimization problem that approximates the conditions (a) (n1 0) and (b) (71 > 0) is as follows:
ming,, E E (r( n, (h,r,1) E S+ fr('' 8') 2 72 En, (N' r, t') E- En. 20
where St, are the set of positive and negative samples respectively. En.t are slack variables to reduce the effect of noise in negative samples (Nayyeri et al.) 2019).
4

---

Under review as a conference paper at ICLR 2020
> =
y12
x>0
Ma
(5)
(4)
140003
140090
20000
19900
Figure 1: Top: Visualization of truth region (positive) of a triple according to Table The residual
vector e, (a) becomes 0, (b) lies on the border of a sphere with radius 71, (c) lies inside sphere with radius 71, and (d) ((h,,r..() lies inside of sphere with radius "(hh, -1,4)) Bottom: The his-
togram of the scores of triples when TransE trained on WordNet (WN18RR) using the losses of Equationn 2 (?1 2) (?1 4). (1 4) and 5 (7 6) respectively. Each of the bottom figures is the approximation of the corresponding conditions (a) to (d).
b) y
y= adlr as1 a<1
y y=drrll,
u
rr
r
-r
Y
X
X
y
Figure 2: Necessity condition for encoding symmetric relation: (a) when a < 1, the model cannot encode it. (b) when a = 1, the intersection of two hyperspheres is a point. u = means embedding vectors of all entities should be same. Therefore, symmetric cannot be encoded. (c) when 1, symmetric can be encoded as there are more than one point in the intersection of two hyperspheres.
One loss function that approximates the conditions (a) and (b) is as follows. Please note that for case (a) we set 71 = 0 and for case (b) we set 71 > 0 in the formula.
Lap
Σ (Ail15-(h,1)
mll +
Σ
A2 max(12 (.(,)
(2)
(h,r.)))++
(h' (.'..') (A.
Condition (c) considers a triple to be positive if its residual vector lies inside a hyper-sphere radius n1 The optimization problem that approximates the condition (c) is as follows (Nayyeri al. 2019):
ming,, E(hr,,)sst E (r( 1) < n, (h,r,() E S+ fr(h' 2 72 En,ts (N,r, ,r E-
(3)
En. 2
The loss function that approximates the condition (c) as follows Nayyeri al. (2019):
C.= Σ
(Ai (axx 71,0) +
Σ
A2 max(%2 7/2 - f.(h(,)),)).
(4)
(h.r:))s++
The loss function which is defined in Zhou et al. (2017)) is slightly different from the loss in The former slides the margin while the latter fixes the margin by inclusion lower-bound for the score of negative triples. Both losses put an upper-bound for scores of positive triples.
Remark:
5

---

Under review as a conference paper at ICLR 2020
Apart from the loss 4 the RotatE loss Sun et al. (2019) also approximates the condition (c). The formulation of the RotatE loss is as follows:
ce RotaatE
Σ
(logo(7- f+(h,)) (R (Rss))
Σ
log (Qgo((((((t)))))
Condition (d) is similar to (c), but provides different n. 72 for each triples. Using (d), there is not a unique region of truth for all positive triples, rather for each positive triple (h, r, t) and its corre- sponding negative (h', rr,') t') there are triple-specific region of truth and falsity. Margin ranking loss (Bordes et al. 2013) approximates (d). Defining [x]+ = max(0, (0, æ), the loss is defined as:
La ΣΣ [fr + ( +()]+
To investigate the limitations, we must assume that the relation vectors is not null otherwise we will have the same embedding for head and tail which is undesirable. Considering the conditions (a) to (d), we investigate the limitations LI to L6 and we prove that existing theories are just valid under (a), which is not fulfilled under the given loss. In this regard we have the following theorem. For complete proofs, please refer to the appendix of the paper.
Theorem T1. (Addressing Ll): TransE and TransComplEx cannot infer reflexive relation pattern with a non-zero relation vector under (a). However, under (b-d), TransE and TransComplEx can infer reflexive pattern.
Theorem T2. (Addressing L2): (i) TransComplEx can infer a relation which is neither reflexive nor irreflexive under (b-d). (ii) TransE cannot infer a relation which is neither reflexive nor irreflexive under (a-d).
Theorem T3. (Addressing 1.3): (i) TransComplEx can infer symmetric relations under (a-d). (ii) TransE cannot infer symmetric relations under (a) with non-zero vector for relation. (iii) TransE can infer a symmetric relation under (b-d).
Proof: Proofs of (i) and (ii) are provided in the appendix. For (iii) we have:
Under (b), for TransE we have |h - t|| = -Yi and || |t++r
h| 71 The necessity condition for
encoding symmetric relation is |h+r-t] t||
||tt--hh h|. This implies ||b] cos(O ||t|| (O..)).
Let t u, by definition we have |u +r|- -., --
-urll r|] = 7/1 Now let 71 = al[r]]] we have:
[u|i + [[=aa)][r= -1-0))(r|=--222.17 =-(uur)
-
(6)
|u| (1 o =2(u,r)
Therefore we have: +1 az 1|=|= ([|||| (1- which can be written ||u||
-
(a² - 1) ||] To avoid contradiction we must have a > 1. Once a > 1 we have cos (f..r) = . T/2. Therefore, TransE can encode symmetric relation with condition (b), when 71 a<|r|] and 1. Figure shows shows different conditions for encoding symmetric relation.
Conditions (c-d) are directly resulted from (b), as it subsumed by (c) and (d). That completes the proof.
Theorem T4. (Addressing L4): For both TransE and TransComplEx, (i) Limitation L4 holds under (a). (ii) Limitation L4 is not valid under (b-d).
Theorem T5. (Addressing L5): For both TransE and TransComplEx, (i) Limitation L5 holds under (a). (ii) Limitation L.5 holds is not valid under (b-d).
Theorem T6. (Addressing L6): For both TransE and TransComplEx, (i) Limitation L6 holds under (a). (ii) Limitation L6 is not valid under (b-d).
4.3 ENCODING RELATION PATTERNS IN TRANSCOMPLEX
Most KGE models learn from triples. Recent work incorporates relation patterns such as tran- sitive, symmetric the top of triples further improve performance of models. For instance, ComplEx-NNE+APE (Ding et al. 2018) encodes implication relation in the ComplEx model. RUGE (Guo et al. 2018) injects First Order Horn Clause rules in an embedding model. SimplEE

---

Under review as a conference paper at ICLR 2020
(Kazemi & Poole 2018) captures symmetric, antisymmetric and inverse relations by weight tying a in the model. Inspired by (Miervini et al. 2017) and considering the score function of TransCom- plEx, in this part, we derive formulae for equivalence, symmetric, inverse and implication to be used as regularization terms in the optimization problem. Therefore, TransComplEx incorporates different relation patterns to optimize the embeddings.
Symmetrie: Assume that r is symmetric relation. To encode we should have fr (h, t) = ft(t, h), therefore (h, t) fr(t, h)]l 0. According the definition of score function of TransCompllxx, we have the following algebraic formulae: Rsym == [e Re(t)] O.
Using similar argument symmetric, the following formulae are derived for composition,,, inverse and implication.
Equivalence: Let p, q be equivalence relations, therefore we should have fy(h,t) fa t). We
obtain, REg == |pp qll = 0.
Implication: Let p q, be the implication rule. We obtain Rimp == max( [1 1),0) 0.
Inverse: Let ++ r- be the inverse relation. We obtain Riav == ||r - r-
Finally, the following optimization problem should be solved:
min L. + Σ Ri
8
where 0 is emhedding parameters, L one of the losses
OF 5 and R2 is one of the derived
formulae mentioned above.
EXPERIMENTS AND EVALUATIONS
In this section, we evaluate performance of our model, TransComplEx, with difffrrent loss functions on the link prediction task. The aim of the task is to complete the triple rh, or (2,rt) by prediction of the missed entity h or t. Filtered Mean Rank (MR), Mean Reciprocal Rank (MRR) and Hit@ 10 are used for evaluations (Wang et al. 2017 Lin et al. 2015b)
Dataset. We use dataset extracted from Freebase (Bollacker al. 2008) (i.e. FB15K (Bordes) jet al. 2013) and FB15K-237 (Toutanova & Chen 2015)) and two others extracted from Word- Net (Miller 1995) (i.e. WN18 (Bordes al. (2013) and WNISRR (Dettmers al. 20118). FB15K and WN18 are earlier dataset which have been extensively used to compare performance of KGEs. FB15K-237 and WNI8RR are two dataset which are supposed be more challenging after remov- ing inverse patterns from FB15K and WN18. Guo et al. (2018) and Ding et al. (2018) extracted different relation patterns from FBI5K and WN18 respectively. The relation patterns are provided
by their confidence level, e.g. (a,, BornIn, b) 0.9 (a, Nationality, b). We drop the relation pat- terns with confidence level less than 0.8. Generally, we use 454 and relation patterns for FB BB15K and WN18 respectively. We do grounding for symmetric and transitive relation patterns. Thanks to the formulation of score function, grounding is not needed for inverse, implication and equivalence.
Experimental Setup. We implement TransComplEx with the losses 2 4 and 5 and TransE with the loss 4 in PyTorch. Adagrad is used as an optimizer. We generate 100 mini-batches in each iteration. The hyperparameter corresponding to the score function embedding dimension We add slack variables to the losses 2 and 4 to have soft margin as in (Nayyeri et al. 2011)). The loss rewritten as follows Nayyeri et al. (2019):
min Eh.
Σ
(No 2 + A1 max(ff(h, - 77,
(8)
d2
Σ
max(12 f1hh',t)) - 0)).
(l.r,))ES a
We set A; and A2 to one and search for the hyperparameters ni > m) and Ag in the sets .......,) 22 and (0.01, 0.1, 1, 10, 100) respectively. Moreover, generate 21,2,5, 10
(0.1, 0.2.
7

---

Under review as a conference paper at ICLR 2020
negative samples per each positive. The embedding dimension and learning rate tuned from the sets {100, 200), (0.0001, 0.0005, 0.001, 0.005, 0.01} respectively. All hyperparameters are ad- validation set according MRR. RPTransComplEx4 denotes the
justed by early stopping TransComplEx model which is trained by the loss function # 2 4
RP indicates that relation patterns injected during learning by regularizing the derived formulae (see TransComplEx# refers to our model trained with the loss # without regularizing relation patterns formulac. The same notation is used for TransE#. The optimal configurations for RPTranSComplEx are provided in the appendix.
FB15k
WN18
MR
MRR
Hits@ 10
MR
MRR Hits 10
TransE Bordes
2013
125
47.1
251
89.2
TransH (bern) Wang 2014
87
388
82.3
TransR (bern) Lin al. 2015b
77
225
92.0
TransD (bern) et al.
91
212
92.2
TransE-RS (bern) Zhou el at.
2017a
63
72.1
371
93.7
TransH-RS (bern) ZZou al.
2017a
77
75.0
357
94.5
TorusE (Ebisu & Ichise 2019
73.3
83.2
94.7
95.4
T(rth WNP'
Ichise 2019
75.1
83.5
94.7
95.4
R-GCN Schlichtkrull al. 2018
65.1
82.5
81.4
95.5
ConvE Detimers al.. 20018
68.9
85.1
504
95.5
ComplEx Touulon al.
106
67.5
82.6
543
ANALOGY (Lm
121
72.2
84.3
RotatE Sun 2019
48
69.0
433
95.5
SimplE Kazemi & Poole 2018
72.7
83.8
94.2
94.7
SimplE+ Fatemi et al.
2018
72.5
84.1
93.7
93.9
PTransF: [in al.
211ba
58
84.6
KALE Guo et al.
2016
73
52.3
76.2
241
53.2
94.4
RUGE Guo 2018
97
76.8
86.5
CompLEx-NNE+4AER (Ding
2018
116
80.3
87.4
450
94.3
94.8
RPTransComplE:
38
70.5
88.3
451
92.7
94.8
RPTransComplE
38
72.4
88.8
275
92.4
95.4
RPTransCompl
59
61.7
82.2
547
94.0
94.7
Trans ConsCompllEy
38
68,2
87.5
284
92.2
95.5
Trans
46
64.8
87.2
703
68.7
94.5
Table 2: Link prediction results. Rows 1-8: Translation- models with no injected relation
patterns. Rows 9-13: basic models with no injected relation patterns. Rows 14-18: models which encode relation patterns. Results labeled with and ++ are taken from (Zhou et al., (017a), (Ebisu & Ichise 2019) and (Akrami et al., (2018) respectively, while the rest are taken from original papers/code. Dashes: results could not obtained.
Results. Table 2 presents comparison of TrransComplEx and its relation pattern encoded variants (RPTranSComplEx) with three classes of embedding models on the most famous two datasets of FB15K and WN18. The first category (CATI) of models consist of translation-based model (e.g. TransX, TorusE). The second category (CAT2) are embedding models which are not translation- based (e.g. ConvE, ComplEx, ANALOGY). The previous two categories (CAT1/2) can learn relation patterns that are reflected in triples. In other words they learn patterns implicitly from existing triples. The last category (CAT3) consist models which can explicitly encode (inject) rules in their training process (e.g. RUGE, ComplEX-NNE+ANE are actually trained on relation patterns as well as triples.
SimplE, SimplE+), The models in CAT3
We trained relation pattern version of model i.e. RPTransComplEx using the losses of All variants of RPTransComplESs except [RTransComplEx4] are generally performing better than models in CAT3 which also explicitly inject relation patterns. The exception, PRTransComplExy due to the fact that uses margin ranking loss which we already showed its restriction regarding condition (d). In this regard, please consider histogram (d) in Figure
As all limitations were mitigated by condition (c), we expected that RPTransComplE:44 which is associated to (c), should perform better than others. This is empirically approved as shown in Table-
8

---

Under review as a conference paper at ICLR 2020
FB15k-237
WNISRR
MR MRR Hits@
MR
MRR Hits@ I0
TransE (Bordes et al. 2013 (our implementation)
205
27.1
45.2
3806
19.5
45.1
DistMult (Bordes al, 2013
24.1
41.9
43.0
49.0
ComplEx Trouillon al. 2016
24.0
41.9
44.0
51.0
R-GCN Schlchikrull
24.8
41.7
ConvE Dettmers
31.6
49.1
46.0
48.0
TorusE Ebisu & Ichuse 2019
30.5
48.4
45.2
51.2
TorusF (with WXP) (Ebisu Tchise 2019
30.7
48.5
46.0
53.4
RotatE (Sun et al. 2019
211
31.1
49.4
4789
47.3
54,9
RPTransComplEs
210
27.7
46.4
RPTransComplLE
226
31.9
49.5
RPTransCompll
216
25.3
43.8
TransComplE 4
223
31.7
49.3
4081
38.9
49.8
Trans
205
27.2
45.3
3850
20.0
47.5
Table 3: Link prediction results. Rows 1-8: basic models with no injected relation patterns. Re- sults labeled with are taken from (Ebisu Ichise 2019) while the rest are taken from original papers/code. Dashes: results could not be obtained.
2.0
2.0
symmetry (With injected rules)
symmetry (Without injected rules)
15
1.5
10
0.5
0.5
0.0
0.0
500
1000
1500
2000
2500
3000
3500
500
1000
1500
2000
2500
3000 3500
Figure 3: The convergence of symmetric relation loss on FB15K
2
The question is that how our model behaves under the loss 4 while we additionally disregard injection of relation patterns. To this end, we considered TransComplE which uses loss 4 but does not do any injection. Since the performances of RPTransCompllEx and TransComplEy are very close, we conclude that the latter also learns the pattern over the existing triples quite efficiently. This also confirmed by the convergence performance of them shown in Figure over the symmetric relation.
To have a fair comparison! with the categories of models which disregard injection of relation pat- terns (CAT1/2) we used the Tra version of our approach. As shown in Table we can observe that CrranSomplEEx]peee performs better than other models in CAT1/2 considering MR and Hits @ 10 and performing very closely to the best models on MRR.
As discussed earlier, FB15K-237 and WN18RE are two more challenging dataset provided recently. Table presents the comparisons of our models with those other models that their performance results were available on these two datasets. Similar to our previous discussion on Table-2 we observe that PRTransComplE:41 and TransComplEx4]] latter has learned the relation pattern over triples very well without any injection. On WN18RR, TorusE was performing better than TransComplEx4loo to big embedding dimension of 10,000. On FB 2515k- RPTransComplE performedd better that others on MRR and Hits 10.
performing closely which means that the
In order to investigate the effect of grounding, train RPTransCommlEEx2 in settings: 1) RP- TransComplFxAJ] w grounding) is trained when the grounded patterns are injected, 2) RPTransCom- plE w/o grounding) is trained when the relation patterns which are grounded used. According to the Table 4 the grounding does not affect the performance significantly. We conclude that the model properly learns the relation patterns even without injection.
Boosting techniques: There several ways to improve the performance of embedding models: 1) designing a more sophisticated scoring function, 2) proper selection loss function, 3) using
Accordingly, we ran the RotatE code in setting (embedding dimension 200 and 10 negative samples). The original paper used very big numbers of 1000 and 1000 respectively.

---

pre- Under review as a conference paper at ICLR 2020
FBI5K
MR
MRR Hits@
RPTransComplEx w grounding)
38
72,4
88.8
RPTransComplE w/o grounding)
40
72.8
88.7
Table 4: Link prediction results. RPTransComplE grounding) trained on triples and the relation patterns which are grounded. RPTransComplEXX! w/o grounding) is trained on triples and the relation patterns which are not grounded.
negative samples 4) using negative sampling techniques, 5) enriching dataset (e.g. adding reverse triples). Among the mentioned techniques, we focus on the first and second ones and avoid using other techniques. We keep the setting used in (Trouillon et al. (0166) have a fair comparison. Using other techniques can further improve the performance of every models including ours. For example, TransComplEx with embedding dimension 200 and 50 negative samples gets 52.2 for @its@ 10. Further analyses of our models in a big setting (bigger embedding dimension and more negative samples) are provided in appendix. Still, loss is performing better and we conclude that our theoretical framework is approved in empirical experiments.
CONCLUSION
In this paper, we reinvestigated the main limitations of Translation-basee embedding models from two aspects: score and loss. We showed that existing theories corresponding to the limitations the models are inaccurate because the effect of loss functions has been ignored. Accordingly, sented new theories about the limitations by consideration of the effect score and loss functions. We proposed TransComplEx, new variant of TransE which is proven to be less limited comparing to the TransE. The model is trained by using various loss functions on standard dataset including FB15K, FB FB15K-237, WN18 and WNISRR. According the experiments, TransComplEx with proper loss function significantly outperformed translation-based embedding models. Moreover, TransComplEx got competitive performance comparing to the state-of-the-ar embedding models while is more efficient in time and memory. The experimental results conformed the presented theories corresponding to the limitations.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Farahnaz Akrami, Linghing Guo, Wei Hu, and Chengkai Li. Re-evaluating embedding-hased knowl- edge graph completion methods. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 1779-1782. ACM, 2018.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collab- oratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247-1250. AcM, 2008.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances neural information processing systems, pp. 2787-2795, 2013.
Tim Dettmers, Pasquale Minervini. Pontus Stenetorp. and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Thirty-Second Second AAAI Conference on Artificial Intelligence, 2018.
Boyang Ding, Quan Wang, Bin Wang, and Li Guo. Improving knowledge graph embedding using simple constraints. arXiv preprint arXiv: 1805.02405, 2018.
Takuma Ebisu and Ryutaro Ichise. Toruse: Knowledge graph embedding on lie group. In Thirty- Second AAAI Conference on Artificial Intelligence. 2018.
Takuma Ebisu and Ryutaro Ichise. Generalized translation- based embedding of knowledge graph. IEEE Transactions on Knowledge and Data Engineering, 2019.
Bahare Fatemi, Siamak Ravanbakhsh, and David Poole. Improved knowledge graph embedding using background taxonomic information. arXiv preprint 1rXiv:18/2.03225. 2018.
Jun Feng. Minlie Huang, Mingdong Wang, Mantong Zhou, Yu Hao, and Xiaoyan Zhu. Knowledge graph embedding by flexible translation. In Fifteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2016.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang. and Li Guo. Jointly embedding knowledge graphs and logical rules. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 192-202, 2016.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. Knowledge graph embedding with iterative guidance from soft rules. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Frank I. Hitchcock. The expression of tensor or polyadic as sum of products. Journal of Mathematics and Physics, 6(1-4):164-189, 1927.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), volume 1, pp. 687-696, 2015.
dynamic mapping matrix.
Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems, pp. 4284-4295, 2018.
Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. Modeling relation paths for representation learning of knowledge bases. arXiv preprint 1XX6:05566.0077 2015a.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference artificial intelligence, 2015b.
Hanxiao Liu, Yuexin Wu, and Yiming Yang. Analogical inference for multi-relational embeddings. In Proceedings of the 34th International Conference Machine Learning- 2178. JMLR. org, 2017.
Learning-Volume Volume 70, pp. 2168-
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
11

---

Under review as a conference paper at ICLR 2020
Pasquale Minervini, Luca Costabello, Emir Munoz, Novacek, and Pierre- Yves Vandenbussche. Reg- ularizing knowledge graph embeddings via equivalence and inversion axioms. In Joins Euro- pean Conference Machine Learning and Knowledge Discovery in Databases, pp. 668-683. Springer, 2017.
Mojtaba Nayyeri, Sahar Vahdati, Jens Lehmann, and Hamed Shariat Yazdi. Soft marginal transe for scholarly knowledge graph completion. arXiv preprint arXiv: 12211, 2019.
Dat Quoe Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. Stranse: novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08/ 0814600000.,,,.. 2016.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pp. 593 607. Springer, 2018.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nic, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10111 2019.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pp. 57-66, 2015.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071-2080, 2016.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29 2724-2743,201
Yanjie Wang, Rainer Gemulla, and Hui Li. On multi-relational link prediction with bilinear models. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by trans- lating hyperplanes. In Twenty-Eightt AAAI conference artificial intelligence, 2014.
Xiaofei Zhou, Qiannan Zhu, Ping Liu, and Li Guo. Learning knowledge embeddings by combining limit-based scoring loss. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1009-1018. ACM, 2017a.
Xiaofei Zhou, Qiannan Zhu, Ping Liu, and Guo. Learning knowledge embeddings by combining limit-based scoring loss. In Proceedings of the 2017 ACM Conference Information and Knowledge Management, pp. 1009-1018. ACM, 2017b.
12

---

Under review as a conference paper at ICLR 2020
A FURTHER EXPERIMENTS WITH A BIGGER SETTING
In this section we compare TransE, TransComplEx and RotatE trained by using the losses 2 (con- dition (a),(b) A (condition (c)) and the RotatE loss (condition (c)). In contrast our previous experiments, we use a bigger setting: For FB 15K-23-2 we set the embedding dimension to 300 and the number of the negative samples to 256, and for WN18RR, we set the embedding dimension and the number of negative samples to 300 and 250 respectively. We additionally use adversarial negative sampling technique used in (Sun et al. 2019) for all models.
FB15K-237
WN18RR
MR
MRR Hits 10
MR
MRR
Hits 10
Trans (71 0)
222
27.4
45.7
3014
19.3
47.4
Trans (71 > 0)
198
31.3
50.5
3942
21.4
50.3
T4
181
32.3
52.1
3451
23.5
53.9
Trans
FRotatt Rotat
179
32.5
51.9
3594
23.3
53.6
TransComplEx2
2 (11
0)
213
28.5
47.3
3014
31.2
49.5
TransComplE
(Y) >
0)
194
31.9
50.8
3942
41.3
50.8
TransComplEy
177
32.8
52.1
3435
44.3
55.0
TransComplEx RotatE
176
32.7
51.9
3537
44.2
54.7
R Rotat RotatE
194
33.0
52.0
3806
47.8
56.9
196
33.0
51.8
3943
47.3
56.5
Table 5: Link prediction results. Rows 1-4: TransE trained using condition (a), (b). (c)(with the loss and (c)(with the RotatE loss) with no injected relation patterns. Rows 5-8 TransComplEx trained using condition (a). (b), (c)(with the loss 4 and (c)(with the RotatE loss) with no injected relation patterns. Rows 9-10: RotatE trained using condition (c)(with the loss and (c)(with the RotatE loss) with injected relation patterns.
Analysis of the results: Table presents comparison of TransE, TransComplEx and RotatE trained by different losses. 2 71 0) trained by using the loss when ?1 0. Trans i > 0)
refers to the TransE model which is trained by using the loss loss when n is a non-zero positive value. The TransE model which trained by the (add the RotatE loss (i.e., C. CRotet&) are denoted by TransF and TransEEC.Rtate RotatE respectively. The similar notations are considered for TransComplEx and RotatE when they are trained by using different loss functions. The loss with 711 = approx- approximates the condition (b). The condition (c) can be approximated by using the loss and the RotatE loss (i.c., EflotatE). However, the loss 4 provides a better separation for positive and the negative samples than the RotatE loss. According the table 5 the loss Jobtains a better performance than the other losses in each class of the studied models. is consistent with our theories indicating that the condition (c) less restrictive. Although we only investigated the main limitations of the translation-based class of embedding models, the theories can be generalized different models including the RotatE model. From the table, we can see that the loss [4] improves the performance of RotatE. Regarding the table 5) the loss 1oss (71 = 0) gets the worst results. It confirms our theories that with the condition (a). most of the limitations are held. However, with the condition (c), the limitations no longer exist. There have not been any losses that approximate the condition (a). However, most of the theories corresponding to the main limitations of the translation-based class of embedding models have been proven using the condition (a) while the used loss didn't approximate the condition. Therefore, the theories and experimental justifications have not been accurate.
imates the condition (a). The loss 2 with "1
B RELATION PATTERN CONVERGENCE ANALYSIS
Figure visualizes the convergence curve of the inverse loss with (RPTransComplEX4 and without (TranCComplEs4) injection when the models are trained on WN18, FFiuree J/shows the convergence of the TransComplfs model
model trained FB15K with and without the relation pattern injection. The
13

---

Under review as a conference paper at ICLR 2020
2,00
2,00
Inverse
175
(With
175
125
125
Loss
100
1.00
0.0
0.0
0.25
025
500
1000
1500
2000
500
1000
1500
2000
Training Steps
Figure 4: The convergence of inverse relation loss WN18
2.0
Inverse
Inverse
(With injected rules)
(Without injected rules)
15
10
10
0.5
0.5
0.0
0.0
500
1000
1500
2000
2500
3000
3500
0
500
1000
1500
2000
2500
3000
3500
2.0
symmetry
symmetry
(With injected rules)
(Without injected rules)
1.5
10
10
0.5
0.5
Loss
500
1000
1500
2000
2500
3000
3500
1000
1500
2000
2500
3000
3500
2.0
2.0
implication
injected rules)
(Without injected rules)
1.5
1.
1.0
10
0.5
0.0
500
1000
1500
2000
2500
3000
3500
500
1000
1500
2000
2500
3000
3500
2.0
2.0
equivalence
equivalence
(With injected rules)
(Without injected rules)
1.5
0.5
0.5
0.0
0.0
500
1000
1500
2000 2500
3000
3500
0
500
1000
1500
2000
2500
3000
3500
Training Steps
Figure 5: The convergence of inverse, symmetric, implication and equivalence relation losses FB15K
14

---

Under review as a conference paper at ICLR 2020
figures show that the models trained by using the loss 4 can properly encode the relation patterss even without any injection mechanism. In other words, the TransComplEX]] model can properly encode several relation patterns by only training the triples (without using any additional relation pattern set to be injected). This shows the advantages of the models and the used losses.
C PROOF OF THE THEOREM
The proof of theorems are provided as follows:
Proof of the Theorem TI 1) Let be reflexive relation and condition a) holds. For TransE, we have
++r-- h 0.
Therefore, the relation vector collapses null vector (r 0). As consequence of emhedding vectors of head and tail entities will be same which is undesired. Therefore, TransE cannot infer reflexive relation with + 0.
For TransComplEx, we have
h+r O.
(10)
We have
Re(r) = 0, Im(r) = -h)
(11)
Therefore, all entities will have same embedding vectors which is undesired.
2) Using condition (b), we have
h 71-
gives ||||| 71. Therefore, in order infer reflexive relation, the length of the relation vector should be 7/1 Consequently, TransE and TransComplEx can infer reflexive relation. The same procedure can be used for the conditions (c) and (d).
Proof of the theorem T2: i) Let the relation be neither reflexive nor irreflexive and two triples (e,,T,e),, (,) be positive and negative respectively. Therefore the following inequalities hold:
|le + eil < \
(12)
[lee +r E21 2 A2
Equation 12 is rewritten follows:
Re((r) i(Im(r) + 2Tm(e:))] s. [e(()) + i(Tm(r) + 2Tm(ez)) 22:
(13)
For TransE in real space, |[e(r)] < 71 and |e(()) 2 %2 cannot be held simultaneously when 72 > Y1. Therefore, TransE in real space cannot encode relation which is neither reflexive nor irreflexive. In contrast, TransE in complex space can encode the relation by proper assignment of imaginary parts of entities. Therefore, theoretically TransComplEx can infer relation which is neither reflexive nor irreflexive.
Proof of the theorem T3: i), ii) Let be symmetric relation and holds. We have
r i h.
(14)
15

---

Under review as a conference paper at ICLR 2020
Trivially, we have
Re(h) + Re(r) Re(t).
Re(t) + Re(r) Re)
(15)
Im(h) Im(r) =t.
Im(t) - Im(r) -Im
For TransE in real space, there is
(e(h) Re(r) Re(t), Re(t) Re(r) (e).,
Therefore, Re(r) = O. It means that TransE cannot infer symmetric relations with condition a). For TransComplEx, additionally we have
Im(h) + Im(r) (t), Im(t) - Im(r) (h).
It concludes 7m(h) + Tm(r) + (t) = 0. Therefore, TransE in complex space with condition a) can infer symmetric relation. Because a) is an special case of b) and c), TransComplEx can infer symmetric relations in all conditions.
3) For TransE with condition b), there
h + t| = 71 + r h| 71-
(16)
(17)
The necessity condition for encoding symmetric relation is mh+r-- t|| ||t++r b/|. This implies
[|A||cos(Op.r) = (ellcco((t,.)) Let u, by 17 we have + r|| 71, ||u
r| = -h-
Let 71 a |r|] We have
||||2 + (1 a²) | -2(u, r)
(18)
|||| + (1 a²) ||||| 2(u,r)
Regarding 18 have
||u|| + (1 - a² [1-a2)]=|]==000 (
|u|| - 2 (a2 1||
To avoid contradiction, a 2 1. a 2 we have (0s() = T T/2. Therefore, TransE can encode symmetric pattern with condition b), i a||r|| and 2 1. From the proof of condition b), conclude that TransE can encode symmetric patterns under conditions c) and d).
Proof of the theorem 74: i) The proof of the lemma with condition for TransE is mentioned in the paper Kazemi & Poole (2018). For TransComplEx, the proof trivial. ii) Now, we prove that the limitation L4 is not valid when b) holds.
Let condition b) holds and relation be reflexive, we have ||ei + - eill mı l +r -2ll = 71-
Let ||e + e2|l = 71. To violate the limitation L4, the triple (e2, r, r1) should be negative i.e.,
|le + - ei| > 7,
|lez + e1 7/-
ez, e, > -2 e1, > + ||ez|1 2 + ||e 2 ||||2 + < e2 -2 <
Considering ||e + - 71. we have
e2, <e,r>> >0,
><e2 e,r 0,
+ cos (0 > 0,
Therefore, the limitation L4 is not valid i.e., if relation is reflexive, it may not be symmetric. TransE is special case TransComplEx and also condition b) is special case of condition c). There- fore using conditions b), c) and d), the limitation L4 is not valid for TransE and TransComplEx.
16

---

Under review as a conference paper at ICLR 2020
Proof of the theorem T5
i) Under condition a), equation + r - holds. Therefore, according to the paper Kazemi & Poole (2018), the model has the limitation L5.
ii) a relation reflexive, with condition b), we have |lei + r - -eill = 71, l|e + r - r-ell = 71.
Therefore, ||||] >11 Let
{ le + e2ll 71
(19)
lez + e3]l = 71-
we need to show the following inequality wouldn't give contradiction: |ee + 03/l > 71.
From 19 we have e2, (ei e2 + e3) >< 0, which is not contradiction.
Therefore, with conditions b) and c), the limitation 1,5 is not valid for both TransE and TransCom- plEx.
Proof of the theorem T6: i) With condition (a), the limitation L6 is proved in Kazemi & Poole 2018
ii) Considering the assumption of L6 and the condition (b), have
ee Si
71
le r 82 71-
(20)
(e + r S1 71-
We show the condition that |lez + $2 > Yi holds.
Substituting 20 in ee + s2ll ?1. we have
cos(0 (s+ se) -es)) < O. Therefore, there are assignments to embeddings of entities that the limi-
tation L6 is not valid with condition (b), (c) and (d).
Figure 6 shows that the limitation L6 is invalid by proper selection of loss function.
52
$1
el
Figure Investigation of L6 with condition (c): The limitation is not valid, be- cause the triple (e2,7, 82) get score to be considered negative while triples ((61,1,8)) (ei, 82), (62, F, 81)) are positive.
C.1 FURTHER LIMITATIONS AND FUTURE WORK
In the paper, have investigated the six limitations of TransE which resolved by revision of loss function. However, revision of loss functions can resolve further limitations including 1-N, N-1 and M-N relations. More concretely, setting upper-bound for the scores of positive samples can mitigate the M-N problem. We will leave as future work.
Our theories be extended to every distance- based embedding models including RotatE etc.
Moreover, the negative likelihood loss has been shown to be effective for training different embed- ding models including RotatE and TransE. This can also be explained by reformulation of negative
17

---

Under review as a conference paper at ICLR 2020
likelihood loss standard optimization problem, showing the the loss put boundary for the score functions.
We will consider the mentioned points as future work.
D OPTIMAL HYPER-PARAMETER
The following tables show the optimal configurations for models included in the Table- and
RPTranSCompIE
FB 15k
FB 15k-337
WN18
Embedding Dimension (d)
200
200
200
A
100
100
100
71
0,4
1.5
1.0
72
0.5
2.0
2.0
Negative Sample (a))
10
10
10
Table 6: Optimal Setting. The best hyyer-paramene setting for RPTransComplEx2 on several
dataset.
RPTransComplE:
FB 1.5k
FB 5k-277
WN18
Embedding Dimension (d)
200
200
200
Ao
10
100
100
71
0.4
1.5
0.6
72
0.5
2.0
1.7
Negative Sample (a)
10
10
2.0
Table 7: Optimal Setting. The best paprmmter setting for RPTranSComplEx4 on several
dataset.
RPTransComplEs5
FB15% 15k
FB 155--337
WN18
Embedding Dimension (d)
200
200
200
5.0
10
Negative Sample (a:
10
10
10
Table 8: Optimal Setting. The best hyper-parameter setting for RPTransComplEx5 on several dataset.
18

---

Under review as a conference paper at ICLR 2020
TransComplEX
FB15k
FB1.k-237
WN18
WN18rr
Embedding Dimension (d)
200
200
200
200
Ao
10
100
100
0,4
1.5
0.6
1.6
722
0.5
2.0
1.7
2.7
Negative Sample (a)
10
10
2.0
2.0
Table 9: Optimal Setting, The best hyper- papee--arameter setting for TransComplEs4 on several dataset.
TransE
FB15k
FB15k-237
WN18
WN18rr
Embedding Dimension (d)
200
200
200
200
Ao
10
100
1
1
Y11
0.4
0.4
1.0
0.6
72
0.5
0.5
2.0
1.7
Negative Sample (a)
10
10
10
2
Table 10: Optimal Setting. The best hyper-parameter setting for Trans on several dataset.
19


Reviews:
[{"review_id": "rklmtk369H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper revisits limitations of relation-embedding models by taking losses into account in the derivation of these limitations. They propose and evaluate a new relation encoding (TransComplEx) and show that this encoding can address the limitations previously underlined in the literature when using the right loss.\n\nThere seems to be merit in distinguishing the loss when studying relation encoding but I think the paper's analysis lacks proper rigor as-is. A loss minimization won't make equalities in (3) and (5) hold exactly, which the analysis do not account for. A rewriting of the essential elements of the different proofs could make the arguments clearer.\n\nPaper writing: \n* The manuscript should be improved with a thorough revision of the style and grammar. Example of mistakes include: extraneous or missing articles, incorrect verbs or tenses. \n* The 10-pages length is not beneficial, the recommended 8-pages could hold the same overall content.\n* The option list on page 8 is very difficult to read and should be put in a table, e.g. in appendix.\n* Parentheses are missing around many citations and equation references\n\nTheory:\nEquation (2) and (4) do not seem to bring much compared to the conditions in Table 1. Eq. (3) and (5) show \"a\" loss function rather than \"the\" loss function since multiple choices are possible. \\gamma_1 should be set to 0 when it is 0 rather than staying in the equations.\n* Minimizing the objective (3) and (5) will still not make the conditions in Table 1 hold exactly, because of slack variables.\n\nExperiments:\n- Can the authors provide examples of relations learned with RPTransComplEx# that go address the limitations L1...L6, validating experimentally the theoretical claims and showing that the gain with RPTransComplEx5 correspond to having learned these relations?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1eX02mVYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThis paper list several limitations of translational-based Knowledge Graph embedding methods, TransE which have been identified by prior works and have theoretically/empirically shown that all limitations can be addressed by altering the loss function and shifting to Complex domain. The authors propose four variants of loss function which address the limitations and propose a method, RPTransComplEx which utilizes their observations for outperforming several existing Knowledge Graph embedding methods. Overall, the proposed method is well motivated and experimental results have been found to be consistent with the theoretical analysis.\n\nSuggestions/Questions:\n\n1. It would be great if hyperparameters listed in the “Experimental Setup” section could be presented in a table for better readability. \n\n2. In Section 2, the authors have mentioned that RotatE obtains SOTA results using a very large embedding dimension (1000). However, it gives very similar performance even with smaller dimensional embedding (such as 200) with 1000 negative samples. In Section 5, RotatE results with 200 dimension and 10 negative samples are reported for a fair comparison. Wouldn’t it be better to instead increase the number of negative samples in RPTransComplEx instead of decreasing negative samples in RotatE?\n\n3. In Table 3, it is not clear why authors have not reported their performance on the WN18RR dataset for their methods. Also, the reported performance of TransE in [1] is much better than what is reported in the paper. \n\n[1] Sun, Zhiqing, Zhi-Hong Deng, Jian-Yun Nie and Jian Tang. “RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.” ArXiv abs/1902.10197 (2019): n. pag.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BylVtzu35S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyses the effect of different loss functions for TransE and argues that certain limitations of TransE can be mitigated by chosing more appropriate loss functions. Furthermore, the paper proposes TransComplEx -- an adaption of ideas from ComplEx/HolE  to TransE -- to mitigate issues that can not be overcome by a simply chosing a different loss.\n\nAnalyzing the behavior and short-comings of commonly-used models can be an important contribution to advance the state-of-the-art. This paper focuses on the performance of TransE, which is a popular representation learning approach for knowledge graph completion and as such fits well into ICLR.\n\nUnfortunately, the current version of the paper seems to have issues regarding methodology and novelty.\n\nRegarding the experimental evaluation: The paper compares the results of TransComplEx and the different loss functions to results that have previously been published in this field (directly, without retraining). However, it seems from Section 5 (Dataset), that this paper is using a modified dataset, as the TransE models are only trained on high-confidence triples. All prior work that I checked doesn't seem to do this, and hence the numbers are not comparable.\n\nEven more serious: Following again Section 5 (Dataset), it seems that the paper imputes all missing triples in the training set for symmetric and transitive relations (\"grounding\"). Hence,  the models get to see _all_ true triples for these relation types and as such the models in this paper are trained on the test set.\n\nRegarding novelty: The short-comings of TransE and improvements to the loss have been discussed quite extensively in prior work. Using complex representations in TransComplEx seems also a straightforward application of the insights of ComplEx/Hole. As such, the main novelty would lie in the experimental results which, unfortunately, seem problematic.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BygbP8ShYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors investigate the main limitations of TransE in the light of loss function. The authors claim that their contributions consist of two parts: 1) proving that the proper selection of loss functions is vital in KGE; 2) proposing a model called TransComplEx. The results show that the proper selection of the loss function can mitigate the limitations of TransX (X=H, D, R, etc) models.\n\nMy major concerns are as follows.\n1.\tThe motivation of TransComplEx and why it works are unclear in the paper.\n2.\tThe experiments might be unconvincing. In the experiments, the authors claim that they implement RotatE [1] in their setting to make a fair comparison. However, with their setting, the performance of RotatE is much worse than that in the original paper [1]. Therefore, the experiments might be unfair to RotatE.\n3.\tThere are some typos in this paper. For example, in Line 11 of Section 4.3, the comma should be a period; in Section 5, the \"Dissuasion of Results\" should be \"Discussion of Results\".\n\n[1] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses the effect of different loss functions for TransE and argues that certain limitations of TransE can be mitigated by choosing more appropriate loss functions.  The submission then proposes TransComplEx to further improve results.  This paper received four reviews, with three recommending rejection, and one recommending weak acceptance.  A main concern was in the clarity of motivating the different models.  Another was in the relatively low performance of RotatE compared with [1], which was raised by multiple reviewers.  The authors provided extensive responses to the concerns raised by the reviewers.  However, at least the implementation of RotatE remains of concern, with the response of the authors indicating "Please note that we couldn’t use exactly the same setting of RotatE due to limitations in our infrastructure."  On the balance, a majority of reviewers felt that the paper was not suitable for publication in its current form.
Decision: Reject