A Group-Theoretic Framework for Knowledge Graph Embedding
Authors: Tong Yang
Long Sha
Pengyu Hong
Venue:  (2020)
Keywords: group theory
knowledge graph embedding
representation learning

Abstract: We have rigorously proved the existence of a group algebraic structure hidden in relational knowledge embedding problems, which suggests that a group-based embedding framework is essential for model design. Our theoretical analysis explores merely the intrinsic property of the embedding problem itself without introducing extra designs. Using the proposed framework, one could construct embedding models that naturally accommodate all possible local graph patterns, which are necessary for reproducing a complete graph from atomic knowledge triplets. We reconstruct many state-of-the-art models from the framework and re-interpret them as embeddings with different groups. Moreover, we also propose new instantiation models using simple continuous non-abelian groups.

Under review as a conference paper at ICLR 2020
A GROUP-THEORETIC FRAMEWORK FOR KNOWL- EDGE GRAPH EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
We have rigorously proved the existence of a group algebraic structure hidden in relational knowledge embedding problems, which suggests that a group- based embedding framework is essential for designing embedding models. Our theoreti- cal analysis explores merely the intrinsic property of the embedding problem itself hence is model independent. Using the proposed framework, one could construct embedding models that naturally accommodate all possible local graph patterns, which are necessary for reproducing a complete graph from knowledge triplets. We reconstruct many state-of-the-ar models from the framework and re-interpret them as embeddings with different groups. Moreover, we also propose new in- stantiation models using simple continuous non-abelian groups.
INTRODUCTION
Knowledge graphs (KGs) are prominent structured knowledge bases for many downstream semantic tasks (Hao et al. 2017). A KG contains an entity set & = {ei}. which correspond to vertices in the graph, and relation collection of factual triplets, each of which has the form (e,, rg ey) where r2 is the relation between the head entity e; and the tail entity e,. Since large scale KGs are usually incomplete due missing links (relations) amongst entities, an increasing amount of recent works (Bordes et al. 2013 Yang jet al.) 2014; Trouillon et al. 2016 [Lin et al. 2015) have devoted to the graph completion (i.e., link prediction) problem by exploring a low-dimensional representation of entities and relations.
{rk), which forms edges. The entity and relation sets form
More formally, each relation r acts as mapping Orl:] from its head entity e₁ to its tail entity 02:
r r:er + Or(ei] =: e2.
(I)
The original KG dataset represents these mappings in a tabular form, and the task of KG embed- ding is to find better representation for these abstract mappings. For example, in the TransE model (Bordes et al. 2013), relations and entities are embedded in the same vector space, and the operation O,1 is simply a vector summation: Or(c] e + In general, the operation could be either linear or nonlinear, either pre-defined or learned.
Since the entity and relation sets in KGs play conceptually different roles, it is reasonable to rep- resent them differently, which demands a more proper operation Or[:] for bridging the entity and relation sets. On the other hand, the graph completion task relies on the fact that relations are not independent. For example, the hypernym and hyponym are inverse to each other; while kinship relations usually support mutual inferences. Previous studies (Sun et al. 2019 Xu & Li 2019) have concerned some specific cases of inter-relation dependencies, including (anti-)sy -smmmmtyymmtry and compositional relations. However, one may ask a more general question: could we establish certain fundamental principles to guide the operation design, such that any inter-relation dependency could be naturally captured? In this work, we find the answer is positive, and we term these principles as hyper-relations. Group theory emerges natural description tool that satisfies the need for finding better operations and hyper-relations. In the following sections, we demonstrate the emergence of group definition, namely closure, identity, inverse, and associativity. from study of knowledge graph itself, and explain in detail an implementation of group theory to tackle general relational embedding problems.
One main contribution of this work is it provides a framework for addressing the KG embedding problem from a novel and more rigorous perspective: the group-theoretic perspective. This frame-

---

Under review as a conference paper at ICLR 2020
work arises from investigation of the graph reconstruction problem. We prove that the intrinsic structure of this task automatically produces the complete definition of groups. To our best knowl- edge, this is the first proof that rigorously legitimates the application of group theory in KG embed- ding. With this framework, we reconstructed several existing models using group theory language (see Sec (.3). including: TransE (Bordes et al., 2013), TransR (Lin et al.) 2013), TorusE (Ebisu &
Ichise 2018 RotatE (Sun et al. 2019). ComplEx (Trouillon et al. (016). DisMult (Yang et al., 2014).
RELATED WORKS
From the group theory perspective, our work may be related to the TorusE (Ebisu & lchise 2018 the RotatE (Sun et al., 2019) and DihEdral (Xu & Li, 2019) models. The TorusE model tries to frame the KG embedding in Lie- Lie-group and deals with the compactness problem. Its authors proved that the additive nature of the operation in the TransE model contradicts the entity regularization. However, if non-compact embedding space is used, entities must be regularized divergence of negative scores. Therefore the TorusE model used n-torus, a compact manifold, as the embedding space. In other words, a group is regarded a manifold, rather than as a set of operations where algebraic structures are more emphasized. Besides, the compactness issuc was solved in many later works by bounding the training objective using Softplus function (Sun et al., 2019). In the DihEdral model, the D4 group the author used plays the same role as in our work: depicting the transformation of entities by assigning each relation a group element. The motivation of DihEdral is to resolve the non-abelian composition (i.e., the compositional relation formed by ri and r2 would change if the two are switched). Nevertheless, DihEdral applics a discrete group Dg for relation embedding while using a continuous entity embedding space, which may suffer two problems as discussed in the later Section 33 The RotatE model was designed to accommodate symmetric, inversion, and (abelian) composition of triplets at the same time.
prevent the
Different from those previous works, this work does not target at one or few specific cases but aims at answering the more general question: finding the generative principle of all possible cases, and thus to provide guidance for model designs that can accommodate all cases. More importantly,
compared with most preceding works connecting groups (Xu & Li 2019 Cai 2019), the analysis in our current work does not focus on impose an implementation of groups. Rather, we start merely by studying the graph reconstruction problem and prove that the intrinsic structure of this task itself automatically produces the complete definition of groups To our the best of our knowledge, this is the first proof that rigorously legitimates the application of group theory in KG embeddings.
GROUP THEORY IN RELATIONAL EMBEDDINGS
In this section, we formulate the group-theoretic analysis for relational embedding problems. For simplicity, our discussion would start from 1-to-1 mappings and discuss the generalization to more complicated cases in the Appendix D
Firstly, as most embedding models represent objects (including entities and relations) as vectors, the task of operation design thus can be stated as finding proper transformations of vectors. Secondly, as we mentioned in the introduction, our ultimate goal of reproducing the whole knowledge graph using atomic triplets further requires certain types of local patterns to be accommodated. We now discuss these structures, which in the end naturally leads to the definition of groups in mathematics.
3.1 HYPER-RELE RELATION PATTERNS: RELATION- OF-RAATIOONE
One difficulty of generating the whole knowledge graph from atomic triplets lies in the fact that different relations are not independent of each other. The task of relation inference relies exactly on their mutual dependency. In other words, there exist certain relation of relations in the graph structure, which term hyper-relation patterns. A proper relation embedding method and the associated operations should be able to capture these hyper-relations.
Now instead of studying exampling cases one by one, we ask the most general question: what are the most fundamental heper-relations? The answer is quite simple and only contains two types, namely, inversion and composition:
2

---

Under review as a conference paper at ICLR 2020
Inversion: given relation there may exist inversion F, such that:
ej H e2
e2 + e1, 4el, e2 E E.
(2)
The inversion captures any relation path with length equal to I (in the unit of relations).
Composition: given two relations rı and r2, there may exist third relation r3, such that:
{
:1:01 02
+ e3 Vei e2, e3 E E.
(3)
P2 12:02 C3
Any relation paths longer than can be captured by a sequence of compositions.
One may notice the phrase may exist in the above definition, this simply emphasizes that the exis- tence of these derived conceptual relations r and r3 depends on the specific KG dataset; while, on the other hand, to accommodate general KG datasets, the embedding space should always contains the mathematical representations of these conceptual relations.
An important feature of KG is that with the above two hyper-relations, one could generate any local graph pattern and eventually the whole graph, as relational paths with arbitrary length have been captured. Note the term of inversion and composition might have different meanings from ones in other works: most existing works study triplets to analyze hyper relations, while the definition we provide above is based purely relations. This is more general in the sense that any conclusion derived would not depend on entities at all, and some different hyper relations could, therefore, be summarized as single one. For example, there are enormous discussions on symmetric triplets and anti-symmetric triplets (Sun et al. 2019), which are defined as:
symmetric: (c1,1,02) anti-symmetric: (c,,
3 r,),), -e,,rr,,)).
(4)
In fact, if for any choice of e1,2, could produce symmetric pair of true triplets using r, this would imply a property of itself, and in which case, one could then simply derive:
F=r.
(5)
This is a special case of the inversion hyper-relation; and similarly, the anti-symmetric case simply implies F * r, which is quite common, and does not require extra design. The deep reason for discussing hyper-relations which relies merely on relations rather than triplets is that the logic of relation inference problem itself is not entity-dependent
3.2 EMERGENT GROUP THEORY
To accommodate both general inversions and general compositions, we now derive explicit require- ments the relation embedding model. We start by defining the product of two relations rı and re: r1 r2, as subsequently "finding the tail" twice according to the two relations, i.e.
O Oryna(
Or, [Or,[]]-
(6)
With the above definition, equation 3 can be rewritten as: r3 ri r2. One would realize that the following properties should be supported by proper embedding model:
Inverse element: to allow the possible existence of inversion, the elements F should also be element living in the same relation-embeddin space"]
2. Closure: to allow the possible existence composition, in general, the elements rı r2 should also be an element living the same embbdddig space]
3. Identity element: the possibly existing inverse and composition together define another special and unique relation:
i=r-F, Vr E R.
This element should map any entity to itself, and thus we call it identity element.
1 'Given graph, all inversions correspond meaningful relations, but an embedding model should be able to capture this possibility in general.
"Given a graph, not all compositions correspond to meaningful relations, but embedding model should he able capture this possibility general.
3

---

Under review as a conference paper at ICLR 2020
4. Associativity: In relational path with the length longer than three (containing three or more relations (ri, F2, ...))) as long as the sequential order does not change, the follow- ing two compositions should produce the same result:
(I1 r2) rg = r (r2 rs).
(8)
See Appendix A for the proof.
5. ComnutaivviyNoorcommmmta
In general, commuting two relations in a composi- tion, i.e. r1 r2 < r2 rı, may compose either the same or different results. In real graphs any cases may exist, and a proper embedding method should be able to accommodate both
The first four properties are exactly the definition of group. In other words, the group theory auto- matically emerges from the relational embedding problem itself, rather than being applied manually, This is quite convincing evidence that group theory is indeed the most natural language for relational embeddings if one aims at ultimately reproducing all possible local patterns in graphs. Besides, the are actually termed as abelian/nonabelian in the group theory language. Since abelian is more special, to accommodate both possibilities, one should, in general, consider nonabelian group for the relation embedding, and guarantee the same time it contains at least one nontrivial abelian subgroup.
fifth property
commuuaivityhhooooommmmcc
More explicitly, given graph, to implement group structure in embedding, one should embed all relations as group elements, which are parametrized by certain group parameters. For instance: the translation group T' can be parametrized by a real number 8, And correspondingly, due its vector nature, the embedding of entities could be regarded as a representation (rep) space of the same group. For the translation group, I (the real field) a rep space of
This suggests the group representation theory is useful in knowledge graph embedding problems when talking about entity embeddings. and we leave this as a separate topic for subsequent works later. In the later section, we provide a general recipe for the graph embedding implementation.
3.3 EMBEDDING MODELS USING DIFFERENT GROUPS
In this section, we discuss embedding methods using different groups, from simple ones as T (the translation group) and (1). to complicated ones including (U),), GL V) (where V could be any type of fields), or even VV) is important to note that, in practice, continuous groups are more reasonable than discrete ones, due to the two following reasons:
The entity embedding space is usually continuous, which matches reps of the continuous group better. If used to accommodate a discrete group, a continuous space always con- tains infinite copies of irreducible reps of that group. which makes the analysis much more difficult.
When training the embedding models, gradient-based optimization search would be ap- plied in the parameter space. However, different from continuous groups whose group parameter are also continuous, the parametrization of a discrete group uses discrete values, which brings in extra challenges for the training procedure.
With the two reasons above, thus mainly consider continuous groups which more reasonable choices. Although for completeness, we also compare the group Da as it is implemented in DihE- dral (Xu & Li 2019). Two other important feature of group are compactness and commutativity, which we would mention in each case below.
Besides the relational embedding group G, the entity embedding space and the similarity measure also need to be determined. As discussed above, the entity embedding should be proper rep space of G. While for similarity measure d(-), we choose among the popular ones including -n-norms (I,p) and the cos-similarity (cos). We would notice many choices reproduce some precedent works.
We summarize the results of several chosen examples Table I and put thorough discussion in Appendix(D D Note in the [T some groups have not been studied, but there are still some existing models which use a quite similar embedding space. The major gap, between the existing models and their group embedding counterparts, is the constraint from group structures on the parametrization. which is discussed in details in Appendix D
See Appendix B for more detailed explamation and practical examples.
4

---

Under review as a conference paper at ICLR 2020
Group T
Space R"
Commutativity
d(-)
Studied
Related work
abelian
Lp
TransE (Bordes al. 2013)
((1)
C
abelian
Lpp
RotatE (Sun et al. 2019)
U(1)
T
abelian
lpp
TorusE (Ebisu Ichise 2018
SO(3)
nonabelian
Lp
(2)
C2
nonabelian
Lp
GL(1R
RT
abelian
COs
DisMult (Yang et al. 2014)
GL(1CC)
abelian
cos
ComplEx Trouillon al. 2016)
GL(nn R.)
R"
nonabelian
RESCAL (Nickel et al. 2011)
Aff (
nonabelian
Lp
TransR (Lin et al. 2015)
D4
R
nonabelian
Lp
DihEdral (Xu Li 2019)
Table Examples of the group embedding.
4 GROUP EMBEDDING FOR KNOWLEDGE GRAPHS
In this section, would firstly provide general recipe for the group embedding implementation, and then provide an explicit example, which applies a continuous nonabelian group that has not been studied in any precedent works before.
4.1 A GENERAL GROUP EMBEDDING RECIPE
We summarize the group embedding procedure as following:
1. Given a graph, choose a proper group & for embedding. The choice may concern property of the task, such as commutativity and on. And stated above, in most general cases, nonabelian continuous group should be proper.
2. Choose rep-space for the entity embedding. For simplicity, could multiple (n) copies of the same rep which the case of most existing works. Suppose is a p-dim rep, then the total dimension of entity embedding would be pn, which is written vector Ue.. Roughly speaking, k captures the relational structure and n encodes other feature.
3. Choose proper parametrization of G, that is, choose set of parameters indexing all group elements in G. Suppose the number of parameters required to specify a group element is q, then the total dimension of relation embedding E, would be qn. A group element can now be expressed as a block-diagonal matrix R,, with each block M; being a p x matrix whose entries determined by the vector J,
4. Choose a similarity measure dl-)). the score value of triplet (ei, r, e2) is then:
d RR Vegs [og]
4.2 AN EXAMPLE OF NONABELIAN GROUP EMBEDDING: SU2E
The 2D special unitary group (2 is one of the simplest continuous nonabelian group. As an illustrative demonstration, we construct an embedding model with SU(2) structure and implement in real experiments. As we mentioned in Table SO(3) is also a nice example group. The reason study SU( rather than SO(3), from well-known mathematical fact": the universal cover of (3)3) is Spin(3) which is isomorphic to SU))); or in the representation language, any rep of SO(3) corresponds to an integer rep of SU) This reveals that a study of SU(2) more general and could easily cover the theory of 80(3).
Following the general recipe above, after determining the group G (U),), we choose proper rep-space for entity embedding: [C2] eon which consists n-copies of C Each C² subspace trans- forms as the standard rep-space (U).). All relations thus act as 2n x 2n block diagonal matrix, with each block being 2 2 complex matrix carrying the standard representation of (U)...
5

---

Under review as a conference paper at ICLR 2020
Next, we choose a proper parametrization of (U) An analysis with the corresponding Lie algebra su(2) shows that any group element could be written as (Hall & Hall 2003):
ta (n(nnJ] cos al + sin on J,
(10)
where a is rotation angle taken from [0, 2 and is unit vector on 2-sphere, represented by two other angles (8, o); moreover, the symbol i means an identity matrix, and are three generators of the group (2): (Ja, Jy: J2), which, in the standard rep have the following form:
Jx 0
0 - 0
1 0 -1
J2
(11)
Put all together, our group embedding is then fixed as:
22. 82: e, = (01 i1 02 fg.
Ue = (I1, V1,
In Un), Onz nn);
Ve E: Vr E R;
(12)
where v; and ys are complex numbers, and o and nt = (0,, (e)) represent angles. a triplet, the relation o, acts as block diagonal matrix R., with each 2 x 2 block M, parametrized as (Hall & Hall 2003):
cos or, + issnn or sin 0g ie- sin og cos 0g cos i isin or sin 0g
i -e-i0 siprssino cos 02
operation of relation r on e would act each block matrix M, in the subspace of (I,, y.). This completes the discussion of group embedding. We design the model loss function follows:
L log (7 (e 02))
Σ p (c e'i) log (d, (e/s: e22i) -7)
(13)
exp afr (e))) p (o/y, ezj ((eni ri, 02i)))
(14)
Σ, exp afr (c c2)
where the Sigmoid function, y is the margin used to prevent over-fitting, d, is the similarity measure, the conventional L2-norm. e;; and ez are negative samples while p (e T, e is the adversarial sampling mechanism with temperature a we adopt self-adversarial negative sampling setting from (Sun et al. 2019). fr is L2-norm score function. We term the resulting model as SU2E. For other implementation details, we would mention the next section.
EXPERIMENTS
EXPERIMENTAL SETUP
Datasets: The most popular public knowledge graph datasets include FB15K (Bollacker et al., 2008) and WN18 (Miller 1995). FB15K-237 (Toutanova & Chen 2015) and WNT8RR (Dettmers) al. 2018) datasets were derived from these two, in which inverse relations were removed. FB15K dataset is a huge knowledge base with general facts containing 1.2 billion instances of more than 80 million entities. For benchmarking, usually, a frequency filter was applied to obtain occur- rence larger than 100 resulting in 592,213 instances with 14,951 entities and 1,345 relation types. WN18 was extracted from WordNet (Miller 1995) dictionary and thesaurus, the entities are word senses and the relations lexical relations between them. It has 151,442 instances with 40,943 entities and 18 relation types.
We use three categories of protocols for evaluations, namely, cut-off Hit ratio (H@N), Mean Rank(MR) and Mean Reciprocal Rank (MRR). H@N measures the ratio of correct entities predictions at top prediction result cut-off. Following the baselines used in recent literatures, we chose n = 1, 3, 10. MR evaluates the average rank among all the correct entities, MRR the average rank inverse rank of the correct entities.
Evaluation Protocols:

---

Under review as a conference paper at ICLR 2020
Implementation Details: We implemented model using pytoret framework and exper- imented on a server with an Nvidia Titan-1080 GPU. The Adam (Kingma & Ba 2014) opti- mizer used with the default 31 and 32 settings. A learning rate scheduler observing vali- dation loss decrease was used to reduce learning rate by half after patience of 3000. Batch-size was set 1024. We did grid search
the following hyper-parameters: embedding dimension d E {100, 200, 250}; learning rate 7 E {le - 4, 3e 4, 5e 5): number of negative samples during training Rneg E {128, 256}; adversarial negative sampling temperature {0.5, 1.0); loss function margin E {6, 9, 12, 24, 36, 48).
5.2 RESULTS AND MODEL ANALYSIS
Empirical results from FB15k and WN18 are reported in Table We compared the embedding results of different groups, including T, ((1,, GL(1RR) GL(11 Da. and SU(2), which are categorized by two major properties: the commutativity and the continuity. As discussed Sec. 3.3 these groups have been implicitly applied in existing state-of-the-art models.
For (U),,, we report the result of our own experiments. Results the other models are taken from their original literature: TransE using group T was proposed in [ordds et al. (2013); RotatE using group (1) was proposed in Sun et al. (2019) while TorusE with the same group was proposed implemented in DisMult (Yang et al. 2014) with
in Ebisu & Ichise (2018); group GL(I,*
IR and in ComplEx (Trouillon et al. 2016) with = C: and DihEdral with group D4 was
proposed in Xu & Li (2019).
Group
Commutativity
Continuity
MR
MRR
H@
H@3 3
H@ 10
Example
abelian
continuous
0.463
0.297
0.578
0.749
TransE
U(1)
abelian
continuous
40
0.797
0.746
0.830
0.884
RotatE
U(1)
abelian
continuous
0.733
0.674
0.771
0.832
TorusE
(L,
abelian
continuous
0.654
0.546
0.733
0.824
DistMult
C)
abelian
continuous
0.692
0,599
0.759
0.840
ComplEx
D4
non-abelian
discrete
0.728
0.648
0.782
0.864
DihEdral
SU(2)
non-abelian
continuous
42
0.776
0.710
0.823
0.881
SU2E
Table 2: Link prediction results on FB15K dataset
Group
Commutativity
Continuity
MR
MRR
H@
H@3
H@ 10
Example
T
abelian
continuous
0.495
0.113
0.888
0.943
TransE
U(1)
abelian
continuous
309
0.949
0.944
0.952
0,959
RotatE
U(1)
abelian
continuous
0.947
0.943
0.950
0.954
TorusE
GL(1,R)
abelian
continuous
0.822
0.728
0.914
0.936
DistMult
GL(1,C) C)
abelian
continuous
0.941
0.936
0.945
0.949
ComplEx
DA
non-abelian
discrete
0.946
0.942
0.949
0.954
DihEdral
SU(2)
non-abelian
continuous
207
0.950
0.944
0.954
0.960
SU2E
Table Link prediction results on WN18 dataset
Results on datasets FB15K-237 and WN18RR are demonstrated in Table 4 and respectively. We remove TorusE from the tables due to the absence of results in the original work, and use results in Nguyen et al. (017) for TransE.
In the FB15k dataset, the main hyper-relation is anti-/symmetry and inversion. The dataset has vast amount of unique entities. Shown in Table 2 the RotatE model achieved good performance in this dataset. We should also point that RotatE model result is achieved on large embedding dimension setting namely 1000 (effective 2000) while we use a setting of only 250 (effective 1000) across all the tasks. SU2E achieved comparable result across the metrics. On the other hand, in FB15k-237 dataset since inversion relations are removed, the dominant portion of hyper-relations becomes the composition. We can see RotatE fail short on non-abelian hyper-relations in this task. Shown in 4 the continuous non-abelian group method SU2E outperformed most of the metrics. The DihEdral model suffers from the problems we mentioned in Section 3.3
$ hhtpss//www.pyoorch.rr

---

Under review as a conference paper at ICLR 2020
Group T
Commutativity
Continuity
MR
MRR
H@1
H@3
H@
Example
abelian
continuous
357
0.294
0,465
TransE
U(1)
abelian
continuous
177
0.338
0.241
0.375
0,533
RotatE
GL(1, R)
abelian
continuous
0.241
0.155
0.263
0,419
DistMult
GL(1, C)
abelian
continuous
0.247
0.158
0.275
0.428
ComplEx
14 SU(2)
non-abelian
discrete
0.300
0.204
0.332
0.496
DihEdral
non-abelian
continuous
169
0.340
0.243
0.376
0.532
SU2E
Table 4: Link prediction results on FB FB15 dataset
Group
Commutativity
Continuity
MR
MRR
H@1
H@3
H@ 10
Example
abelian
continuous
3384
0.226
0.501
TransE
U(1)
abelian
continuous
3340
0.476
0.428
0.492
0.571
RotatE
GL(1, R)
abelian
continuous
0.430
0.390
0.440
0.490
DistMult
GL(1,
abelian
continuous
0.440
0.410
0.460
0.510
ComplEx
14
non-abelian
discrete
0.486
0.442
0.505
0.557
DihEdral
SU(2)
non-abelian
continuous
2968
0.474
0.424
0.490
0.574
SU2E
Table
Link prediction results on WNI8RR dataset
In the WN18 dataset, the SU2E outperformed all the baselines on all metrics shown in Table The recent model RotatE and DihEdral achieved comparable results. The WN18RR dataset removes the inversion relations from WN18, left only 11 relations and most of them are symmetry patterns. We can see from Table 5 the DihEdral model and SU2E model performed well on this dataset due to their nonabelian nature. Besides, as DihEdral uses discrete parametrization in relation embedding, it may benefit in special cases where the number of relations is extremely small (11 in this task).
Drawn from the experiments, two factors significantly impact the embedding model performance: the embedding dimension, and group attributes (including commutativity and continuity). Our cur- rent study provides the first systematic analysis of the second factor. As theoretically analyzed in Section 3.2 and empirically shown above, continuous nonabelian groups are more reasonable choices for general tasks. It is important to note that the SU2E proposed above plays as exam- pling model for our group embedding framework, and it uses the simplest continuous non-abelian group. Much more efforts could be devoted in this direction in the future.
CONCLUSION AND FUTURE WORK
We proved for the first time the emergence of a group definition the KG representation learning. This proof suggests that relational embeddings should respect the group structure. A novel theoretic framework based on group theory was therefore proposed, termed as the group embedding of rela- tional KGs. Embedding models designed based our proposed framework would automatically accommodate all possible hyper-relations, which are building- blocks of the link prediction task.
From the group-theoretic perspective, we categorize different embedding groups regarding commu- tativity and the continuity and empirically compared their performance. We also realize that many recent models correspond to embeddings using different groups. Generally speaking, continuous non-abelian group embedding should be powerful for generic KG completion task. We demon- strate this idea by examining a simple exampling model: SU2E. With SU(2) as the embedding group, it showed promising performance in challenging tasks where hyper-relations become crucial.
In the proposed framework, beside embedding relations as group elements, entity embeddings live in different representation space of the corresponding group. And therefore an investigation of group representation theory in entity embedding is highly demanded. We leave this future works. On the other hand, although empirical evaluations focus linear models, it is important to note that the proof of the group structure only relies on the KG task itself. This means our conclusion also works for more general models, including neural-netwe net-ork- based ones. Beyond KG embeddings, the same analysis could be applied to other representation learning where intrinsic relational structures are prominent, An implementation of group structures in more general cases would be very interesting.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collab- oratively created graph database for structuring human knowledge. In SIGMOD Conference, pp. 1247-1250, 2008.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th Interna- tional Conference on Neural Information Processing Systems Volume 2, NIPS 13, pp. 2787- 2795, USA, 2013. Curran Associates Inc. URL http: al a.m. org/ hwtp:/iiaiiiiiiiiooooo a.t.ooo/iitiiiiiiioooo.o. ofm?
1d-2999792. 2999923
Chen Cai. Group representation theory for knowledge graph embedding. arXiv:1909.05100, 2019.
arXiv preprint
Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedingy of the 32th AAAI Conference Artificial Intelli- gence, pp. 181 1-1818, February 2018. Uttpttps: /ttpipp://rxxi....o www.i..wogg........a..o. htrgg/bbs17077 00476
Takuma Ebisu and Ryutaro Ichise. Toruse: Knowledge graph embedding on a lie group. In Pro- ceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, Febru- ary 2-7, 2018, pp. 1819-1826, 2018. URL https: www. aaai orq/ ccs index.nex... php/
MAAI/AAAA18/Paaa
B. Hall and B.C. Hall. Lie Groups. Lie Algebras, and Representations: An Elementary In- troduction. Graduate Texts Mathematics. Springer, 2003. ISBN 9780387401225. URL
https:://oon httpss////ooss.ooooeee.. com/ cotp//ookkss.1a11111111ee.aa
Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu Hc, Zhanyi Liu, Hua Wu, and Jun Zhao. An end- to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 221-231, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/11/111-10011 URL https: www. aclweb. org/anthin wotphhooooyyyppy///eeeee
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1411 190./14,,,690, 2014.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation em- beddings for knowledge graph completion. In Proceedings of the Twenry-Ninth AAAI Conference Artificial Intelligence, AAAI115, pp. 2181-2187 AAAI Press, 2015. ISBN 0-262-51129-0. URL http:://lll... cm www.a/.....oo.oo....ooooo.o ctt.:/iaaaaatt ion wwt.::mmmmmcccc.. 228866661128888694 2886624
George A. Miller. Wordnet: lexical database for english. Commun. ACM. 38(11):39 41, Novem- ber 1995. ISSN 0001-0782, doi: 10. 10.1145/219717.11974. URL http:://o/// acm org/
11.1544/1 219499
Dai Quoe Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based on convolutional neural network. arXiv preprint 17Xiv:1712.02121.. 2017.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multionelational data. Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML'1 11, pp. 809-816, USA, 2011. Omnipress. ISBN 978-1- 4503-0619-5. URL http: acm. www.a/c......tcciicooocoooooooee wtt.:/oww....mmmmddd.....33 3104444
Zhiqing Sun, Zhi-Hong Deng, Jin Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019. URL wtt.tttps:// /etts://openreet wwt. wtt.rrrrrmmmm.meoooooeooooeeeee

---

Under review as a conference paper at ICLR 2020
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pp. 57-66, Beijing, China, July 2015. Association for Computational Lin- guistics. doi: 00.1665/w1/W11-40007 URL https::pp//............t /ttp:///wwww//// ac web. warrg/anthloogg ogy/
4007
Théo Trouillon. Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In Proceedings of the 33rd International Con- ference on International Conference on Machine Learning Volume 48, ICML'16, pp. 2071- 2080. JMLR.org, 2016. URL http://dd acm. httg://itatioon.ffi...i......
centiion.cf?icce
3045609
Canran Xu and Ruijiang Li. Relation embedding with dihedral group in knowledge graph. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp- 263-272, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19-1026. URL http:: http://wwwwww..../cccccccocc htt.://www....aaaoooooog/o//////
1026
Bishan Yang. Wen tau Yih, Xiaodong He, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference in knowledge bases. CoRR, abs/1442.6557, 2014.
10

---

Under review as a conference paper at ICLR 2020
A PROOF OF HYPER- IN KNOWLEDGE GRAPH RELATIONS
The associativity is actually rooted our definition of r1 r2 in equation 6 through the subsequent operating sequence in the entity space:
O Opera [.] Or, [Or,1-]]
(15)
from which, we can derive directly that:
(ri-r)) [-] O r2 [Or
On Or, 1 10
Or, rrr. [:]]
(16)
which then leads to the association equation below:
(ri r2) r3 = ri (r2 - ra).
(17)
This completes the proof of associativity in knowledge graphs
B AN EXPLICIT EXAMPLE OF HYPER-ASSOCIATIIVE
While the other four properties derived in the main part easy to comprehend, the associativity may not be intuitive. To help readers understand the practical meaning of associativity in real life cases, here we provide simple example of the relational typer-associatiiityy
isBrotherOf.
r2 isMotherOf.
r3 isFatherOf.
(18)
Meanwhile, the following composition relations are also meaningful:
ri r2 isUnleeOf, F2 r3 isGrandmotherOf.
(19)
In this example, one could easily see that:
(rı r2) rg = r1 (r2 r3) = isGranduncleOf.
(20)
This completes simple demonstration of the hyper-associativity.
C EXAMPLES OF NON-COMMUTATIVE RELATIONS
We provide simple illustrative examples for non-commutative compositions. following real world kinship:
We consider the
ri isMotherOf. r2 isFatherOf.
(21)
Clearly, the composition and r2 correspond to isGrandmothhrOOf and isGrandfatherOf relations respectively, which are different. This is simple example non-commutative cases.
D DETAILED DISCUSSION OF EXAMPLING GROUP EMBEDDINGS
In this appendix, we give a thorough discussion on examples listed the Table [ We would explain in details some minor differences between models obtained directly from group embedding theory here, and existing models which may not concern/obey group structures in precedent works.
D.1 GROUP: T
Use multiple (n) copies of T. the translation group, for the relation embedding- This is a noncom- pact abelian group. The simplest rep-space would be the real field R, which should also appear n times R The group embedding then produces the following embedding vectors:
T_ = (I1,
In),
Ve E E;
J, = (81, 82, ..., 6n). Vr E R;
(22)
11

---

Under review as a conference paper at ICLR 2020
both of which are n-dim. Here both 2; and ò; are real numbers. In triplet, the relation i acts as an addition vector added to the head entity e1. If one further chooses Ly- -y-norm as the similarity measure:
(23) this actually corresponds to the well-known TransE model (Bordes al. 2013). There was regularization in the original TransE mode that changes the entity rep-space, which however has been removed in many later works by properly bounding the negative scores.
[ E21pp
D.2 GROUP: (()
Use n-copies of (1), the 1-dim unitary transformation group, for the relational embedding. This compact abelian group. The simplest rep-space would be the real field C, which should also appear times as C The group embedding then produces the following embedding vectors:
(x) F2:
Fn): Sn),
Ve E E;
r
8 = (0), 02,
Vr E R:
(24)
where r; is complex number containing both real and imaginary part, while oi is phase vari- able take values from 0 to 2x. Therefore the eebiid-ibb dimension is 2n, while the relation dimension is n. In triplet, the relation J. acts phase shift on the head entity es. In matrix form, one could define R, as the diagonal matrix with the i-th diagonal element being e If one further chooses Lp no-norm the similarity measure:
|R, Teg J. ll = [I[e"]
oe - 22
(25)
where means Hadamard product. This precisely leads to the RotatE model (Sun et al.
2019
On the other hand, one could also use the n-torus T as the rep-space:
U (0), 02.
6n). Ve E E
U, (01, 02.
2n).
Vr E R;
(26)
where 0, represents a coordinate on the torus. Still using the norm similarity measure:
RR Tee Je 1pp l e
(27)
which leads to the TorusE model (Ebisu & Ichise 2018) In the original implemnentttonn of TorusE, there is an additional projection π from R" to T".
D.3 GROUP: SU(
The 2D complex special unitary group (U) is one of the simplest continuous nonabelian group, which is also the one we use in our instantiation. Use n-copies of (U), the 2-dim special unitary group, for the relational embedding. This a compact nonabelian group. The natural choice of entity rep-space would be [C which consists of n-copies of C Each C subspace transforms as the standard rep-space (U)... All relations thus act as 2n x 2n block diagonal matrix, with each block being 2 x 2 matrix carrying the standard representation of (2). our group embedding then fixed as:
0. (Ii, F2, 12: ".", Fn, Wn),
Ve E;
0, = (a1. fig 02 ng. ..., an nn)
Vr R;
(28)
where x; and yi complex numbers, and o, and n. (0;, di) represent angles. In triplet, the relation 7, acts as a block diagonal matrix R,, with each 2 x 2 block M, parametrized as:
cos as + -isinn ar sin 0,
- sin a; cos cos az sin ae sin 0:
ie 10: sin a cos
An operation of relation would act each block matrix M, in the subspace of (#,, (z). complete model form after Lp-norm is implemented would be:
9:). The
R
Jep Vop llp
(29)
12

---

Under review as a conference paper at ICLR 2020
D.4 GROUP: SO(3)
The 3D rotation group S0(3) is also one of the simplest continuous nonabelian group and has not been studied yet. A proper rep-space for entity embedding would be [R*]@n which consists of n- copies of R Each I subspace transforms as the standard rep-space of SO(3). All relations thus act 3n 3n block diagonal matrix, with each block being 3 x 3 matrix carrying the standard representation of S0(3).
Next, we choose a proper parametrization of SO(3). Instead the more general angular momentum parametrization, due to our choice of using the standard representation, we could parameterize the (3) elements using Euler angles (Ø, 8, v2), which is easier for implementation.
Put all together, our group embedding is then fixed as:
0. = (2), 1/1, #1, T2, 12, 22, "H Tn. Nn: Zn); 1h, 02, 1/2, ... on. fn vn),
Ve E E
r
ह, = (Φ), ( 1,
Vr E R;
(30)
both of which are 3n-dim. In triplet, the relation vector o. acts as a block diagonal matrix R,, with each 3 x 3 block M, parametrized as following
cos tht cos Φ; cos 0g sin 4/s sin Φ; cos t/s sin Φi + cos fg cos th: cos di sin 1i, cos 6; cos 0 sin b); cos oi -in dh sin Φ; + cos f2 cos u/s cos Φ; cos 1 sin 0;
sin the sin cos th, sin 0 cos 0g
- cos fbs cos 0;
operation of relation r on e would act each block matrix M, in the subspace of (Iz y/s 2.). The complete model form after Lp norm is implemented would be:
RR, Je, - Fezllp
(31)
D.5 GROUP: GL(m, V)
This is class of groups (m-dim general linear groups defined field-V). We separate the discus- sion into two parts according the commutativity property of the group:
1. Group GL(,, V): Use n-copies of GL(1, V). the I-dim general linear group defined on Field-V, for the relational embedding. This a noncompact abelian group. Suppose cos- similarity is used, and, most naturally, the entity rep-choice is chosen as V One has at least two simple choices for the field-V: either real or complex. For the real field F, embeddings are:
T. (I1, F2.
VC E;
i (s1, $2,
sn)), Vr R;
(32)
with both x; and s; being real numbers. In a triplet, the relation vector i, acts as a diag- onal matrix R,, with the i-th element being s;. If one uses a cos-similarity which can be captured by the inner product, the eventual form would be:
(33) which leads to the DisMult model. And similarly a complex field C leads to a model struc- ture similar to ComplEx (Trouillon et al., 2016). There is minor difference in ComplEx: authors actually use the cos-similarity in a projected subspace (the real-subspace of C"))
JT R, -
Re
R [R,]
(34)
2. Group GT,nn, V): Use GTín, V), the n-dim general linear group defined on field-V, while keeping both the similarity measure (cos- similarity) and entity rep-space (V") choices un- changed. This is noncompact nonabelian group. The choice of V = R would leads to a model similar to RESCAL (Nickel et al., 2011). However, different from the corre- spondences above, the original RÉSCAL model does not have built-in group structure: it uses arbitrary n X n real matrices, some of which may not be invertible, and hence are not group elements in (n R). It is, therefore, worth to add the extra invertible constraint in RESCAL, which requests matrices constructed through group parameterization rather than assigned arbitrary matrix elements.
"See Appendix for detailed explanation.
13

---

Under review as a conference paper at ICLR 2020
D.6 GROUP: AFF(V)
Use Af. the affine group defined the field-V, V, for the relational embedding. This is non- compact nonabelian group. One could choose V IR" since the entity rep-space should always share the same one. If one further takes Ln-or as the similarity measure, the embedding model would be quite similar to that in TransR (Lin et al. 2015) relation and entity embeddings share the same dimension. However, similar to the RESCAL case, important distinction is the original model does not explicit constraint the relational embedding on group manifold. An extra invertible requirement would then produce exactly group embedding model as proposed here.
E EXPLANATION OF EULER ANGLES AND RELATION OPERATIONS
In this appendix section, we provide pictorial explanation for Euler angles used in the exampling model of (0)3).
The idea is that all 3D rotations that preserve orientation could be decomposed into the following subsequent rotations:
sin e
0
D =
sin Øb
COs Φ
O
(35)
0
1
0
C
cos
sin
(36)
0
sin 0
0
008 w/
sin m/
B
sin vb
cos t/
(37)
0
0
The composed operation M = BCD could then be written as:
cos cos c cos sin sin sin 1/ cos e cos sin ab cos cos vb sin
cos s sin ø + cos & cos e cos
sin sin A cos 1 sin
sin vi sin ø + cos 0 cos di cos
s cos
O
which is the matrix used in the main text. The meaning of the above three rotations can be easily visualized following:
Z,
Y
Z
Yi
W
Figure visualization of Buler angles.
14

---

Under review as a conference paper at ICLR 2020
The action in the whole R3a space looks like:
21
M
91
3/2
0
M2
9/2
2/2
(38)
In
0
M
Yn
Zn
15


Reviews:
[{"review_id": "SylHMCB0FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper start merely by studying the graph reconstruction problem and prove that the intrinsic structure of this task itself automatically produces the complete definition of groups. it seems to be a novel result. Based on this result, one could construct embedding models that naturally accommodate all possible local graph patterns, and the paper also shows a few simulations.\n\nMy main concern is that, while the focus on this work is the theoretical finding, there is no rigorous statement of it as a theorem. As a result, I am not exactly sure what the proofs in the appendix is trying to show. In addition, the proofs seems to be very trivial.\n\nFor the algorithm section, I feel that it is also lacking in the sense that there is still no automatic way to choose which group to embed. It is also unclear what is the purpose of the simulation section. While it says \"As theoretically analyzed in Section 3.2, and empirically shown above, continuous nonabelian groups are more reasonable choices for general tasks\", the advantage of continuous nonabelian groups are not so significant in the tables. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SygVUgzHqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors approach the problem of representing knowledge graphs from a top-down perspective, in which they argue for certain fundamental desiderata for hyper-relations (composition, inversion) followed by properties that define a mathematical group.\n\nI found the paper extremely difficult to follow. As defined in Eq. 1, knowledge graph embeddings are a model family with a choice of domain for the entity and relation, and a choice of how that relation operates on a head entity. This means one can devise arbitrary properties and restrictions on that family. It's not clear to me what motivates selecting a (abelian) group, where inversion, closure, identity, associativity, and commutativity are demanded to be properties of knowledge graph embedding models. This seems more a definition of what models they consider, rather than a novel insight about knowledge graph embedding models itself (the authors claim \"we proved for the first time the emergence of a group definition in the KG representation learning\", which seems hard to wrap one head's around).\n\nGiven this overarching family of models, the authors proceed to identify existing models as certain choices of that family.  I see little use in inventing this abstraction as the authors do not show any practical insights, or interesting theoretical analysis that comes from this higher-level abstraction.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg7z_3TKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I would like to thank the authors for the detailed rebuttal. \n\nMy concern is generalizing something that is not correctly describing the object we should model (KGs).\n\nFor instance, \"all properties mentioned in the paper, including closure, identity, inversion, and associativity, are desired properties of relation embedding spaces, rather than existing properties of specific knowledge graphs. \" => should we not care about the actual relations of knowledge graphs? Should we not build a theory that closely matches the properties of the actual objects we are modeling? The argument seems to be: forget the actual knowledge graph, here is a generalization of existing KGE methods. Even the proof provided was unrelated to my question: I gave an example: can you list all relations in FB15K and prove that they fit the group axioms? If they don't fit, what is the advantage of modeling the relations as a group? What is the representation error this causes? \n\n\nThe authors misunderstood my comment about (Bloem-Reddy and Teh 2019). \"Therefore the previous works mentioned by the reviewer above are irrelevant.\" => this is a weird statement. Permutation invariance is relevant to all objects that are graphs. I recommend reading the long history relating graph models and exchangeability (Persi Diaconis has a good overview). I was just mentioning that I don't see why KGs should have another associated permutation group besides permutation invariance. I would also recommend following the emerging literature in graph representation learning using group theory, of which KGs are but a special case.\n\n\"The (entity) manifold, however, means that the relations no longer form a group. I see no easy fix.\" => I clarified it, since I was talking about the entity embedding.\n\nI did not see a revised version of the manuscript.\n\n\nI will raise my score because the paper could be published as a niche paper. We generalize KGE but we acknowledge its shortcomings: And here is a way to measure the embedding error of modeling the relations as a group, when they are not actually a group. \n\n--------------\n\nThe paper is well written and an interesting read. I think it complements well the existing literature. \n\nUnfortunately, I think the paper overstate its claims. It is clear that permutation groups are the natural language of all graphs (Kondor and Trivedi, 2018) and (Bloem-Reddy and Teh, 2019). It is less clear that knowledge graphs also comply with another set of group axioms. Why would the set of relations be closed? Say, Owns \\cdot Spouse_of = ? . I cannot see why and how this is necessary. Must all the relations be closed? I understand why this is true for KGE but this is not true for KGs, which only shows that KGEs may not be the right method to represent KGs moving forward. I see no easy fix. \n\nThe rest of the paper is straightforward, just applying the definitions of groups. I found the classification of different methods interesting, and should be made more clear in the experiments. \n\nThe experimental results are interesting but their significance is unclear. Please add standard deviations to all experiments. We cannot have a sense of the significance of the results without knowing how many runs were executed, how they were executed (e.g., k-fold cross-validation, bootstrapping), and their standard deviation.\n\nI dispute that the paper proves that the definition of groups emerges purely from the nature of the knowledge graph embedding (KGE) task. I would actually say the opposite, it is clear to me that for most KGs, their relations do not fit the definition of a group. I am open to be (formally) proven wrong. E.g., can you list all relations in FB15K and show that they fit the group axioms?\n\nFixing the paper: Maybe the paper could be rewritten, constrained to Euclidean spaces? Then prove (formally) that the group axioms are a sufficient(?) and necessary(?) condition for such embeddings? \n\nMinor issues:\nhonestly could not make sense of the abstract:\nThe sentence choices\n-  \"which suggests that a group-based embedding framework is essential for model design\" \n- \"Our theoretical analysis explores merely the intrinsic property of the embedding problem itself without introducing extra design\"\nin the abstract are very strange. It means nothing to a reader at that point. What a model design? What is an extra design? A modeling assumption? A model prior? \n\n\"Using the proposed framework, one could construct embedding models that naturally accommodate all possible local graph patterns, which are necessary for reproducing a complete graph from atomic knowledge triplets\" => what is an \"atomic knowledge triplet\"? Why are they necessary to reproduce a complete graph? This is all very confusing.\n\n\" contradicts to the entity regularization\" =>  contradicts the entity regularization\n\nReferences:\nKondor, R. and Trivedi, S., 2018. On the generalization of equivariance and convolution in neural networks to the action of compact groups. ICML\nBloem-Reddy, B. and Teh, Y.W., 2019. Probabilistic symmetry and invariant neural networks. arXiv:1901.06082.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a rigorous mathematical framework for knowledge graph embedding. The paper received 3 reviews. R1 recommends Weak Reject based on concerns about the contributions of the paper; the authors, in their response, indicate that R1 may have been confused about what the contributions were meant to be. R2 initially recommended Reject, based on concerns that the paper was overselling its claims, and on the clarity and quality of writing. After the author response, R2 raised their score to Weak Reject but still felt that their main concerns had gone unanswered, and in particular that the authors seemed unwilling to tone down their claims. R3 recommends Weak Reject, indicating that they found the paper difficult to follow and gave some specific technical concerns. The authors, in their response, express confusion about R3's comments and suggest that R3 also did not understand the paper. However, in light of these unanimous Weak Reject reviews, we cannot recommend acceptance at this time.  We understand that the authors may feel that some reviewers did not properly understand or appreciate the contribution, but all three reviewers are researchers working at highly-ranked institutions and thus are fairly representative of the attendees of ICLR; we hope that their points of confusion and concern, as reflected in their reviews, will help authors to clarify a revision of the paper for another venue. 


Decision: Reject