Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
Authors: Yushi Bai
Zhitao Ying
Hongyu Ren
Jure Leskovec
Venue: NeurIPS 2021 Poster (2021)
Keywords: Machine Learning
Knowledge Graph
Hyperbolic Embedding
Hierarchical Reasoning

Abstract: Hierarchical relations are prevalent and indispensable for organizing human knowledge captured by a knowledge graph (KG). The key property of hierarchical relations is that they induce a partial ordering over the entities, which needs to be modeled in order to allow for hierarchical reasoning. However, current KG embeddings can model only a single global hierarchy (single global partial ordering) and fail to model multiple heterogeneous hierarchies that exist in a single KG. Here we present ConE (Cone Embedding), a KG embedding model that is able to simultaneously model multiple hierarchical as well as non-hierarchical relations in a knowledge graph. ConE embeds entities into hyperbolic cones and models relations as transformations between the cones. In particular, ConE uses cone containment constraints in different subspaces of the hyperbolic embedding space to capture multiple heterogeneous hierarchies. Experiments on standard knowledge graph benchmarks show that ConE obtains state-of-the-art performance on hierarchical reasoning tasks as well as knowledge graph completion task on hierarchical graphs. In particular, our approach yields new state-of-the-art Hits@1 of 45.3% on WN18RR and 16.1% on DDB14 (0.231 MRR). As for hierarchical reasoning task, our approach outperforms previous best results by an average of 20% across the three datasets.

Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
Yushi Bai" bys18 8maallm ts tsinng sinhhuu edu. cn Tsinghua University
Rex Ying" stayinggstanforoe edu Stanford University
Hongyu Ren
Jure Leskovec
hyrenêcs. stanford. edu Stanford University
jureêca. stanford. inffordeedu Stanford University
Abstract
Hierarchical relations are prevalent and indispensable for organizing human knowl- edge captured by knowledge graph (KG). The key property of hierarchical relations is that they induce partial ordering over the entities, which needs modeled order to allow for hierarchical reasoning. However, current KG embeddings can model only a single global hierarchy (single global partial order- ing) and fail to model multiple heterogeneous hierarchies that exist in a single KG. Here present ConE (Cone Embedding). KG embedding model that is able to simultaneously model multiple hierarchical as well as non-hierarchical relations in a knowledge graph. ConE embeds entities into hyperbolic cones and models relations as transformations hetween the cones. In particular, ConE uses containment constraints in different subspaces of the hyperbolic embedding space to capture multiple heterogeneous hierarchies. Experiments on standard knowledge graph benchmarks show that ConE obtains state-of-the performance on hierarchical reasoning tasks as well as knowledge graph completion task on hierarchical graphs. particular,
approach yields new state-of-the-ar Hits @ of 45.3% on WN18RR and 16. % on DDB 14 (0.231 MRR). As for hierarchical reasoning task, our approach outperforms previous best results by an average of 20% across three hierarchical datasets.
Introduction
Knowledge graph (KG) is prevalent data structure that stores factual knowledge in the form of triplets, which connect two entities (nodes) with relation (edge) [[]. Knowledge graphs play important role in many scientific and machine learning applications, including question answering 2 information retrieval (3) and discovery in biomedicine [4]. Knowledge graph completion is the problem of predicting missing relations the graph, and is crucial in many real-world applications.
Knowledge graph embedding (KGE) models [5] 6 7 approach the task by embedding entities and relations into low-dimensional vector space and then use the embeddings to learn a function that given a head entity and relation predicts the tail entity
Hierarchical information is ubiquitous in real-world KGs, such as WordNet [ or Gene Ontology [9] since much human knowledge is organized hierarchically. The relations in these KGs can be separated into non-hieren non-hierarchical relations (c.g., friendOf)) and hierarchical relations (e.g., isA,
* Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Non-bierarchical relation -derivationally related form -iister term
derivationally related form
Sugar palm
tree
partor
subCless
Trenket
Hypernym relation partOf
Hyponym relation
-subClass
Pain
subClass
partOr
subClass
Plum free
Wine
Sugo pales
Fint
Maple tree
(a) Multiple heterogeneous hierarchies in knowledge graph.
(b) Hyperbolic entailment cones in 2D hyper- bolic plane for K 0.1.
Figure 1: (a) There are three categories of relations: non- non-hierarchical relation (sister term), hypernym (partOf) and hyponym relation (subClass). Relations induce multiple independent hierarchies. (b) ConE uses d 2D hyperbolic entailment cones to model entity. Entities PalmTree and SugarPalm are connected by a hyponym relation subClass and therefore the cone of PalmTree contains the cone of SugarPalm.
partOf), where oon-hierarchical relations capture interactions between the entities at the same level while hierarchical relations induce tree-like partial ordering structure of entities.
Recent works propose the of variety of embedding geometries such as hyperbolic embeddings, box embeddings, and cone embeddings [10 to model partial ordering property of hierarchical relations, but two important challenges remain: (1) Existing works that consider hierarchical relations 13 do not take into account existing non-hierarchical relations [1]. (2) These methods can only be applied to graphs with a single hierarchical relation type, and thus not suitable to real-world knowledge graphs that simultancously encode multiple hierarchies using many different relations. For example, Figure 1, subClass and partOf each define unique hierarchy over the same set of entities. However, existing models treat all relations in KG as part of one single hierarchy, limiting the ability to reason with different types of heterogeneous hierarchical relations. While there are methods for reasoning over KGs that use hyperbolic space (MuRP [15. RotH [16), which is suitable for modeling tree-like like graphs, the choice of relational transformations used these works (rotation) prevents them from faithfully capturing all the properties of hierarchical relations. For example, they cannot model transitivity of hierarchical relations: there exist triplets (hi, r, ha2) and (h2,, r; ha), then (hi,2 r, ha) exists, iee. hi and ha are also related by relation
Here we propose a novel hyperbolic knowledge graph embedding model ConE. ConE is motivated by the transitivity of nested angular cones [12] that naturally model the partial ordering defined by hierarchical relations. Our proposed approach embeds entities into the product space hyperbolic planes, where the coordinate in each hyperbolic plane corresponds to 2D hyperbolic cone. To address challenge (1), we model non-hierarchical relations as hyperbolic cone rotations from head entity to tail entity, while we model hierarchical relations as a restricted rotation which guarantees cone containment (Figure [(b)). To address challenge (2), we assign distinct embedding subspaces corresponding to product spaces of different set of hyperbolic planes for each hierarchical relation, to enforce cone containment constraints. By doing so, multiple heterogeneous hierarchies are preserved simultaneously in unique subspaces, allowing ConE to perform multiple hierarchical reasoning tasks accurately.
We evaluate the performance of ConE on the KG completion task and hierarchical reasoning task. single trained ConE model can achieve remarkable performance on both tasks simultaneously. On KG completion task, ConE achieves new state-of -f--oh--rr---
the-art art results on two benchmark knowledge graph datasets including WN18RR [5] [7], DDB14 [8] (outperforming by 0.9% and 4.5 % on Hits 1 metric). We also develop novel biological knowledge graph GO21 from biomedical domain and show that ConB successfully models multiple hierarchies induced by different biological processes. We also evaluate our model against previous hierarchical modeling approaches on ancestor- descendant prediction task. Results show that ConE significantly outperforms baseline models (by 20% % average when missing links are included), suggesting that it effectively models multiple

---

heterogeneous hierarchies. Moreover, ConE performs well on the lowest common ancestor (LCA) prediction task, improving over previous methods by 100% in Hits@ metric.
2 Related Work
Hierarchical reasoning. The most related line of work is learning structured embeddings to perform hierarchical reasoning on graphs and ontologies: order embedding, probabilistic order embedding, box embedding, Gumbel-box embedding and hyperbolic embedding [O These embedding-basee methods map entities to various geometric representations that can capture the transitivity and entailment of hierarchical relations. These methods aim to perform hierarchical reasoning (transitive closure completion), such as predicting if an entity is an ancestor of another entity, However, the limitation of the above works that they can only model single hierarchical relation, and it remains unexplored how to extend them to multiple hierarchical relations in heterogeneous knowledge graphs. Recently, [23] builds upon the box embedding and further models joint (two) hierarchies using two boxes as entity embeddings. However, the method is not scalable since the model needs to learn a quadratic number of transformation functions between all pairs of hierarchical relations. Furthermore, the missing part is that these methods do not leverage non- hierarchical relations to further improve the hierarchy modeling. For example in Figure [(a) with the sisterTerm( PalmTree, MapleTree) and subClass( Palmmree, Tree), we may infer MnbClass(MapleTTT Tree). In contrast to prior methods, ConE is able to achieve exactly this type of reasoning as it can simultaneously model multiple hierarchical well non-hierarchical relations.
19
20
21 22).
Knowledge graph embedding. Various embedding methods have been proposed to model entities and relations in heterogeneous knowledge graphs. Prominent examples include TransE [5], DistMult 24], ComplEx (25), RotatE 7 and TuckER 14]. These methods often require high embedding dimensionality to model all the triples. Recently KG embeddings based on hyperbolic space have
shown success in modeling hierarchical knowledge graphs. MuRP [15] learns relation-specife parameters in the Poincaré ball model. RotH [16] uses rotation and reflection transformation in n-dimensional Poincaré space to model relational patterns, and achieves state-of-the-an for the KG completion task, especially under low-dimensionality However, transformations used in MuRP and RotH cannot capture transitive relations which hierarchical relations naturally are.
To the best of our knowledge, ConE is the first model that can faithfully model multiple hierarchical as well as -ieraarcchial relations in a single embedding framework.
ConE Model Framework
3.1 Preliminaries
Knowledge graphs and knowledge graph embeddings. We denote the entity set and the relation set in knowledge graph as & and R respectively. Each edge in the graph is represented by a triplet (h, r, f). connecting the head entity h E & and the tail entity E & with relation r E R. In KG embedding models, entities and relations are mapped to vectors: E > IRds, R - Rdd, Here de, dR refer the dimensionality of entity and relation embeddings, respectively. Specifically, the mapping is learnt via optimizing a defined scoring function RRds x Ad x Rd - IR measuring the likelihood of triplets [16), while maximizing such likelihood only for true triplets.
Hierarchies in knowledge graphs. Many real-world knowledge graphs contain hierarchical rela- tions 110 11, 26). Such hierarchical structure is characterized by very few top-level nodes corre- sponding to general and abstract concepts and a vast number of bottom-level nodes corresponding to concrete instances components of the concept. Examples of hierarchical relations include isA, partOf. Note that there may exist multiple (heterogencous) hicrarchical relations in the same graph, which induce several different potentially incompatible hierarchies (i.e., partial orderings) over the same set of entities (Figure I(a)p. In contrast to prior work, our approach is able to model many simultaneous hierarchies over the same set of entities.
Hyperbolic embeddings. Hyperbolic embeddings can naturally capture hierarchical structures. Hyperbolic geometry is non-Euclidean geometry with constant negative curvature, where curvature measures how geometric manifold deviates from Euclidean space. In this work, we use Poincaré ball model with constant curvature = -1 as the hyperbolic space for entity embeddings [10].
3

---

We also investigate more flexible curvatures, see Appendix B results show that our model is robust enough with constant curvature 11. In particular, we denote d-dimensional Poincaré ball centered at origin as 8 = (x Te ||x|| < 1), where ||-I| is the Fuclidean norm. The Poincaré ball model of hyperbolic space equipped with Riemannian metric:
g = ( 1 ||x|| )2
where ge denotes the Euclidean metric, i.e., g Ia. The mobius addition @ defined on Poincaré ball model with -1 curvature is given by:
x (1 2(x, y) + || X + (1 |x
1 2(x, + X |
For each point x E Bd, the tangent space TxB is the Euclidean vector space containing all tangent vectors at x. One can map vectors in TxB to vectors in B through exponential map cxpx : 8d as follows:
| x
expx (u) x A tanh
(3)
|
Conversely, the logarithmic map logx (-) Bd TxB maps vectors in Ddd back vectors in TxB, in particular:
logx (u) = (1 ||x|]) tanh -11 (11-
vll
(4)
2 v/
Also, the hyperbolic distance between x. y E Bd is:
dg(x,y) = tanh -ttaaan(1 (11-x yy)
(5)
A key property of hyperbolic space is that the amount of space covered by ball of radius r in hyperbolic space increases exponentially with respect to r, rather than polynomially as Euclidean space. This property contributes to the fact that hyperbolic space naturally model hierarchical tree-like structure.
Hyperbolic entailment cones. Each hierarchical relation induces partial ordering over the entities. To capture a given partial ordering, we use the hyperbolic entailment cones [2]. Figure [(b) gives an example of 2D hyperbolic cones.
Let Cx denotes the cone at apex x. The goal is model partial order by containment relationship between cones, in particular, the entailment cones satisfy transitivity:
Vx.y Bda (00) y E Cx = Cy Cx
Also, for y E 1d, we define the angle of to be the angle between the half-lines ox and xy and denote it as Zxy. It can be expressed as:
(x, (1 + |xx13) |x|| (1 11x113)
Lx cos -1
|x|
x yll 1 1+ |x| |yy| 2(xx
To satisfy transitivity of nested angular cones and symmetric conditions expression of Poincaré entailment cone at apex x E Bd,
12 we have the following
||x }}
Cx = [y E Bdd <x & sin -(K
(8)
|x|
where K E R is hyperparameter (we take K = 0.1). This implies that the half aperture Φx of cone Cx as follows:
sin [K 1 ||
(9)

---

Rotation
2D hyperbolic cones
hyperbolic planes
Restricted rotation
k Relation-specific subspace, | for hierarchical relation r
Figure 2: ConE model overview: Embedding space is the product space of hyperbolic planes and ConE learns different transformation in each hyperbolic plane. ConE uses restricted rotation in assigned relation-specifee subspace model each hierarchical relation and enforces cone containment constraint in the subspace so that partial ordering of cones is preserved in the subspace. For hyperbolic planes not in the subspace, we use a general rotation to model How to choose relation-specific subspace for each hierarchical relation is essential and further explained in Sec. 3.3
3.2 ConE Embedding Space and Transformations
We first introduce the embedding space that ConE operates in, and the transformations used to model hierarchical as well non-hic non-hierarchical relations.
For ease discussion let's assume that the relation type is given a priori. In fact, knowledge about hierarchical relations (i.e., transitive closure) is explicitly available in the definition of the relation in KGs such as ConceptNet [28]. WordNet [8] and Gene Ontology (9) When such information is not available, ConE can infer "hierarchicalness" of relation by a simple criteria with slight modification to the Krackhardt scores [29]. see Appendix H
Embedding space. The embedding space of ConE, S, is product space of d hyperbolic planes 30 resulting in a total embedding dimension of 2d. S can denoted as S =B x B x x.. B Note that this space is different from RotH embedding space [16]. which is single 2d-dimensional hyperbolic space. ConE's embedding space is critical in modeling descendad relationships for heterogeneous KGs, since it is multiple hyperbolic planes) to heterogeneous hierarchical relations.
natural when allocating its subspaces (product space of
We denote the embedding of entity / E È as h (h1, b2 ha) where h, B is the apex of the i-th 2D hyperbolic cone. We model relation r as a cone transformation on each hyperbolic plane from head entity cone to tail entity cone. Let r (r1, r2. .ra) be the representation of relation r. We use r; = (s,, 0,) to parameterize transformation for the i-th hyperbolic plane as shown in Figure
>> 0 is the scaling factor indicating how far to go in radial direction and (0; ob. /T) is the rotation angle restricted by half aperture Φn. (0: E [-T, π )). To perform hierarchical tasks such ancestor-dest descenddnd prediction, ConE uses nested cones in each hyperbolic plane to model the partial ordering property of hierarchical relations, by the cone containment constraint in Def. T
Definition 1. Cone containment constraint. If entity h is an ancestor of t, then the cone embedding of has to reside in that of the entity h, i.e., Ct. C Crp Vi E (1....).
The cone containment constraint can be enforced in any the hyperbolic plane components S. Next we introduce ConE's Es transformations for characterizing hierarchical and non-hierarchical patterns of relation r triple (h, r. t). Note that we utilize both transformations model hierarchical relations to capture non oon-hicrarh properties, i.e., symmetry, composition, etc, well hierarchical properties, i.c., partial ordering. We do this by performing different transformations in different subspaces of S, as discussed in detail in Sec. 3.3
Transformation for modeling non-hierarl properties. Rotation is an expressive transforma- tion to capture relation between entities [7]. Analogous to RotatE, we adopt rotation transformation fi model oinhiiearll properties (Figure 3(a)). For rotation the i-th hyperbolic plane,
fi (b, r.) exp. (G, (h,))
(10)
where G0,)) is the Givens rotation matrix:
G(O.) = cos cos(0,) (0:)
sin)
(1I)
sin(O.) cos (0.)
5

---

(a) Cone rotation
(b) Restricted cone rotation
Figure 3: Transformations ConE in Poincaré ball: (a) Cone rotation from to used for non-
hierarchical relations; (b) Restricted rotation from the cone of parent h to the cone of child used for hierarchical relations, where "1" corresponds to scaling and "2" to rotation (s,, 0,) in Eq. 12
We also show that the rotation transformation in Eq. 10 expressive: can model relation patterns including symmetry, anti-symmetry, inversion, and composition (Appendix A.T..
Transformation for modeling hierarchical properties. However, fi cannot directly applied to model hierarchical relations, because rotation does not obey transitive property: rotation by 0; twice will result in rotation of 20,, instead of 0;. Hence it cannot guarantee (h1 r, h2) when (h1, r, h22 and (h2, r, h2) are true. We use restricted rotation transformation f2 to model hierarchical relations. We impose cone containment constraint to preserve partial ordering of cones after the transformation. Without loss of generality we assume relation r is hyponym type relation, the restricted rotation from i-th hyperbolic plane as follows (we perform restricted rotation from to h if is hypernym relation):
Le (h,, r;) expn, (ss GO, h.), ) r, (8s.
(12)
where on, is the half aperture of cone h, h; is the unit vector of h, in the tangent space of h;:
1+ [b.
h h, h =
logn,
h.)
(13)
h
Figure 3(b) illustrates the two-step transformation described Eq. namely the scaling step and the rotation step.
3.3 ConE Model of Heterogeneous Hierarchies
In the previous section, we explained how we enforce cone containment constraint for hierarchical relations, however two challenges remain when simultancously modeling multiple heterogencous hierarchies: (1) Partial ordering: Suppose that there a hyponym relation between entities h; and h2, and different hyponym relation between entities h2 and h3. Then a naïve model would enforce that the cone of h1 contains the cone of h2 which contains the cone of hy, implying that a hyponym relation exists between h and h3, which not correct. (2) Expressive power: Cone containment
constraint, while ensuring hierarchical structure by geometric entailment, limits the set of possible rotation transformations and thus limits the model's expressive power.
To address these challenges we proceed as follows. Instead of enforcing cone containment constraint in the entire embedding space, ConE proposes novel technique to assign unique subspace for each hierarchical relation, i.e. we enforce cone containment constraint only in a subset of hyperbolic planes. Next we further elaborate on this idea.
In particular, for hierarchical relation r, we assign corresponding subspace of S, which is product space of a subset of hyperbolic planes. Then, we use restricted rotation in the subspace and rotation in the complement space. We train ConE to enforce cone containment constraint in the relation-specific subspace. The subspace can be represented by d-dimensional mask m. m, E (0, 1). and m = indicates that cone containment is enforced in the i-th hyperbolic plane. We then extend such notation to all relations where m 0 for non-hierarchical relations.

---

Dataset
Rentities
#relations
#training
#validation
#est
Examples of hierarchical relations
WNISRR
hyperyy,, has part subtype of subset of part of. is a location/contains, /musiccggenr/ppaeet genne Table 1: Datasets statistics. Note that FB15k-237 has very few such hierarchical relations.
40,943
11
86,385
3,034
3,134
DDB14
9,203
14
38,233
4,000
4,000
GO21
89.127
21
796,136
5,000
5,000
FB15k-237
14.541
237
272.115
17,535
20,466
Our design of leveraging both transformations model hierarchical relations crucial in that they capture different aspects of the relation. The use of restricted rotation along with cone containment constraint serves preserve partial ordering of hierarchical relation in its relation- -ppcciic subspace. But restricted rotation alone is insufficient: hierarchical relations also possess other properties such as composition and symmetry that cannot modeled by restricted rotation. Hence we augment with the rotation transformation to capture these properties, allowing composition of different hierarchical and oon-hierarchical relations through rotations in the complement space. We further provide theoretical and empirical results in Appendix A to support that both transformations are of great significance to the expressiveness of our model.
Putting it all together gives us the following distance scoring function (we use (U:)sE ,d. in the following to denote a d-dimensional vector v):
(h, r,)
d m (ds(f2(be, ra), (())ee
(14)
+(1 m) (dg(Si (h;, rs), t)))ie(1;--.a)))) + by b2
where the first term corresponds to the restricted rotation in celation-speciffee subspace, and the second term corresponds to the rotation in complementary space. A high score indicates that cone of entity h after Ellation-specffie transformation r is close to the cone of entity in terms of hyperbolic distance dg. Note that bn, b, are the learnt radius parameters of h, t which can be interpreted margins (15)
Subspace allocation. We assign equal dimensional subspaces for all hierarchical relations. We discuss and compare several strategies in assigning subspaces for hierarchical relations in Appendix including whether to use overlapping subspaces or orthogonal subspaces for different hierarchical relations, as well as the choice of dimensionality of subspaces. Overlapping subspaces (Appendix B allow the model to perform well and enable to scale to knowledge graphs with a large number of relations, since there exponentially many possible overlapping subspaces that can potentially correspond to different hierarchical relations.
3.4 ConE Loss Function
We use a loss function composed of two parts. The first part of the loss function aims to ensure that for a given head entity h and relation the distance the true tail entity t is smaller than to the negative tail entity t!:
r, = logo
Σ CET
T log o (-v(,,,,,t)
(15)
where (h, r,t) denotes a positive training example/triplet, and we generate negative samples (h, r, t') by substituting the tail with random entity in T E, a random set of entities in KG excluding (.
However, the distance loss La does not guarantee embeddings satisfying the cone containment constraint, since the distance between transformed head embedding and tail embedding can still be non-zero after training. Hence additionally introduce the angle loss (without loss of generality let r be a hyponym relation):
La (h, r,t) -m- =m-mmax(0. <n, t, o(h,)))
(16)
which directly encourages cone of contain cone of L in relation-specific subspaces, by constraining the angle between the cones. The final loss is then a weighted sum of the distance loss and the angle loss, where weight hyperparameter (We investigate the choice of w in Appendix B):
La + w Ln
(17)
7

---

WNT8RR
DDB14
GO21
Fraction of inferred descendant pairs among all true descendant pairs in the test set
Model
0%
50%
100%
0%
50%
100%
0%
50%
100%
Order 19
889
.739
.498
.731
.63
.533
642
.592
.534
Poincaré 10
810
.855
.508
.76
.32
.571
.525
.599
.516
HypCone 12
.799
.677
.504
.973
.823
.594
.554
.539
.519
RotatE
.601
.593
.582
.615
.500
.565
.566
.534
.526
RotH [16]
.611
.608
.611
.609
.566
.578
.596
.583
.564
ConE
.95
8011
.679
.981
.999
.818
.789
.744
.693
Improvement (%)
+1.9%
+9.6%
+11.1%
+0.5%
10.3%
+38,4%
+22.9%
+25.7%
+22.9%
Table 2: Ancestor-descendan prediction results in mAP (mean average precision). Best score in bold and second best underlined. We create different test sets that get harder as they contain more and more test cases (0%, 50%, 100%) of inferred descendant pairs.
4 Experiments
Given a KG containing many hierarchical and non-hierarchical relations, our experiments evaluate: (A) Performance of ConE hierarchical reasoning task of predicting if entity h is ancestor of entity h2- (B) Performance of ConE on generic KG completion tasks.
Datasets. We use four knowledge graph benchmarks (Table WordNet lexical knowledge graph (WNISRR [S] 17), drug knowledge graph (DDB14 14 [8), and a KG capturing common knowledge
(FB15K-237 (II). Furthermore, we also curated a new biomedical knowledge graph GO21, which models genes and the hierarchy of biological processes they participate in.
Model training. During training, we use Adam (32] as the optimizer and search hyperparameters including batch size, embedding dimension, learning rate, angle loss weight and dimension of subspace for each hierarchical relation. (Training details and standard deviations in Appendix G
We single trained model (without fine-tuning) for all evaluation tasks: On mncestrr-dsscenda relationship prediction, our scoring function for a pair (h, t) with hierarchical relation r is the angle loss in Eq. [6 where a lower score means h is more likely to be an ancestor of I. For KG completion task we use the scoring function (bt,, r, () in Eq. 14 to rank the triples.
4.1 Hierarchical Reasoning: Ancestor-descendant Prediction
Next we define ancestor-descendar relationship prediction task to test model's ability on hierarchical reasoning. Given two entities, the goal makes binary prediction if they have ancestor-descendant relationship:
Definition 2. Ancestor-descendaun relationship. Entity pair (h1, h2) is considered to have ancestor- descendant relationship if: there exists a from by to h2 that only contains one type of hyponym relation, or path from h2 to hy that only contains one type of hypernym relation.
Our evaluation setting is a generalization of the transitive closure prediction 199
12 which
is defined only over single hierarchy, but our knowledge graphs contain multiple hierarchies (hierarchical relations). More precisely: (1) When heterogeneous hierarchies coexist in the graph, we compute the transitive closure induced by each hierarchical relation separately. The test for each hierarchical relation is a random collection sampled from all transitive closures of that relation. (2) To increase the difficulty of the prediction task, our evaluation also considers inferred descendant pairs, which are only possible to be inferred when simultaneously considering hicrarchical and non-hierarchical relations in KG, due to missing links in KG. We call a descendant pair (u, v) an inferred descendant pair if their mncsstor-dsscendaan relationship can be inferred from the whole graph but not from the training set. For instance, (Tree, WinePalm) would be an inferred descendant pair if the subClass relation between Tree and PalmTree is missing in training set. We construct the inferred descendant pairs by taking the transitive closures of the entire graph, and exclude the transitive closures of relations in the training set. In our experiments, we consider three test settings: 0%, 50%, 100%, corresponding to the fraction of inferred descendant pairs among all true descendant pairs in the test set, and the setting with a higher fraction is harder.
2 The code of our paper is available at httttp://map.sttffoed.eedde////

---

Hyponar
Hypanymm
mal..00 tree
Nis 2030 paln pain Trea
paln
gum, Iree
www.ttyppunnooo free
-1
(a) Visualization of RotH's embedding
(b) Visualization of ConE's E embedding
Figure 4: The embeddings of RotH and ConE, trained on WNI8RR, projecting to one hyperbolic plane. We show the embedding of family of trees, and the arrows point from higher level entities to lower level entities, representing the hierarchical relation "Hyponym"" Different levels of entities and their corresponding cones ConE model (Figure 4(b) are marked with different colors. In ConE model, the embeddings of high-level entities (e.g., tree, palm tree) are close to the center of the hyperbolic plane, while embeddings of their descendant entities (e.g., wine palm, mallee) fall in their hyperbolic cones.
On each dataset, we extract 50k uncestor-descendant pairs. For each pair, we randomly replace the true descendant with random entity in the graph, resulting in a total of 100k pairs. Our way of selecting negative examples offsets the bias during learning that is prevalent in baseline models: the models tend to always give higher scores to pairs with a high-level node as ancestor, since high-level nodes usually have more descendants presented in training data. We replace the true descendant while keeping the true ancestor unchanged for the negative sample, and thus the model will not be able to "cheat" by taking advantage of the fore- fore-mentioned bias. For each model, we then use its scoring function to rank all the pairs. We use the standard mean average precision (mAP) to evaluate the performance on this binary classification task. We further show the AUROC results in Appendix E
Baselines.
We compare our method with sfate-of-the- art methods for hierarchical reasoning, including Order embeddings [9]. Poincaré embeddings [10] and Hyperbolic entailment cones [2].. Note that these methods can only handle a single hierarchical relation
time. So each baseline trains separate embedding for each hierarchical relation and then learns scoring function on the embedding of the two entities. ensure that the experiment controls the model size, we enforce that baselines, the sum of embedding dimensions of all relations is equal to the relation embedding dimension of ConE. We also perform comprehensive hyperparameter search for all baselines (Appendix GG. Although KG embedding models (RotatE (7 and RotH [16) cannot be directly applied to this task, we adapt them to perform this task by separately training an MLP to make binary classification on pair. taking the concatenation of the two entity embeddings as input. Note that ConE outperforms these KG completion methods without even requiring additional training.
Results. Table reports the meseetan prediction results of ConE and the baselines. We observe that the novel subspace transformation of ConE results in its superior performance this task. Our model consistently outperforms baseline methods on all three datasets. As we expected, KG embedding models cannot perform well on this task (in the range of 0.5 0.6 across all settings), since they do not explicitly model the partial ordering property of the hierarchical relations. In contrast, our visualization of ConE's embedding the cone containment constraint in modeling hierarchical relations, while RotH's embedding exhibit less bicrarchical structure. As a result, ConE simultancously captures the heterogencous relation modeling and partial ordering, combining the best of both worlds. Our improvement is more significant as the fraction of inferred descendant pairs increases. This shows that ConE not only embeds given hierarchical structure, but can also infer missing hierarchical links by modeling other non non-hierarchical relations at the same time. Thanks to the restricted rotation transformation and the of product spaces of hyperbolic planes, ConE can faithfully model the hierarchies without requiring all transitive closures in the training set. We further perform additional studies to explore reasons for the performance of each method ancestor-descendant prediction task in Appendix E
4 that ConE faithfully preserves
Lowest common ancestor prediction task. Moreover, we demonstrate flexibility and power of ConE using hicrarchical analysis task: lowest common ancestor (LCA) prediction, which requires both the

---

WNTERK
DDB1
GOZI
FIISK-2577
6=
(1.00.0.61,0 0.9....699....... 0.59.00.0)
(1.00. O.S.L........ 0.788.78.000 0.18)
(1.00.
(1.00.0.65, 0.65..65... 0.96.0.221
=
(1.00. 0.18, 0.36, (L06)
Model
MRK
HGT
H6/3
Hor 10
MRR
H61
H0R
Hor 10
MRK
H61
HG
Hor10
MRR
HoF
H85
H TO
.103
.337
.179
.294
.465
RotatE
476
.428
.429
-571
225
.144
.245
362
2003
.123
.234
.357
.338
.241
.375
.533
TuckER
14
.470
.443
.482
-26
.98
.137
.219
3344
.055
.136
.22
.342
.358
.266
.394
.544
HAKE
.96
.451
-513
.522
217
.146
.237
.361
.169
.104
.185
.295
.341
.243
.378
.535
214
.116
.39
.181
.335
.213
.518
RotH
[6
495
.449
514
-386
223
1152
.245
337
.51
0079
.171
.289
.344
2466
.380
.535
ConE
.96
.43
.515
.599
2311
.161
.252
.364
.211
.140
.237
.347
.345
.247
_381
.540
Table 3: Knowledge graph completion results, best out of dimension d E {100, 250, 500). Best score
in bold and second best underlined. x is a tuple denoting the 4 Krackhardt scores [29] that measure how hierarchical a graph is, higher scores mean more hierarchical. ConE achieves the best MRR and Hits@ results in hierarchical KGs.
ability model ancestor-descendant relationship and distinguish the lowest ancestor. Results show that ConE can precisely predict LCA, outperforming over 100% on Hits@3 and Hits@ 10 metrics compared to previous methods (See detailed results and analysis in Appendix FF.
4.2 Knowledge Graph Completion
We also experiment on knowledge graph completion task where missing links include hierarchical relations as well as non-hierarchical relations. We follow the standard evaluation setting (5).
Baselines. We compare ConE model state-of-the-e art models knowledge graph completion task, including TransE [5]. RotatE 77). TuckER [14] and HAKE 333 as well as MuRP [5] and RotH [6]. which both operate hyperbolic space.
Results. Table 3 reports the KG completion results. Over the first three hierarchical datasets considered, ConE achieves state-of-the- art results over many recent baselines, including the recently proposed hyperbolic approaches RotH and MuRP. We also notice that the margins on Hits 1 and Hits @ 3 scores are much larger than Hits @ 10, indicating that our model provides the most accurate predictions. We further use Krackhardt scores x to measure how hierarchical each graph is [29] The score consists of four metrics (fcomnectednesss hierarchy, efficiency, LUBedness), Appendix where if a graph is maximally hierarchical (i.e., a tree) then its Krackhardt score is (1, 1, 1), and higher score on four metrics indicate a more hierarchical structure. Notice that the Krackhardt scores of FB 15k-237 are approximately three times lower than those of WN 8RR, DDB 14 and GO21, indicating that FB15k-237 is indeed non-hierarchical. We can see that ConE model still performs better than other hierarchical KG embedding models (RotH and MuRP) on FB 15k-237 and comparable SOTA model (TuckER). Overall, this shows that ConE can scale to a large number of relations, and that it has competitive performance even in -on-hicrarchical knowledge graphs
We further analyze the performance of ConE in low-dimensional regimes in Appendix C Similar to previous studies, the hyperbolic-ssace--aale based ConE model performs much better than Euclidean KG embeddings in low dimensions (d 32). ConE performs similar previous hyperbolic KG embedding baselines in low dimensions, but outperforms them in high-dimensional regimes (Table Z).
Ablation study. We further compare the performance of our model with one that does not use cone restricted rotation for modeling hierarchical relations and one that does not use rotation for modeling hierarchical relations. Ablation results suggest that both transformations, i.e., cone restricted rotation and rotation, are critical in predicting missing hierarchical relations (Appendix (A.2). In particular, ablation results each individual hierarchical relation suggest that with cone restricted rotation, ConE can simultaneously model heterogeneous hierarchical relations effectively.
5 Conclusion
In this paper, we propose ConE, a hierarchical KG embedding method that models entities as hyper- bolic cones and uses different transformations between cones to simultaneously capture hierarchical and non-hierarchical relation patterns. We apply cone containment constraint to relation-specific subspaces to capture hierarchical information in heterogeneous knowledge graphs. ConE can simulta- neously perform knowledge graph completion task and hierarchical task, and achieves state-of-ther the-t-rrrt results on both tasks across three hierarchical knowledge graph datascts.
10

---

Acknowledgments and Disclosure of Funding
We gratefully acknowledge the support of DARPA under Nos. HR001 12190039 (TAMI), N660011922 1924033 (MCS); ARO under Nos. W911NF-16- --0422 (MURD, W91INF-16-1-0171 (DURIP); NSF under Nos. OAC-1835598 (CINES), OAC-1934578 (HDR). CCF-1918940 (Expe- ditions), IIS-2030477 (RAPID), NIH under No. R56LM013365 Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, Amazon, JPMorgan Chase, Docomo, Hitachi, Intel, JD.com, KDDI, NVIDIA, Dell, Toshiba, Visa, and UnitedHealth Group. Hongyu Ren supported by the Masason Foundation Fellowship and the Apple PhD Fellowship. Jure Leskovec a Chan Zuckerberg Biohub investigator.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
References
[1] A. Hogan, E. Blomqvist, M. Cochez, C. d' Amato, G. D. Melo, C. Gutierrez, S. Kirrane, J. E. L. Gayo, R. Navigli, S. Neumaier, al., "Knowledge graphs," ACM Computing Surveys (CSUR), 2021.
[2] Y. Hao, Y. Zhang, K. Liu, S. He, 2. Liu, H. Wu, and Zhao, "An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge," in ACL, 2017.
[3] C. Xiong, R. Power, and J. Callan, "Explicit semantic ranking for academic search via knowledge graph embedding," in www, 2017.
[4] M. Zitnik, M. Agrawal, and J. Leskovee, "Modeling polypharmacy side effects with graph
convolutional networks," Bioinformatics, 2018.
[5] A. Bordes, N. Usunier. A. Garcia-Duran J. Weston, and O. Yakhnenko, "Translating embeddings for modeling multi-relational data," in NeurIPS, 2013.
[6] Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu, "Learning entity and relation embeddings for knowledge graph completion," " in AAAI, 2015.
[7] 7. Sun, Z.-H. Deng, J.-Y. Nie, and J. Tang, "Rotate: Knowledge graph embedding by relational rotation complex space," in ICLR, 2019.
[8] G. A. Miller, "Wordnet: a lexical database for english," Communications of the ACM, 1995.
[9] A. Michael, A. B. Catherine, A. B. Judith, B. David, B. Heather, C. J Michael, P. D. Allan,
D. Kara, S. D. Selina, T. E. Janan, and et al, "Gene ontology: tool for the unification biology."
Nature genetics, 2000.
[10] M. Nickel and D. Kiela, "Poincaré embeddings for learning hierarchical representations." in NeurlPS, 2017.
[11] L. Vilnis, X. Li, S. Murty, and A. McCallum, "Probabilistic embedding of knowledge graphs with box lattice measures," " in ACL, 2018
[12] O.-E. Ganea, G. Bécigneul, and T. Hofmann, "Hyperbolic entailment cones for learning hierar- chical embeddings," in ICML, 2018.
[13] M. Nickel and Kiela, "Learning continuous hierarchies in the lorentz model of hyperbolic geometry," in ICML, 2018.
[14] I. Balažević, C. Allen, and T. M. Hospedales, "Tucker: Tensor factorization for knowledge graph completion," in EMNLP-IJCNLLI 2019.
[15] I. Balazevic, C. S. Allen, and T. Hospedales, "Multi-relational poincaré graph embeddings." in NeuriPS, 2019.
[16] I. Chami, A. Wolf, D.-C. Juan, F. Sala, S. Ravi, and C. Ré, "Low-dimensional hyperbolic knowledge graph embeddings," in ACL, 2020.
[17] T. Dettmers, P. Minervini, P. Stenetorp, and Riedel, "Convolutional 2d knowledge graph embeddings," in AAAI, 2018.
[18] H. Wang, H. Ren, and J. Leskovec, "Entity context and relational paths for knowledge graph completion," arXiv preprint arXiv:2002.06757. 2020.

---

[19] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, "Order-embeddiig of images and language," in ICLR, 2016.
[20] A. Lai and J. Hockenmaier, "Learning predict denotational probabilities for modeling entail- ment," in EACL, 2017.
[21] X. Li, L. Vilnis, D. Zhang, M. Boratko, and A. McCallum, "Smoothing the geometry of probabilistic box embeddings." in ICLR, 2018.
[22] S. S. Dasgupta, M. Boratko, D. Zhang, L. Vilnis, X. L. Li, and A. McCallum, "Improving local identifiability in probabilistic box embeddings." in NeurIPS, 2020.
[23] S. S. Dasgupta, X. Li, M. Boratko, D. Zhang, and A. McCallum, "Box-to-box transformation for modeling joint hierarchies," 2021.
[24] B. Yang, W.-t. Yih, X. He, J. Gao, and L. Deng, "Embedding entities and relations for learning and inference in knowledge bases," in ICLR, 2015.
[25] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard, "Complex embeddings for simple link prediction," in ICML, 2016.
[26] I. Chami, Z. Ying. C. Ré, and J. Leskovec, "Hyperbolic graph convolutional neural networks," in NeurIPS, 2019.
[27] O.-E. Ganea, G. Bécigneul, and T. Hofmann, "Hyperbolic neural networks," in NeurIPS, 2018. [28] R. Speer, J. Chin, and C. Havasi, "Conceptnet 5.5: An open multilingual graph of general knowledge," AAAI, 2017.
[29] D. Krackhardt, "Graph theoretical dimensions of informal organizations." Computational organization theory, 1994.
[30] A. Gu, Sala, B. Gunel, and C. Ré, "Learning mixed-curvature representations in product spaces," in ICLR, 2018.
[31] K. Toutanova and D. Chen, "Observed versus latent features for knowledge base and text inference," " in Proceedings of the 3rd workshop on continuous vector space models and their compositionality, 2015.
[32] D. Kingma and J. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015.
[33] Z. Zhang, J. Cai, Y. Zhang, and Wang, "Learning hierarchy-aware knowledge graph embed- dings for link prediction," in AAAI, 2020.
[34] J. Piñero, J. M. Ramirez-Anguita. J. Satich-Pitarch, F. Ronzano, E. Centeno, F. Sanz, and L. I. Furlong, "The DisGeNET knowledge platform for disease genomics: 2019 update." Nucleic Acids Research. 2019.
[35] A. P. Davis, C. J. Grondin, R. J. Johnson, D. Sciaky, J. Wiegers, T. C. Wiegers, and C. J. Mattingly, "Comparative toxicogenomics database (ctd): update 2021," Nucleic acids research, 2021.
[36] O. Bodenreider, "The Unified Medical Language System (UMLS): integrating biomedical terminology," Nucleic Acids Research, 2004.
[37] D. S. Wishart, C. Knox, A. C. Guo, D. Cheng, S. Shrivastava, D. Tzur, B. Gautam, and M. Hassanali, "DrugBank: knowledgebase for drugs. drug actions and drug targets," Nucleic Acids Research, 2007.
[38] Y. D. Feunang, R. Fisner, C. Knox, L. Chepelev, J. Hastings, G. Owen, E. Fahy, C. Stein- beck, S. Subramanian, F. Bolton, al., "Classyfir:: automated chemical classification with comprchensive, computable taxonomy," Journal of cheminformatics, 2016.
[39] E. L. Carolyn, "Medical subject headings (mesh)," Bulletin of the Medical Library Association, 2000.
[40] C. Ruiz, M. Zitnik, and J. Leskovec, "Discovery of disease treatment mechanisms through the multiscale interactome," bioRxiv, 2020.
12


Reviews:
[{"review_id": "kKIADbQJsdE", "reviewer": "Reviewer_s4ku", "summary": "Methods for embedding hypergraph-based knowledge bases, where different edge types encode different relation types, have a long history in machine learning. Alternately, there has been a more recent interest in geometric embedding approaches for knowledge bases, which associate more complex geometric objects, such as hyperbolic cones, with knowledge base entities, due to the favorable geometric properties of hyperbolic space for modeling hierarchies. However, the latter approaches have usually been limited to representing only single relation types, due to representing the relation of interest with some geometric property of the embedding space such as enclosure between entity cones. This work attempts to bridge the gap between the two approaches by modeling relations as transformations between hyperbolic cones, specifically containment in different subspaces. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "UPDATE: The authors have addressed my concerns about interactions between the transitive and non-transitive parts of the model and I am raising my score accordingly. \n\n\n**Originality/Quality**\n\nAs the paper points out, there have been a couple efforts previously to produce multi-relational hyperbolic cone embeddings. However, they argue that approaches based on unrestricted Möbius matrix-vector product and addition (MuRP) cannot capture transitivity. In response, they introduce a “restricted rotation” operation that is specified when relations are known to be transitive, which scales the rotated cone to still lie within the parent cone. Transitive relations are modeled as restricted rotation in one subset of dimensions, while non-transitive ones are modeled as general rotations in another subset. \n\nIt is unclear how this differs from training two different models and sets of embeddings, one which corresponds to the transitive relations and another which corresponds to the non-transitive relations. Is this correct? This makes the claim of using a “single” model a bit vacuous if these different relation types never interact. The results in experiments are good and the experiments are extensive, but the technical contribution seems fairly minimal.\n\n\n**Clarity**\n\nThe paper is well written but could be better organized, it is hard to follow exactly how the proposed model differs from previous work, and the focus on one relation at a time in the exposition makes it often hard to initially understand how the proposed model differs from simply training a separate model for each relation, until the “subspace allocation” section of 3.3. \n\n**Significance**\n\nBoth knowledge graph embedding and nonstandard embedding geometries are popular areas of inquiry, and doing principled multi relational modeling in hyperbolic space is also interesting. The use of the restricted rotation is a good tool for practitioners to have in their arsenal, but the synthesis between the transitive and non-transitive relations is lacking.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Experiments seem adequate to address limitations, aside from the questions about interactions between transitive and non-transitive parts raised above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "a65_DrjAWmC", "reviewer": "Reviewer_G45q", "summary": "The authors propose ConE (Cone Embedding) to simultaneously model multiple hierarchical and non-hierarchical relations in a knowledge graph. ConE represents entities as hyperbolic cones and models relations as transformations between the cones. Experiments demonstrate that ConE outperforms many baselines on standard knowledge graph benchmarks. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe idea of modeling multiple hierarchical and non-hierarchical relations simultaneously in a knowledge graph is interesting.\n\nQuality: \nMy concerns are as follows.\n1. The authors may want to explain why ConE performs on par with RotH [1] on WN18RR and FB15k-237 (see Table 1).\n2. Some related works are missing. The authors may want to compare ConE with more works related to hierarchy modeling [2,3].\n3. The distance scoring function in Equation (14) may not make sense. The authors claim that the rotation transformation $f_1$ cannot explictly model hierarchical relations, while the authors compute $f_1$ in Equation (14) for the hierarchical relations in some cases, e.g. $m_1=1$ and $m_i=0,i=2,3,\\dots,d$.\n\nClarity:\nThe authors may want to clarify the implementation of ConE. The restricted rotation transformation in Equation (12) and the angle loss in Equation (16) are only defined for the hyponym relation. However, the information about hyponym or hypernym relation is usually unavailable in knowledge graphs, e.g. FB15k-237. The authors may want to illustrate how to implement the restricted rotation transformation and the angle loss in practice.\n\nSignificance: \nThe work aims to model hierarchical structures in knowledge graphs, which is an important topic.\n\n[1] I. Chami, A. Wolf, D.-C. Juan, F. Sala, S. Ravi, and C. Ré, “Low-dimensional hyperbolic knowledge graph embeddings,” ACL, 2020. \n\n[2] Z. Zhang, J. Cai, Y. Zhang, and J. Wang, “Learning hierarchy-aware knowledge graph embeddings for link prediction,” in Proc. AAAI, 2020, pp. 3065–3072\n\n[3] Z. Zhang, F. Zhuang, M. Qu, F. Lin, and Q. He, “Knowledge graph embedding with hierarchical relation structure,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2018, pp. 3198–3207.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11 hours"}, "extra_text": {}}, {"review_id": "Mp8WdqZdD2", "reviewer": "Reviewer_odL8", "summary": "This paper proposes a knowledge graph (KG) embedding method that can simultaneously model hierarchical (e.g., is-a) and non-hierarchical (e.g., friend-of) relations. The authors propose to embed entities into hyperbolic space. Inspired by Hyperbolic Entailment Cones (Ganea et al., 2018), they apply the cone containment constraint to relation-specific subspaces to deal with hierarchical relations. Inspired by RotatE (Sun et al., 2019) and RotH (Chami et al., 2020), they use hyperbolic cone rotations from the head entity to the tail entity to model non-hierarchical relations.\n\nExperiments are conducted on two tasks: KG completion and hierarchical reasoning. Results show that the proposed method can outperform both Euclidean and hyperbolic KG embedding baselines. Results in low dimensions, ablation studies, and results on a biological dataset are further shown in the supplementary material.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written with solid and comprehensive evaluations, reflecting authors' massive effort. \n\nAlthough some ideas are directly borrowed from existing studies (e.g., Hyperbolic Entailment Cones and RotH), the general motivation of simultaneously modeling hierarchical and non-hierarchical relations is intuitive. Also, the idea of modeling heterogeneous hierarchical relations is well-motivated. Although hypernymy heterogeneity has been studied in taxonomy-related studies [1], using relation-specific subspaces to model it in the hyperbolic space is novel from my perspective.\n\nSome questions and concerns:\n- Statistical significance tests are missing in the experiments. For example, in Table 2, the gap between your methods and the best baseline is quite subtle when there are 0% inferred descendant pairs.\n\n- Although the quantitative evaluation is comprehensive, it would be better to see some case studies and visualization results. For example, does your embedding result really preserve the cone containment constraint? For symmetric relations, does the rotation preserve the symmetry?\n\n[1] Expanding Taxonomies with Implicit Edge Semantics. WWW 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9uJQb0S-yj", "reviewer": "Reviewer_LaLE", "summary": "This work proposes ConE, a knowledge graph embedding model which embeds entities into hyperbolic cones and represents relations as transformations between the cones. Non-hierarchical relations are modelled using rotations and hierarchical relations using restricted rotations, which impose the cone containment constraint. The proposed method is evaluated on the KG completion and ancestor-descendant prediction tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed model is a relatively novel approach for modelling both hierarchical and non-hierarchical relations in knowledge graphs, based on hyperbolic cones by Ganea et al., 2018.\n- The ability to do ancestor-descendant prediction is an important improvement compared to standard KG completion models.\n- The performance on KG completion is comparable to existing state-of-the-art models.\n\nWeaknesses\n- The proposed method requires a priori knowledge of whether a relation is hierarchical or not, and not just that, but also knowing the direction of hierarchy (i.e. head->tail or tail->head). None of the previous methods (e.g. MuRP and RotH) require this knowledge, which gives ConE unfair advantage over those. \n- Despite having access to extra information, the proposed model does not outperform RotH/AttH on FB15k-237 and WN18RR, which are standard KG completion datasets.\n\n\nDetailed questions/comments:\n1. In the abstract, you claim that \"curent KG embeddings...fail to model multiple heterogeneous hierarchies that exist in a single KG.\". However, this is factually incorrect, since there has been several works for modelling hiearchical relations in knowledge graphs, such as MuRP and RotH/AttH.\n2. Again, same applies to the introduction, i.e. \"These methods can only be applied to graphs with a single hierarchical relation type, and are thus not suitable to real-world knowledge graphs with multiple hierarchical relations.\" - both MuRP and RotH/AttH have been specifically designed to model multiple hierarchies. Later in that some paragraph, you contradict yourself by mentioning MuRP and RotH.\n3. Have you tried learning the space curvature? Chami et al. show that fixing the curvature has a negative impact on performance.\n4. Your method assumes a priori knowledge on whether a relation is hierarchical or not, but in most real world KGs, this information isn't available. What would happen for KGs where the number of relations is really large and checking whether a relation is hierarchical or not is infeasible?\n5. Do you have any intuition on why the multi-relational RotH performs much worse than the single-relational Poincaré embedding (Nickel & Kiela, 2017) on the ancestor-descendant prediction task ?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a knowledge graph embedding method called ConE, which can simultaneously model both hierarchical and non-hierarchical relations by embedding entities into hyperbolic cones with relations as transformations between cones. The reviewers agree that this is a strong paper, and the authors did an excellent job in their rebuttal in addressing any remaining concerns.
Decision: Accept (Poster)