Box-To-Box Transformation for Modeling Joint Hierarchies
Authors: Shib Sankar Dasgupta
Xiang Li
Michael Boratko
Dongxu Zhang
Andrew McCallum
Venue:  (2021)
Keywords: Box embeddings
Representation Learning
Joint Hierarchy
transitive relations
knowledge graph embedding
relational learning.

Abstract: Learning representations of entities and relations in knowledge graphs is an active area of research, with much emphasis placed on choosing the appropriate geometry to capture tree-like structures. Box embeddings (Vilnis et al., 2018; Li et al., 2019; Dasgupta et al., 2020), which represent concepts as n-dimensional hyperrectangles, are capable of embedding trees by training on a subset of the transitive closure. In Patel et al. (2020), the authors demonstrate that only the transitive reduction is required, and further extend box embeddings to capture joint hierarchies by augmenting the graph with new nodes.  While it is possible to represent joint hierarchies with this method, the parameters for each hierarchy are decoupled, making generalization between hierarchies infeasible. In this work, we introduce a learned box-to-box transformation which respects the geometric structure of the box embeddings.  We demonstrate that this not only improves the capability of modeling cross-hierarchy compositional edges but is also capable of generalizing from a subset of the transitive reduction.

Under review as a conference paper at ICLR 2021
Box-To-Box TRANSFORMATIONS FOR MODELING JOINT HIERARCHIES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning representations of entities and relations in knowledge graphs is active area of research, with much emphasis placed on choosing the appropriate geometry capture tree-like structures. Box embeddings (Vilnis et al. 2018 Li et al. 2019 Dasgupta et al. 2020), which represent concepts as n--dimensional hyperrectangles. are capable of embedding trees when training on a subset of the transitive closure. In Patel et al. (2020), the authors demonstrate that only the transitive reduction is required, and further extend box embeddings to capture joint hierarchies by augmenting the graph with new nodes. While it is possible to represent joint hierarchies with this method, the parameters for each hierarchy are decoupled, making generalization between hierarchies infeasible. In this work, we introduce a learned box-to-box transformation which respects the geometric structure box embeddings. We demonstrate that this not only improves the capability of modeling cross- pross-hierarchy compositional edges, but is also capable of generalizing from subset of the transitive reduction.
INTRODUCTION
Representation learning for hierarchical relations is crucial in natural language processing because of the hierarchical nature of common knowledge, for example, <Bird IsA Animal> (Athiwaratkun & Wilson 2018 Vendrov et al. 2016; Vilnis et al. 2018 Nickel & Kiela| (017). The ISA relation represents meaningful hierarchical relationships between concepts and plays an essential role in generalization for other relations, such as the generalization of <organ PARTOF person>> based on <eye PARTOF of person>, and <organ IsA eye>. The fundamental nature of the IsA relation means that it is inherently involved in a large amount of compositional human reasoning involving other relations.
Modeling hierarchies is essentially the problem of modeling a poset, partially ordered set. The task of partial order completion, a general term to describe tasks which require learning a transitive relation, introduced in (Vendrov et al. 2016). The authors also introduce model based on the reverse product order on IR", which essentially models concepts as infinite cones. Region-based representations have been effective in representing hierarchical data, as containment between regions is naturally transitive. Vilnis et al. (2018) introduced axis- aligned hyperrectangles (or boxes) that are provably more flexible than cones, and demonstrated state-of-the-arr performance multiple tasks.
Thus far, not as much effort has been put into modeling joint hierarchies. Patel et al. (2020) proposed to simultaneously model IsA and HASPART hierarchies from Wordnet (Miller 1995). To do so, however, they effectively augmented the graph by duplicating the nodes to create a single massive hierarchy. Their model assigns two boxes BISA and BHASPART for each node n, which are unrelated, and therefore misses out on large amount of semantic relatedness between ISA and HASPART.
In this paper we propose a box-to-box transformation which translates and dilates box representations between hierarchies. Our proposed model shares information between the IsA and HASPART hierarchies via this transformation as well as cross-hierarchy containment training objectives. We compare FOX-TRANSFOR
MODEL with multiple strong baselines under different settings. We substantially outperform the prior Two-Bon MODEL while training with only the transitive reduction of both hierarchies and predicting inferred composition edges. As mentioned above, our model's shared learned features should allow for more generalization, and we test this by training on a subset of the transitive reduction, where we find able to outperform strong baselines. Finally,

---

Under review as a conference paper at ICLR 2021
perform detailed analysis of the model's capacity to predict compositional edges and transitive closure edges, both from an overfitting and generalization standpoint, identifying subsets where further improvement is needed.
2 RELATED WORK
Recent advances in representing one single hierarchy mainly fall in two categories: 1) representing hierarchies in non- -uc-lucidiaan space (eg. hyperbolic space, due to the curvature's ' inductive bias to model tree-like structures) 2) using region-based representations instead of vectors for each node in the hierarchy (Erk) 2009). Hyperbolic space has been shown be efficient in representing hierarchical relations, but also encounters difficulties in training (Nickel & Kiela 2017
Ganea et al. 2018b
Chamberlain et al. 2017).
Categorization models in psychology often represent concept as a region (Nosotsky 1986 Smith jtt al. 1988 Hampton 1991). |iinis & McCallum (2015) and Athiwaratkun & Wilson (2018) 8) use Gaussian distributions to embed each word in the corpus, the latter of which uses thresholded divergences which amount to region representations. Vendrov et al. (2016) and Lai & Hockenmaier
(2017) make use of the reverse product order on R7, which effectively results in cone representations. Vilnis et al. 2018)) further extend this cone representation to axis-aligned hyperrectangles (or boxes), and demonstrate kate-of-the-ar performance
modeling hierarchies. Various training improvement methods for box embeddings have been proposed (Li et al.) 2019 Dasgupta et al. 20200, the most recent of which is termed GumbelBox after it's use of a latent noise model where box parameters represented via Gumbel distributions.
also used for tasks which do not require modeling hierarchy. In Vilnis et al. (2018). the authors also model conditional probability distributions using box embeddings. Abboud al. (020)) and Ren et al. (2020) take different approach, using boxes for their capacity to contain many vectors to provide slack in the loss function when modeling knowledge base triples or representing logical queries, respectively. [Ren et al.) (2020) also made use of an action on boxes similar to ours, involving translation and dilation, however our work differs in both the task (i.e. representing logical queries vs. joint hierarchies) and approach, as their model represents entities using vectors and loss function based a box-to-vector distance. The inductive bias of hyperbolic space is also exploited to model multiple relations, Ganea et al. (2018a) learn hyperbolic transformations for multiple relations using Poincare embeddings, and show model improvement in low computational resource settings. Patel et al. (2020), which our work is most similar to, represent joint hierarchies using box embeddings. However, they represent each concept with two boxes ignoring the internal semantics of the concepts.
Region representations
Modeling joint hierarchies shares some similarities with knowledge base completion, however the goals of the two settings are different. When modeling joint hierarchies you are attempting to learn simultaneous transitive relations, and potentially learn relevant compositional edges involving these relations. For knowledge base completion, on the other hand, you may be learning many different relations, and primarily seek to recover edges which were removed rather than inferring new compositional edges. Still, the models which perform knowledge base completion can be applied to this task, as the data can be viewed as knowledge base triples with only 2 relations. There have been multiple works that aim build better knowledge representation (Bordes et al. [003]] Trouiloon let al. 2016 Sun et al. 2019 Balazevic et al. 2019a). Most relevant, Chami et al. (2020); Balazevic et al. (2019b) recently proposed KG embedding methods which embeds entities in the Poincaré ball model of hyperbolic space. These models are intended to capture relational patterns present in multi-relational graphs, with particular emphasis hierarchical relations.
3 BACKGROUND
3.1 BOX LATTICE MODEL.
Introduced in Vilnis al. (2018). a box lattice model (or box model) a geometric embedding which captures partial orders and lattice structure using n-dimensional hyperrectangles. Formally, we define the set of boxes R. as
B(R") [ ... a
2

---

Under review as a conference paper at ICLR 2021
where 24, x² R. and we represent all degenerate boxes where I; xx' with 0. A box model for set S is function Box : S + B(R") which captures some desirable properties of the set S. As the name implies, the box lattice model is particularly suited to representing partial orders and lattice structures.
Definition 1 (Poset). A partially ordered set, poset, is a set P along with relation 3 such that, for each a, b, c E P, we have
1. a 3 a (reflexivity)
2. if a N and b 3 a then a = (antisymmetry)
3. if a N and b N c then a N c (transitivity)
Definition 2 (Lattice). A lattice is poset where each pair of elements have unique upper bound called the join, denoted by A, and a unique lower bound called the meet, denoted by V.
The authors note that there are natural geometric operations which form a lattice structure on B:
Box(x) A Box(y) := II [max( v.), min(x', yy')],
(2)
Box(x) Box(y)
II [min( (Ze, ye), max(x"; y')],
(3)
In other words, the meet of two boxes is the smallest containing box, and the join is the intersection, or 0 the boxes are disjoint. These geometric operations map very neatly to hierarchies, where meet of two nodes is their closest common ancestor and the join is the closest common descendent (or 9 no such node exists). The ability of this model capture lattice structure using geometric operations makes it a natural choice to embed hicrarchies.
3.2 PROBABILISTIC Box MODEL TRAINING
In Vilnis et al. (2018), the authors also introduced probabilistic interpretation box embeddings and learning method which was improved upon in Li et al. (2019) and Dasgupta al. (2020). By using probability measure μ on R2 (or by constraining the space to [0, 1 one can calculate box volumes as y (Box( (X)). The pullback of this measure yields a probability measure
S, and thus the box model can be imbued with valid probabilistic semantics. In particular, since the box space closed under calculate joint probabilities by computing P(X,Y) (BBox A Box(Y)) and similarly compute conditional probabilities as
intersection,
P(X |Y) m(Box(X) A Box(Y))
u(Box(Y))
The conversion from a poset or lattice structure to probabilistic semantics is accomplished by assigning conditional probabilities, namely 3 b if and only if P(b a) 1. We note that the properties required of the relation = follow as natural consequence of the axioms for conditional probability. Apart from providing rigor and interpretability, the calibrated probabilistic semantics also inform and facilitate the training procedure for box embeddings, which is accomplished via gradient descent using KL-divergence with respect to the aforementioned probability distribution as loss function.
As one might expect, care must be taken to handle the case when boxes are disjoint, as there is no gradient. In (Vilnis et al.] (2018) the authors made use of the lattice structure derive lower bound on the probability, and (Li et al. 2019) introduced an approximation to Gaussian convolution over the boxes which similarly handled the case of disjoint boxes. (Dasgupta et al. (0200) improves this further by taking a random process perspective, ensembling over an entire family of box models. The endpoints of boxes are represented using Gumbel distributions, that
GumbelBox(X)) = Il X,, X"), X, MaxGumbel(44,))
X xi~Mii MinGumbel((pp ,8), (5)
where μ., 9 are the location and scale parameters of the Gumbel distribution respectively. The MaxGumbel distribution is given by
1
f(x: pt, B)
exp( z-4
(6)
B
3

---

Under review as a conference paper at ICLR 2021
Animal
Auimal
HasPart
Group
Bird
1sA
HasPPari
Dove
Wing
Wing
Composition Edges
HasPart Appendde
Forewing
Dove
Covey
Appendage
Figure 1: An overview of Box- TRANSFORM MODEL on joint ISA and HASPART hierarchies. Composition edges are created following certain rules and it should be correctly inferred for well- trained model. The ISA Wing box is transformed into HASPART Wing box representing concepts that has wings, and Bird subset of Same follows for Appendage, and the monotonicity in the ISA space is preserved in HASPART space.
and the MinGumbel distribution given by negating x μ. The Gumbel distribution was chosen due to it's min/max stability, making the set of Gumbel boxes closed under intersection, i.e. the intersection of two Gumbel boxes is another Gumbel box. We denote the space of all such boxes as g. The expected volume of a Gumbel box can be efficiently calculated analytically, and in Dasgupta al. (2020) the authors use this expected volume to calculate the conditional probabilities mentioned in equation equation 4 This training method leads to improved performance on number of tasks, and is particularly beneficial when embedding trees, thus we will use this Gumbel box approach in our setting.
3.3 MODELING JOINT HIERARCHIES
Many existing methods have been proposed for modeling a single hierarchy, however entities are often simultancously part of multiple hierarchies, for example hypernymy (i.e. IsA) ) and meronomy (i.e. HASPART). ). Furthermore, useful information can be shared across inferred compositional edges between the two hierarchies. For example, as shown in T based on <Bird, HASPART ,Wing> and <Dove,IsA, BBrrd>, we can infer <Dove,HASPArE Wing>. Due to the compositional nature of these relations, we can infer not only the per-relation transitive closure edges but also the compositional edges, i.e <Dve, HASPART, Wing>>
Formally, for two hierarchical relations rı and r2, composition edges can be formulated following certain rules. In figure I the rules are designed as follows: for <Hccad.AsPARRT Tail>>, < 21, IsA Head>> represent the sub-class of Head, and <Til, ISA 22 >> is the super-class Tail. Composition edges can be generated < I1 HASPART .T2 >, < Ij HAASART .Tai>> < Head .HAPPART ,22 >. These compositional edges were identified in Patel et al. 2020). where it was observed that a model which effectively captures both hierarchies should make the correct prediction not only over the transitive closure of each individual relation but also on these compositional edges.
METHODS
4.1 BOX-TO-BOX TRANSFORMATION
As mentioned previously, our goal is to not only capture intra-relation transitivity, but also require the model to capture cross- hiss-hhirrrrchy compositional edges; that is, for a set S with two partial orders 31. Z2. want model capable of learning (a 21 b) A (b 32 c) = 32 and (a 32 6) (b 31 c) + a Z2 C. Furthermore, we hope to do so without including these compositional edges in our training data, in fact will remove parts of these implications in the to recover them.
data, with the expectation that the embedding parameters capture relevant structure which allows us
As shown in Dasgupta et al. (2020), Gumbel boxes are able to model hierarchies, we would like to benefit from this capability, particularly for modeling the ISA hierarchy, and thus seek to learn
4

---

Under review as a conference paper at ICLR 2021
function S G, where
Elp(fi(o) n fi()) Elu(fi(a))
a 1
For given Gumbel box,
((x) - II Xx, X; ~ MaxGumbellus; 8), X MinGumbel(j + B),
(8)
where the free parameters are br and Aj. To simultaneously model second relation, train function şp G + G such that
(l)( n fi Elu(p(fi(a)))
b
=1.
For notational simplicity, we abbreviate f0
We choose the transformation y operate the "min" coordinate of Gumbel box and the "side-lengths", that is, we transform a given Gumbel box
f(x) II [x
[ x'), X₃ MaxGumbbb(((..)) 9), X ~ MinGumbel((pp A),
(10)
to
d (GumbelBox((M)) X)) ())= II [X, y"] ==1
(11)
where
Y MaxGumbel(0A444 b,, 9), and softplus(0'A, + b)),
(12)
y MinGumbel(0,ju.
(13)
and the 0,, 0', b,, b are learned parameters. This effectively translates and dilates the location parameters of the Gumbel distributions which represent the "corners" of given Gumbel box. We call this model the FOX-TRANSFORM MODEL.
The softplus function is used here as a way to ensure the max coordinate remains larger than the min, and it also provides simple overflow protection for the expected box volume, might happen with side-lengths larger than one high dimensions. While mathematically simple, this transformation allows for parameter sharing between the embedding of concept with respect 3 and with respect to 52. Importantly, the transformation is capable of capturing both a global translation and dilation as well as a scaled transformation of the existing learned representation, allowing the absolute position in space (which, for previous box embedding models, was irrelevant) to potentially capture relevant features of the entities.
The lack of transformation on fi (b) is not oversight. Using figure I as example, if we consider the Bird box as representative of "all things which are birds", and the HASPART Wing box the representative of "all thing which have wings". then encouraging containment of the Bird box inside the HASPART Wing box is quite natural. This conceptual motivation is precisely captured by the lack of transformation in section 3.2 and is also the method employed by Patel et al. (2020), where this cross-hierarchy containment objective soley responsible for any flow of information between hierarchies in the Two-Box MODEL..
Remark 1.
fi (b). This also coincides with the probabilistic semantics discussed
4.2 CONNECTION TO Two-Bon BBx MODEL
There are two main differences between our model and the model introduced (Patel et al. 2020 which, for reasons which will become clear, we call the Two-Box MODEL First, the Two- B Box MODEL preceeded the Gumbel box model, and instead uses the Soft box model from (Li et al. 2019). To ensure that the benefits from model not conflated with the improvements from using Gumbel boxes we also train a model using the method from (Patel et al. 2020) which makes of Gumbel boxes.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Details of the hypernymy and meronymy hierarchies and the composite edges formed by composition of them.
Transitive Reduction
Transitive Closure
Validation (pos/neg)
Test (pos/neg)
Hypernym 84,363
661,127
28,838/288.
28,833/288.388 5,164/51,64D
Meronym
9,678
30,333
51,64/51,640
Composite Edge
94,807/948,070
94,806/948,070
Second, both models use different boxes to represent different relations, however the model from (Patel et al., 2020) allows both boxes to have free parameters, relying on containment between boxes representing different relations to pass information. Under the framework we have currently presented, this would be equívalent to learning two functions, f1 and f2, both of which have separate parameters for the min and side length of the boxes for each entity. While such model has significant representational capacity, we would expect that it would suffer greatly from a lack of generalization. We evaluate this hypothesis by creating a second test, discussed section 5.4 which removes edges from the transitive reduction of the training data.
5 EXPERIMENTS
5.1 DATASET
We demonstrate the efficacy of BOX-TRANSFORM MODEL by using the joint hierarchy that has been created by Patel et al. (2020) from WordNet (Miller 1995). In this dataset, hypernymy (ISA ) and meronymy (HASPART are two hierarchical relations of WordNet over noun sysnets, which are 82, 114 in total. Individually, the hypernymy part of the hierarchy contains 82, 114 nodes (i.e., all the synsets) with 84, 363 edges in its transitive reduction and the meronymy portion has 11, 235 synsets (out of 82, 114 synsets) with 9, 678 edges in its transitive reduction.
Joint Hierarchy In order to evaluate the performance on the joint hierarchy, Patel et al. (2020) created composition edges using the inter-relational semantics between hypernymy and meronymy. In particular they use the following composition rules:
IsA ISA IsA HASPART ISA ISA ... ISA
HASPART
(14)
O or or 2 times
Dur or 2 times
To illustrate from Figure I <Dove ISA Bird> A <Bird HASPART Wing> A < Wing IsA Appendage>> implies that <birds HASPART appendage>, total, 189, 613 composition edges are generated by the method described above for evaluation of the model on the joint hierarchy task. For each test/validation edge, a fixed set of negative samples of size 10 was generated by corrupting the head and tail 5 times each.
We provide the overall statisties for the dataset in the Table I training dataset which further removes part of the transitive reduction to evaluate the models on their generalization capability (refer to Section 5.4 & 5.5). The dataset used for those section has different statistics and they are reported in the respective sections.
We have also created second
5.2 BASELINE MODELS AND TRAINING DETAILS
We compare BOX-TRANSFORM MODEL against geometric embedding methods as well as knowledge base completion methods. We give brief description for each baseline below.
1. Two-Box MODEL As mentioned in
4.2 Patel al. (2020) extends the idea of Box embeddings |Vinnis et al. 2018 Li et al. 2019) to model joint hierarchies by defining two boxes per node, one for each relation.
2. Order Embeddings: Vendrov al. (2066) treats each concept as axis parallel cones in positive orthant. We considered two different cone parameters for each entity following the Two-Box MODEL. (Patel et al. (2020))

---

Under review as a conference paper at ICLR 2021
3. Poincaré Embeddings: (Nickel & Kiela 2017) & Hyperbolic Entailment Cones (Ganea et al., 2018b): stru-structure data are best captured in hyperbolic space (Chamberlain et al. 2017). Thus in Nickel & Kiela (2017). the authors learn embedding n-dimensional Poincaré ball. For similar reasons, Ganea et al. (2018b) uses the hyperholic space however they extend the hyperbolic point embeddings entailment cones. Again, for these models, two separate parameters are considered for each entity.
4. TransE and RotatE (Bordes et al., 2013, Sun et al. 2019): This task can be posed as knowledge base completion for KB with only two relations. Thus we evaluate TransE and RotatE which are simple yet effective methods for knowledge base embeddings, which achieve state-of- sate-of-the-art the--rr for many knowledge base embedding tasks. Unlike the two box model (Patel et al.) 2020) or the other baselines, these methods have shared representation for each entity, and thus they are expected to generalise better on missing edges.
5. Hyperbolic KG Embeddings (Balazevic et al. 2019b Chami et al., 2020): We also com- pared our method against recently proposed KG embedding methods based on hyperbolic embeddings to model hierarchical structures present in KGs. The Multi- Relational Poincaré model (MuRP) (Balazevic al., 2019b) learns relation-specific transforms of the entities that are embedded in hyperbolic space. The RoTH (Chami et al. 2020) parameterize the relation specific transformations as hyperbolic rotation, where as the AttH (Chami al.. (2000) combines hyperbolic reflection and rotation using attention. We provide more training related details in Appendix A.
5.3 COMPOSITION EDGES FROM TRANSITIVE REDUCTION
In order to demonstrate the ability of the model to capture partially ordered (tree-like) data most embedding methods (Ganea et al., 2018b; Nickel & Kiela 2017 Patel et al. 20200) train their model on the transitive reduction and predict on the transitive closure. For an evaluation on modeling the joint hierarchy, therefore, it is natural to train the models only the transitive reduction of hypernymy and meronymy and evaluate on the composition edges, as done in PPtel et af. (2020). We report the Fl score (with 1:10 negatives) for those edges table [ The threshold used for the classification is determined by maximizing the Fl score on the validation set.
Table 2: Test Fl scores(% jof various methods for predicting the Composition edges.
Table 3: Test Fl scores(%) of various methods for generalization capability.
Methods
Fl score
Methods
F1 score
Poincaré Embeddings
43.8
Poincaré Embeddings
33.5
Hyperbolic Entailment Cones
44.0
Hyperbolic Entailment Cones
36.0
TransE
57.0
TransE
57.0
RotatE
51.0
RotatE
55.0
Order Embeddings
68.5
Order Embeddings
541.
MuRP
21.4
MuRP
20.1
AttH
51.3
AttH
27.0
RotE
51.5
RotE
48,8
RotH
55.8
RotH
46.7
Two-Box Box MODEL.
PPttel et al. 2020
68.1
Two- TWO-BoX MODEL (with GumbelBox)
58.9
Two-Bok MODEL (with GumbelBox)
73.7
Box-TE NOX-TRANFFORM MODEL
63.9
NOX-TRANSFORE MODEL
82.2
From Table 2 we observe that BOX-TRANSHORN TRANSFORM MODEL outperforms the other baselines by significant margin. As mentioned Patel et al. (2020) and so do we observe that in the next section that the Poincaré embeddings and Hyperbolic entailment cones do face difficulty in learning when presented only with transitive reduction edges. However, the hyperbolic KG method Atth RoTH are able to learn the composite edges to a certain extent. The performance gain of RotH over its euclidean counterpart RotE can be attributed to its inductive bias towards modeling hierarchies. The performance of Box embedding method as proposed by Patel et al. (2020) performs at par order embedding method. However using GumbelBox formulation (Dasgupta et al. 2020), we observe significant performance boost as GumbelBox improves the local identifiability of the parameter space.
7

---

Under review as a conference paper at ICLR 2021
Still, the capability of the Box- TRX-SRRNSORRM MODEL to benefit from shared cross-hierarchy features allows to substantially outperform even this improved version of the Two-Box Box MODEL. This is likely due to the fact that the inductive bias provided by the transformation is more in line with the data; the model can benefit from the containments learned as result of the IsA relation, and learn a HASPART transformation which potentially preserves these containments.
5.4 LEARNING FROM INCOMPLETE TRANSITIVE REDUCTION
In Patel al. (2020). and also in our previous experiment, we already observe that box embedding methods are highly capable of to recovering the transitive closure (in our case, composition edges) given the transitive reduction only. In this experiment, we train with even less of the transitive reduction, moving some of these edges to the test set. Now, reconstruction of the closure and the composition edges require models to generalize over the missing parts of the graph. We train on 9175 meronymy edges and 80372 hypernymy edges and test/validate edges. Please refer to the Appendix A.2 for details on dataset creation and statistics.
aggregated pool of 1783
From Table observe that Box-TRe TEX-TRANSFRR MODEL outperforms all the baseline methods by MODEL,
a large extent. Although the two box model is performing worse than BOX-TRANSE able beat other baselines. Out of the Knowledge base completion methods TransE performs the best and achieves comparative performance to two box model. Although the hyperbolic KG embeddings were able to perform well on the composite edges, their generalization performance is relatively lower than other KG embedding methods. We also observe that the RotE model that was under performing in composite edges, outperforms RotH by some margin in this generalization setting. We select the top three best performing methods for further analysis for each type of edges in the graph.
5.5 PERFORMANCE ANALYSIS ON DIFFERENT SPLITS
Training on a subset of the transitive reduction showed that our model could generalize to composition edges with the absence of essential edges make such prediction. We further perform evaluation analysis using the same training data with the best-performed model selected by maximizing the f1 score on composition edges. We evaluate the model performance on the transitive closure for each hierarchy (ISA and HASPART) ), and the composition edges on the joint hierarchy.
Table 4: Single hierarchy Fl score (%) analysis on IsA and HASPART The overall dataset is the combination of overfitting, generalization and extended generalization
Extended Generalization TC(X) (CXX)TC(XI))
Overall TC(X
Overfitting TC(11)
Generalization X-X1
Type
-(X-XI)
TransE Two Box Model Box- TRANSRRNNOORM MODEL
52.9
52.1
66.5
46.0
ISA
47.8
58,9
19.9
22.9
57.3
60.0
65.9
44.4
TransE Two Box Model TRX-FRASFORRM MODEL.
59.9
63.0
56.1
48.3
HASPART
51.6
54.8
40.8
37.8
58.8
64.2
33.4
25.4
For each single hierarchy, some edges are removed from the transitive reduction to create the incomplete transitive reduction training data X1. Evaluating the transitive closure of X directly evaluates the model's performance on each hierarchy, denoted as TC(X). This can be further divided into three categories: dataset that evaluates model ability to capture transitive closure of X1, TCXX1), dataset that evaluates model generalization ability on missing edges - X1, and dataset that evaluates model's extended generalization ability TC(X) TC(X1).
8

---

Under review as a conference paper at ICLR 2021
Table 5: Joint hierarchy Fl score (%) analysis. The overall data is the combination of overfitting and
generalization.
Overall COMP(X, Y)
Overfitting COMP(XI, Y1)
Generalization COMP(X, Y) COMP(X1, Y1)
TransE Two Box Model BOX-TRANSFORM MODEL
58.8
70.1
68.6
62.5
72.7
63.6
69.6
86.1
70.0
Composition edges from the joint hierarchy can be analyzed the same way. COMP(X, Y) represent all the composition edges in the full wordnet dataset, composed by IsA transitive reduction X and HASPART transitive reduction Y. It can be further divided into two categories: data that evaluate model overfitting ability capture COMP(X,; Y1) where X1 and Y1 is the corresponding training IsA and HASPART data in section 5.4 and data that evaluate model generalization ability on learning logical operations COMP(XX X, Y)
COMP Y1). The detailed statisties on each of these splits are provided in Appendix A..3 The evaluation dataset is created by randomly creating negative examples with the pos: neg ratio I:10. We select the top best models from section 5.4 then choose the threshold that maximized the F1 score for the validation data of each split and report the test F1. As shown in table and (5 our model performs the best overall across different dataset splits. TRX-RRA MODEL performs much better on the full transitive closure of IsA and all the composition edges. In general, -30X-TRAC TRANSFORM MODEL performs much better on transitive closure and composition edges by large margin in all overfitting settings. TransE does better on predicting removed edges from the transitive reduction (which serves more as an analysis of the model's capability, it is not typical evaluation for partial order completion), however we note that our model does surprisingly well on the IsA missing edges, which we attribute to the shared semantics between the hierarchy made possible by this box-to-box transformation.
6 CONCLUSION
We proposed box-to-box transformation which facilitates sharing of learned features across hierar- chies. We demonstrate the TOX-TRRNSFORM MODEL is capable of excellent performance when predicting compositional edges across joint hierarchy. Furthermore, the model does an excellent job modeling the transitive closure of each relation independently. In the future, extending from two relations to modeling multiple relations is essential in order to obtain more generalization from hicrarchical IsA edges.
REFERENCES
Ralph Abboud, ismail İlkan Ceylan, Thomas Lukasiewiez, and Tommaso Salvatori. Boxe: box embedding model for knowledge base completion. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems NeurIPS, 2020.
Ben Athiwaratkun and Andrew Gordon Wilson. Hierarchical density order embeddings. In Interna- tional Conference on Learning Representations, 2018.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019a.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. Multi-relational poincaré graph embeddings. In H. Wallach, H. Larochelle, A. Beygelzimer, dAlché-Bue Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32, pp. 4463-4473, Curran Asso- ciates, Inc., 2019b. URL http:/tttta:://ooeeetettee iewpoeeeeeddnnn...enneisee. neurips wttpppppcccppppep//////////////e .pdf
E8b932c70d0ss 00b552277002ee600117244446a6aa.2
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: www. hwtp/wwwwww..www............. wandb
com/ Software available from wandb.com.

---

Under review as a conference paper at ICLR 2021
Antoine Bordes, Nicolas Usunier, A. Garcia-Duran Jason Weston, and Oksana Yakhnenko. Translat- ing embeddings for modeling nulti-relational data. In Neural Information Processing Systems, 2013.
Benjamin Paul Chamberlain, James R. Clough, and Mare Peter Deisenroth. Neural embeddings of graphs in hyperbolic space. 13th international workshop on mining and learning from graphs held in conjunction with KDD, 2017.
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low- dimensional hyperbolic knowledge graph embeddings. arXiv preprint arXiv:2005.005:0 2020.
Shib Sankar Dasgupta, Michael Boratko, Dongxu Zhang, Luke Vilnis, Xiang Lorraine Li, and Andrew McCallum. Improving local identifiability for probabilistic box embeddings. In Neural Information Processing Systems, 2020.
Katrin Erk Representing words as regions in vector space. In Proceedings of the Thirteenth Conference Computational Natural Language Learning, 2009.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in neural information processing systems, pp. 5345-5355, 2018a.
Octavian-Eugee Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning, 2018b.
James A Hampton. The combination of prototype concepts. The psychology word meanings, pp. 91-116, 1991.
Alice Lai and Julia Hockenmaier. Learning to predict denotational probabilities for modeling entailment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017.
Xiang Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. Smoothing the ge- ometry of probabilistic box embeddings. In International Conference on Learning Representations, 2019.
George A Miller. WordNet: a lexical database for English. Communications of the ACM, 1995.
Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Neural Information Processing Systems, 2017.
Robert M Nosofsky. Attention, similarity, and the identiication-categriiaaat relationship. Journal of experimental psychology: General, 1 115(1))39, 1986.
Dhruvesh Patel, Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, and Andrew McCallum. Representing joint hierarchies with box embeddings. Automated Knowledge Base Construction, 2020.
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. International Conference on Learning Representations, 2020.
Edward E Smith, Daniel N Osherson, Lance Rips, and Margaret Keane. Combining prototypes: A selective modification model. Cognitive science, 2(4)):85-521) 1988.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. International Conference on Learning Representations, 2019.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, 2016.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In International Conference on Learning Representations, 2016.
Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. International Conference Learning Representations, 2015.
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic embedding of knowledge graphs with box lattice measures. In Association for Computational Linguistics, 2018.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 TRAINING DETAILS
In our experiments, we have kept the number of parameters same across all the methods. We use dimensional box embeddings for the Two Box Model (Patel et al. 2020). Since box embeddings specified using min and side length in the same dimension. Thus we compare with 10 dimensional order embeddings, Poincaré embeddings, and hyperbolic entailment cones. However, since the above mentioned methods has two different number of parameters for each node, we use 20 dimensional vectors for RotatE, TransE to account for that. Our Box-TRANE TRANSFOR MODEL uses 10 dimension box embeddings for similar reason.
Hyperparameter range: We use Bayesian hypermeter optimizer with Hyperband algorithm for all the methods using the web interface Biewald (2020). The hyperparameter ranges Gumbel3 E [0.001,3] 3). Softplus temperature for box volume T & [1,30] Ir & [0.0005, 1]. batch size {8096, 2048, 1024, 512}. number of negative samples [2, 30) for all the methods. For max margin trainging we searched for the margin E [1, 50).
The best hyperparameters for our method and few competitive baselines are provided in appropriate config files along with the source code. We will make the code public after the anonymity period.
A.2 DATASET CREATION STEPS FROM SECTION .44
In order to remove edges from the transitive reductions, we iterate through the transitive reduction edges of meronymy. With 0.5 probability we choose the edge for further processing. For each chosen HASPART edge, we select an outgoing IsA edge and pair them. We drop the IsA edge from the pair with 0.9 probability (the ratio of HASPART to ISA transitive reduction) and drop the HASPART edge in case the ISA is not dropped already.
This procedure ensures that all the edge removals happen around the composition edges, thus, the results reflect the models true capacity to generalize well for this joint hierarchy task. We evaluate the model on the composition edges, the removed reduction edges, and the closure edges with 251783 in numbers which we split into parts for validation and test. In Table report the F1 score this aggregated evaluation data with 1:10 fixed true negatives.
A.3 DETAILS OF THE SPLITS FROM SECTION 5.5
Table 6: Dataset statistics for different parts of individual IsA and PARTOF hierarchy.
TC(X) TC(XI)
Hierarchy
TC(X)
TC(XI)
X-XI
-X-XI)
IsA
61,667 30,335
51,195 26,388
3,991
6,481
HasPart
503
3,444
Table 7: Dataset statistics for different composition edges in Joint Hierarchy.
COMP(X. YI) COMP(XI, Y1)
Hierarchy
Comp(X, Y)
COMP(XI, YI)
Joint Hierarchy
189,613
146,867
42,746
A.4 VISUALIZATION
We plot 2-dimensional box embeddings inspect the quality of our proposed BOX-TRANSEE MODEL. We use the box embedding parameters of the best performing model of experiment [.3 (Table 2 Note that, the model is 10 dimensional. However, for perfectly trained model, should
11

---

Under review as a conference paper at ICLR 2021
observe containment along each dimension, Thus, we pick two dimensions randomly out of the 10-d to visualize the embeddings.
Haifert Movebe Darker
Movable_barrine
Hodfer
Door
HasPart
Car Door
Car
Composition Edgesi
IsA
Sedan
(a) Example of Joint Hierarchy extracted from the WordNet dataset.
(b) We plot the transformed IsA box for "Sedan" & "Car" and transformed HASPART box for "Door", "Car Door", "Movable Barrier" the same space. The transformations do preserve the containment and provide an consistent assignment on left.
embedddings
example
Figure 2: 2-dimensional visualization of proposed Box embedding model.
From the example, the facts that <Car.HASPART CCarDoor>> and <CarDoor, .Door>> would infer that <Car,HASPART. Door>. We observe from the Figure 1hhat the HASPART transformation of the "Car Door" and "Door" successfully encloses the IsA transformation of the "Car", thus our model is able infer that composition edge All the other composite edges such as <Sedan, HASPART CarDoor>,
Door> etc. can be similarly inferred from the visualization.
12


Reviews:
[{"review_id": "nqstfm0ADkb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with tree-like structure embedding with box embedding on the lattice (poset). This paper is well-motivated and well-presented. Though there is a limitation on data structure, this paper still presents a novel idea in this area. This method also achieved promising results in experiments. Thus, I would like to recommend to accept this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eRP16290P4b", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to model multiple hierarchical relations using box embeddings, motivated by the natural transitivity property of the containment between regions in region-based representations. The proposed model is evaluated on a dataset containing two relations (is-a and has-part). Although the proposed model shows promise by outperforming several baselines on the above mentioned dataset, I believe that the paper is not ready for publication in its current form, mainly due to (i) missing comparison to the highly relevant line of work on hyperbolic embeddings of hierarchical multi-relational data; and (ii) lacking additional experiments on a dataset with more than 2 relations.\n\n\nDetailed comments and questions for the authors:\n\nSec. 2\n1. I find the claim that \"Modeling joint hierarchies is not quite the same as knowledge base completion.\" to be unsubstantiated. This is true to an extent, since the ultimate goal of KB completion is inferring which other facts are true based on existing ones. However, to achieve this goal, KG completion models need to learn entity and relation representations which capture various properties of entities (e.g. semantics) and relations (e.g. transitivity, symmetry, etc.), which is very similar to the main idea of this work.\n2. A whole line of very relevant work on hyperbolic embeddings of hierarchical relations in knowledge bases is missing [1, 2].\n\nSec. 3\\\nIf I understood Definition 2 correctly, the meet (i.e. union) of two boxes will be another box which in most cases contains an area that is not part of either boxes (since a union of two boxes is not necessarily a box). Doesn't this introduce errors into box embeddings which increase with increasing the dimensionality of the embeddings?\n\nSec. 4\\\nIt is not clear to me why the lack of a transformation on f_1(b) encourages the containment in figure 1. Could you please explain this point further?\n\nSec. 5\n1. While the achieved results seem impressive, as mentioned above, a highly relevant comparison to [1] and [2] is missing. Both missing works are embedding models that represent multiple simultaneous hierarchies in hyperbolic space and where entity embeddings are shared across relations, which should lead to better generalisation on missing edges (as claimed by the authors).\n2. The authors evaluate the proposed model on a single dataset with only 2 relations. The proposed model should be evaluated on at least one more dataset, e.g. WN18RR [3], since [1] show several relations in that dataset to be hierarchical.\n3. I'm surprised that the improvement over the TWO-BOX model is lower when testing for generalisation capability (5%) than in the original setup (8.5%), given the original premise that the proposed model should benefit from sharing information across hierarchies.\n4. It would be nice to see a visualisation of the learned embeddings.\n\nMinor comments:\\\nBackground section should be made shorter, especially the part regarding the probabilistic box model training, which is not that relevant to the overall goal of this work. This space could be used for additional experiments proposed above.\n\n[1] Balazevic et al. Multi-relational Poincaré Graph Embeddings, NeurIPS 2019\\\n[2] Chami et al. Low-Dimensional Hyperbolic Knowledge Graph Embeddings, ACL 2020\\\n[3] Dettmers et al. Convolutional 2D Knowledge Graph Embeddings, AAAI 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "QCMSzfv8AM1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds upon the work of Patel et al. (2020) in modeling two hierarchies jointly within the box embedding framework. It also incorporates the GumbelBox formulation of Dasgupta et al. (2020) to resolve local identifiability issues during training. \n\nThe contribution of the paper seems to only lie in the learning of a function \\phi that maps entity boxes to HasPart-* boxes. This function constrains the HasPart-* boxes in two ways: (a) their \"minimum\" corners remain at the same relative positions as their corresponding entity boxes, and (b) their lengths are scaled proportionately in each dimension. In contrast, Patel et al. (2020) does not have these constraints in their model. I find the novelty of these constraints to be incremental, especially in view that the joint hierarchy problem and evaluation methodology have already been formulated by Patel et al (2020) in the context of box embeddings. Though seemingly straightforward, the constraints do help the paper to improve upon the state of the art by significant margins in the experiments.  \n\nThe paper is well organized and clearly written for the most part, but the exposition can be improved in some areas.\n\n* Section 4.1, (a <_1 b) ^ (b <_2 c) => (a <_2 b): Could the authors provide examples of what <_1, <_2, a, b, and c represent? \nI interpret \"a <_1 b\" to be b IsA a, \"b <_2 c\" to be c HasPart b, which then leads to \"c HasPart a\". This means that the consequent should (a <_2 c) rather than (a <_2 b), no? \n\n* Section 1, 3rd para: hiearchy->hierarchy, \n\n* Section 1, 4th para: dialate->dilate\n\n* Section 5.3, 1st para, \"for those edges in table 5\" -> should be \"table 2\"?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Khrm-tvfVEL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on modeling multiple hierarchical relations on a heterogenous graph.  The task “modeling joint hierarchies” is essentially trying to infer whether a given pair of entities has a hierarchical connection especially when there exists multiple hierarchical relations (2 in the paper), and missing links. The paper proposes to embed entities using boxes whose endpoints follow the Gumbel distribution. Given there exists two hierarchical relations, the paper transforms the box of one entity under relation 1 to the box of the entity under relation 2 with a parameterized linear function. This is in contrast to previous work that parameterized the box of two relations using separate independent parameters. \n\nThe model seems sound, however I have two major concerns. (1) I do not think the model is motivated well, especially on why the model uses Gumbel distribution to parameterize the box. (2) The paper has no introduction how they train the model and use it for inference, what is the loss? This makes it hard to evaluate the correctness of the model.\n\nI am very satisfied with the extensive experiments the paper has conducted. They include many strong baselines including the order embeddings, hyperbolic embeddings and even some KG embeddings. The results on the KG embeddings clearly show that their methods work much better in this (a little specific) hierarchical relation modeling setting. The paper also introduces a new missing-edge setting, where they show that joint modeling achieves better generalization than independent parameters. \n\nSome detailed questions are listed below.\n1. The related work states the difference between modeling hierarchies and knowledge base completion, however, it lacks discussion how their Gumbel box is different from previous box embedding methods (this should be added in the second paragraph). I understand the difference between the Gumbel box and the Two-box model, namely the Two-box model learns independent parameters. However, I did not find the discussion on the connection between the Gumbel box and hard/smooth box. Why cannot we apply the same transformation idea to previous hard and smoothed box embeddings so that they can also model joint hierarchies without optimization issues? Why is Gumbel distribution special and useful in parameterizing the boxes and modeling hierarchies?\n2. The paper has some vague sentences like “the authors demonstrate that this method of training boxes leads to better representation of trees thus we will use this Gumbel box approach in our setting.” and “since gumbel boxes effectively model hierarchies, we would like to benefit from the inductive bias of this model for intra-relation edges and thus we seek to learn a function ...”, but what is the inductive bias of Gumbel? It’s better to clearly state it.\n3. The paper lacks a short discussion and introduction to the Gumbel distribution in the background section, especially on the parameters \\mu and \\beta.\n4. As defined in Eq. 3, the meet of two boxes may include some blank space that does not belong to the input boxes, do you think this will have any issues, especially when the two input boxes are far away from each other?\n5. Sec 4.1, first paragraph, “$(a \\leq_1 b) \\wedge (b \\leq_2 c) \\to (a \\leq_2 b)$” is wrong. Bird has part Wing, and Wing is an Appendage, but Bird is not a Wing.\n6. Sec 4.1, end of page 4, “To simultaneously model a second relation, we ...”, so the model can only model two hierarchical relations? If so, I think it is a little limited and can the model provide a way to generalize beyond two hierarchical relations?\n7. Sec 4.1, “the free parameters are $\\mu_i$ and $\\Delta_i$”, why does the model not learn $\\beta$?\n8. As in Eq. 11 and 12, the transformation is a rather simple linear transformation, have you tried something that is more complex, e.g. a MLP? \n9. I am also confused by Remark 1 and Eq. 8. For Bird, there should be two boxes where one represents the IsA relation and the other represents the HasPart relation, right? Then in Figure 1,  why is the IsA-Bird box inside the HasPart-Wing box, I think it should be the HasPart-Bird box inside the HasPart-Wing box.\n10. The paper does not introduce how to train the model or even how to make predictions during inference in Sec 4. I understand the page limit but these two aspects are essential to a machine learning model.\n11. What is the difference between the two-box model and the order embeddings in the experiments? I assume if you apply the order embeddings to this multi-hierarchical relation setup, then it is the same as the two-box model?\n12. I am curious about the performance of the proposed model in an imbalanced dataset (as introduced in Li et al. ICLR 2019), where the ratio of positive and negative is 1:10? \n\nminors: The paper does not have grammar mistakes and here are some minor points.\n1. Make it explicit in the introduction that the “Two-Box Model” is referred to Patel et al. (2020)\n2. The definition of box lattice model is not self-contained in Eq. 1, what is $x_i$ and $x^i$? I guess it is the two end points of the box in one dimension. Better to state it clearly.\n3.  Sec 3.3, “For example, as shown in 1, based on…” -> “For example, as shown in Figure 1, ..”\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is concerned with modeling multi-relational data with joint hierarchical structure. For this purpose, the authors extend box embeddings to multi-relational settings, supporting the modeling of cross-hierarchy edges and generalizing from a subset of the transitive reduction. The reviewers highlight that the paper is, overall, well-written and organized, relevant to the ICLR community, and that the proposed method offers promising experimental results. Furthermore, the author's rebuttal clarified some concerns of the initial reviews (e.g., relation to GumbelBox, comparison to additional baselines etc.) and improved the manuscript.

However, after rebuttal there exist still concerns regarding the current version. Reviewers raised concerns regarding novelty, clarity, and the empirical evaluation (importantly modeling more than 2 relations; it would also be good to understand more clearly why some of the newly added multi-relational hyperbolic baselines perform worse than uni-relational Poincare embeddings). While the paper and the proposed method clearly have promise, I agree with reviewers that the manuscript would require an additional revision to clarify these points. Given the positive aspects of the paper, I'd strongly encourage the authors to revise and resubmit their work given this feedback.
Decision: Reject