Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper deals with tree-like structure embedding with box embedding on the lattice (poset). This paper is well-motivated and well-presented. Though there is a limitation on data structure, this paper still presents a novel idea in this area. This method also achieved promising results in experiments. Thus, I would like to recommend to accept this paper.
review_title: An interesting paper.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper builds upon the work of Patel et al. (2020) in modeling two hierarchies jointly within the box embedding framework. It also incorporates the GumbelBox formulation of Dasgupta et al. (2020) to resolve local identifiability issues during training. 

The contribution of the paper seems to only lie in the learning of a function \phi that maps entity boxes to HasPart-* boxes. This function constrains the HasPart-* boxes in two ways: (a) their "minimum" corners remain at the same relative positions as their corresponding entity boxes, and (b) their lengths are scaled proportionately in each dimension. In contrast, Patel et al. (2020) does not have these constraints in their model. I find the novelty of these constraints to be incremental, especially in view that the joint hierarchy problem and evaluation methodology have already been formulated by Patel et al (2020) in the context of box embeddings. Though seemingly straightforward, the constraints do help the paper to improve upon the state of the art by significant margins in the experiments.  

The paper is well organized and clearly written for the most part, but the exposition can be improved in some areas.

* Section 4.1, (a <_1 b) ^ (b <_2 c) => (a <_2 b): Could the authors provide examples of what <_1, <_2, a, b, and c represent? 
I interpret "a <_1 b" to be b IsA a, "b <_2 c" to be c HasPart b, which then leads to "c HasPart a". This means that the consequent should (a <_2 c) rather than (a <_2 b), no? 

* Section 1, 3rd para: hiearchy->hierarchy, 

* Section 1, 4th para: dialate->dilate

* Section 5.3, 1st para, "for those edges in table 5" -> should be "table 2"?


review_title: Seemingly incremental contributions, but with significant empirical improvements.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper focuses on modeling multiple hierarchical relations on a heterogenous graph.  The task “modeling joint hierarchies” is essentially trying to infer whether a given pair of entities has a hierarchical connection especially when there exists multiple hierarchical relations (2 in the paper), and missing links. The paper proposes to embed entities using boxes whose endpoints follow the Gumbel distribution. Given there exists two hierarchical relations, the paper transforms the box of one entity under relation 1 to the box of the entity under relation 2 with a parameterized linear function. This is in contrast to previous work that parameterized the box of two relations using separate independent parameters. 

The model seems sound, however I have two major concerns. (1) I do not think the model is motivated well, especially on why the model uses Gumbel distribution to parameterize the box. (2) The paper has no introduction how they train the model and use it for inference, what is the loss? This makes it hard to evaluate the correctness of the model.

I am very satisfied with the extensive experiments the paper has conducted. They include many strong baselines including the order embeddings, hyperbolic embeddings and even some KG embeddings. The results on the KG embeddings clearly show that their methods work much better in this (a little specific) hierarchical relation modeling setting. The paper also introduces a new missing-edge setting, where they show that joint modeling achieves better generalization than independent parameters. 

Some detailed questions are listed below.
1. The related work states the difference between modeling hierarchies and knowledge base completion, however, it lacks discussion how their Gumbel box is different from previous box embedding methods (this should be added in the second paragraph). I understand the difference between the Gumbel box and the Two-box model, namely the Two-box model learns independent parameters. However, I did not find the discussion on the connection between the Gumbel box and hard/smooth box. Why cannot we apply the same transformation idea to previous hard and smoothed box embeddings so that they can also model joint hierarchies without optimization issues? Why is Gumbel distribution special and useful in parameterizing the boxes and modeling hierarchies?
2. The paper has some vague sentences like “the authors demonstrate that this method of training boxes leads to better representation of trees thus we will use this Gumbel box approach in our setting.” and “since gumbel boxes effectively model hierarchies, we would like to benefit from the inductive bias of this model for intra-relation edges and thus we seek to learn a function ...”, but what is the inductive bias of Gumbel? It’s better to clearly state it.
3. The paper lacks a short discussion and introduction to the Gumbel distribution in the background section, especially on the parameters \mu and \beta.
4. As defined in Eq. 3, the meet of two boxes may include some blank space that does not belong to the input boxes, do you think this will have any issues, especially when the two input boxes are far away from each other?
5. Sec 4.1, first paragraph, “$(a \leq_1 b) \wedge (b \leq_2 c) \to (a \leq_2 b)$” is wrong. Bird has part Wing, and Wing is an Appendage, but Bird is not a Wing.
6. Sec 4.1, end of page 4, “To simultaneously model a second relation, we ...”, so the model can only model two hierarchical relations? If so, I think it is a little limited and can the model provide a way to generalize beyond two hierarchical relations?
7. Sec 4.1, “the free parameters are $\mu_i$ and $\Delta_i$”, why does the model not learn $\beta$?
8. As in Eq. 11 and 12, the transformation is a rather simple linear transformation, have you tried something that is more complex, e.g. a MLP? 
9. I am also confused by Remark 1 and Eq. 8. For Bird, there should be two boxes where one represents the IsA relation and the other represents the HasPart relation, right? Then in Figure 1,  why is the IsA-Bird box inside the HasPart-Wing box, I think it should be the HasPart-Bird box inside the HasPart-Wing box.
10. The paper does not introduce how to train the model or even how to make predictions during inference in Sec 4. I understand the page limit but these two aspects are essential to a machine learning model.
11. What is the difference between the two-box model and the order embeddings in the experiments? I assume if you apply the order embeddings to this multi-hierarchical relation setup, then it is the same as the two-box model?
12. I am curious about the performance of the proposed model in an imbalanced dataset (as introduced in Li et al. ICLR 2019), where the ratio of positive and negative is 1:10? 

minors: The paper does not have grammar mistakes and here are some minor points.
1. Make it explicit in the introduction that the “Two-Box Model” is referred to Patel et al. (2020)
2. The definition of box lattice model is not self-contained in Eq. 1, what is $x_i$ and $x^i$? I guess it is the two end points of the box in one dimension. Better to state it clearly.
3.  Sec 3.3, “For example, as shown in 1, based on…” -> “For example, as shown in Figure 1, ..”


review_title: nice experimental results, lack motivation and training details

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposes to model multiple hierarchical relations using box embeddings, motivated by the natural transitivity property of the containment between regions in region-based representations. The proposed model is evaluated on a dataset containing two relations (is-a and has-part). Although the proposed model shows promise by outperforming several baselines on the above mentioned dataset, I believe that the paper is not ready for publication in its current form, mainly due to (i) missing comparison to the highly relevant line of work on hyperbolic embeddings of hierarchical multi-relational data; and (ii) lacking additional experiments on a dataset with more than 2 relations.


Detailed comments and questions for the authors:

Sec. 2
1. I find the claim that "Modeling joint hierarchies is not quite the same as knowledge base completion." to be unsubstantiated. This is true to an extent, since the ultimate goal of KB completion is inferring which other facts are true based on existing ones. However, to achieve this goal, KG completion models need to learn entity and relation representations which capture various properties of entities (e.g. semantics) and relations (e.g. transitivity, symmetry, etc.), which is very similar to the main idea of this work.
2. A whole line of very relevant work on hyperbolic embeddings of hierarchical relations in knowledge bases is missing [1, 2].

Sec. 3\
If I understood Definition 2 correctly, the meet (i.e. union) of two boxes will be another box which in most cases contains an area that is not part of either boxes (since a union of two boxes is not necessarily a box). Doesn't this introduce errors into box embeddings which increase with increasing the dimensionality of the embeddings?

Sec. 4\
It is not clear to me why the lack of a transformation on f_1(b) encourages the containment in figure 1. Could you please explain this point further?

Sec. 5
1. While the achieved results seem impressive, as mentioned above, a highly relevant comparison to [1] and [2] is missing. Both missing works are embedding models that represent multiple simultaneous hierarchies in hyperbolic space and where entity embeddings are shared across relations, which should lead to better generalisation on missing edges (as claimed by the authors).
2. The authors evaluate the proposed model on a single dataset with only 2 relations. The proposed model should be evaluated on at least one more dataset, e.g. WN18RR [3], since [1] show several relations in that dataset to be hierarchical.
3. I'm surprised that the improvement over the TWO-BOX model is lower when testing for generalisation capability (5%) than in the original setup (8.5%), given the original premise that the proposed model should benefit from sharing information across hierarchies.
4. It would be nice to see a visualisation of the learned embeddings.

Minor comments:\
Background section should be made shorter, especially the part regarding the probabilistic box model training, which is not that relevant to the overall goal of this work. This space could be used for additional experiments proposed above.

[1] Balazevic et al. Multi-relational Poincaré Graph Embeddings, NeurIPS 2019\
[2] Chami et al. Low-Dimensional Hyperbolic Knowledge Graph Embeddings, ACL 2020\
[3] Dettmers et al. Convolutional 2D Knowledge Graph Embeddings, AAAI 2018
review_title: Promising paper with lack of experimental support

Meta-review: The paper is concerned with modeling multi-relational data with joint hierarchical structure. For this purpose, the authors extend box embeddings to multi-relational settings, supporting the modeling of cross-hierarchy edges and generalizing from a subset of the transitive reduction. The reviewers highlight that the paper is, overall, well-written and organized, relevant to the ICLR community, and that the proposed method offers promising experimental results. Furthermore, the author's rebuttal clarified some concerns of the initial reviews (e.g., relation to GumbelBox, comparison to additional baselines etc.) and improved the manuscript.

However, after rebuttal there exist still concerns regarding the current version. Reviewers raised concerns regarding novelty, clarity, and the empirical evaluation (importantly modeling more than 2 relations; it would also be good to understand more clearly why some of the newly added multi-relational hyperbolic baselines perform worse than uni-relational Poincare embeddings). While the paper and the proposed method clearly have promise, I agree with reviewers that the manuscript would require an additional revision to clarify these points. Given the positive aspects of the paper, I'd strongly encourage the authors to revise and resubmit their work given this feedback.
Decision: Reject