Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: When embedding entities and relations in knowledge graphs, operations such as translations, matrix-vector products, quadratic forms, etc are used to represent relations between entity vectors. While these vector-based methods are flexible, they do not specifically encode an inductive bias towards transitive relations that is necessary for modeling certain hierarchies. This paper proposes an approach based on projections (idempotent transformations) and rotations taken together, which allows it to exactly model transitivity in relations.
main_review: Update: I will stick with my current score, I believe that the authors have adequately enough addressed reviewers' comments.

**Originality**

There has been much work on modeling transitive relations with specialized embedding techniques such as order embeddings, hyperbolic embeddings, box embeddings etc. However, integrating these approaches into standard multirelational / hypergraph KG embedding based on vectors, which allows for multiple heterogeneous relations to be embedded, is an open problem. As the authors note, the recent RotatE model is able to model (a)symmetry, composition, and inversion, but cannot specifically guarantee transitivity. They add a learned subspace projection step to the RotatE model and prove this can represent transitive relations. 

**Quality**

The theoretical argument is sound. The evaluation is reasonable and standard for KG embedding techniques, and showing the model can indeed infer transitivity is welcome. The discussion of limitations and local optima in the model is welcome.

**Clarity**

The paper is mostly well written and argued. The final error-analysis section could use some light editing to tighten it up, as it currently reads more like an appendix discussion.

**Significance**

The RotatE model is simple and powerful, and this extension of it seems similarly simple and powerful. Knowledge base embedding approaches are popular and this might become a new baseline technique. 

limitations_and_societal_impact: Limitations with respect to local optima in optimization, and initialization, are discussed extensively in the experiments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper noticed that if a relation projected orthogonally to the translation direction, it becomes transitive, and then make this insight work with other relational patterns.

main_review: I liked this paper. It had a good insight and seems to work well. It has good originality, and is mostly clear. It is potentially significant.

However, I am very suspicious of whether there are side effects. For example, ancestor is the transitive closure of parent. Because ancestor has to have some particular form in order to be transitive, doesn't that imply that parent must also have some particular form?  What happens when parent is used in other relations? While I believe your theorem (although I didn't check the related papers), it doesn't show that they can be handled simultaneously. If there was also a modularity result, the consequent of the design patterns do not affect the representation of the condition  of the design pattern  (i.e., they work for all representations of the condition), I would be more convinced of your claim that you can handle all relation patterns. (Claiming to handle all is ambiguous as to whether you can handle them separately or together.)

Isn't transitivity an instance of composition, with r = r \circle r?  I think there are more assumptions behind the design patterns than you state.

Do your results imply other patterns, eg  r = r1 \circle r?  Are there other patterns that can exist? You should tell us whether this is the end of the story of representing design patterns of whether there are more. (Students looking for research problems want to know).

I very much appreciated section 4.6 on limitations.

Please label x and y axes of all figures. Figure 5 b has loss on the y-axis but how can it be negative?  


limitations_and_societal_impact: Yes. Section 4.6 was nice to see!
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary: This study proposes a new geometric model Rot-Pro that enables the learning of transitive inference patterns by combining rotation and projection. The core idea is to involve a learnable idempotent projection onto a reduced basis. This achieves the transitivity pattern without forcing the entity embeddings to be the same as it is in previous models. The study shows the effectiveness in experimental results mostly on the small Countries dataset, but also shows that the model performs well on larger KG benchmarks especially on the Yago3-10 dataset. The authors also report their investigation into initialization schemes that are required for successfully training the model. 
main_review: Originality: There is a lot of recent work in geometric models that explicitly investigate modelling certain inference patterns and this study proposes to investigate transitivity in a rotational model that expands its capabilities to more inference patterns. I was missing a comparison and discussion to BoxE, which is a strong model in this area.

Quality: The line of arguments and to motivate the study are well laid out, and proofs and experiments support the claim. The authors point out the challenge of initialization for the model and they mention a special loss to enforce zeros and ones in the projection. Did the authors also consider something like [1]? In the experimental results I was also missing a comparison to BoxE, which reports better results on Yago3-10.
 
Clarity: The paper was written very well and has a nice presentation. 
 
Significance: The paper presents a solid idea which is executed well to show the effectiveness of explicitly allowing a model to learn transitive patterns in the data. While the authors note that there is room for improvement, this does not limit the usefulness of the analysis, proposed model and results. 
 
[1] Louizos, C., Welling, M., and Kingma, D. P. Learning sparse neural networks through l0 regularization. CoRR, 2017.

limitations_and_societal_impact: The limitations and societal impact haven been  addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is related to the problem of Knowledge Graph (KG) Embedding. It seems that none of the existing approaches for KG embedding is capable of modeling the transitivity property of the relation which is an important property for many relations to have in real-world KGs. The aim of this paper, therefore, is to bridge this specific research gap. 

This paper proposes a KG embedding model called __Rot-Pro__ which not only is capable of modeling the transitive property of a relation but also other properties including symmetry, asymmetry, inversion, and composition. The approach is based on applying a projection operator on the embedding of the head and tail entities before applying RotatE so as to be able to model transitive relations. The experiments are performed on two well-known datasets FB15k-237 and WN18RR. The experiments are also performed on two other datasets called YACO3-10 and Countries because they have lots of transitive relations present in them. 

On the datasets, where transitive relations are dominant, the approach seems to do justice but the approach seems to be facing some hiccups in the presence of other types of relations such as symmetry.
main_review: __Originality__
- The problem of KG embedding is a well-known research topic in the field. This paper identifies an important limitation of the existing approaches and tries to fix that.
- The proposed approach in this paper builds on top of the well-known approach called RotatE. The proposed approach suggests adding one extra pre-processing step to the entity embeddings before calling the routine of RotatE.  The pre-processing step suggests that all the entity embeddings should be first (orthogonally) projected into a subspace of the embedding space via projection matrices. These projection matrices are specific to each relation as well as each embedding dimension. This extra step introduces only 3 extra model parameters ($\theta_p(k)$ and $a$ or $b$) per embedding dimension. 
- This extra step alone allows RotatE to model the transitive property of relations. Overall, this extra step, though seemingly small, does offer the desired effect (at least theoretically). Theorem 1 confirms this in a theoretical manner.  
- In light of the above, I feel the contributions can be considered somewhat novel. 
- However, there are several doubts/clarifications that pop in the mind of a reader (listed below) and clarification of them will help me judge the merits of this work in an informed manner.

__Quality__
- The overall technical quality of the paper seems reasonable. The supported claim is proven theoretically and also validated experimentally. Although experiments don't support it fully and I guess that is scope for future research.
- Authors are honest about articulating the weakness of the method observed during their experimentation.

__Clarity__

Overall, I felt that this paper lacks clarity and has ample scope for improving its writing. Below are some of the doubts/clarifications that struck me while reading the paper.

- In equation (2), I assume the matrix $M$ would be different for different coordinate axes as well as relations? If yes, you may want to use the symbol $M_r(k)$ or $M_{rk}$, where $k$ is the index of a coordinate axis and $r$ denotes the considered relation. Given that you have used index $k$ in the definition for the parameter $\theta_p(k)$, introducing $k$ for $M$ will make notations a bit more consistent. In fact, by the same token, matrices $S, S^{-1}$, as well as parameters $a$ and $b$ should also bear the index $k$ and $r$. Please correct me if I have misunderstood something here.  
- By the same logic, even for Equation (3), you may want to consider using the symbol $p_{rk}$ or $p_r(k)$ if $M$ is indexed with $r$ and $k$ on the right side of the equation? If these notions become clumsy then you may want to add a disclaimer saying that you are omitting some indices to keep the notation less cluttered.
- While I could infer from reading that Rat-Pro model embeds entities into a complex space ${\cal{C}}^d$, would it make sense to state it explicitly somewhere? 
- After reading the last line of the _Conclusion_ section, I am a bit confused as it seems to suggest that the current embedding is being performed in ${\cal{C}}^1$ space? Table 2 of the appendix seems to suggest that $d>1$. Please clarify this.
- In Equation (10), the outer summation is over variable $x$ which doesn’t show up in summand term. Can you make it clear?
- In equation (10), it's not clear what is $\mathbf{x}_j$ vector and what is its dimension? If running index $j$ corresponds to an axis then I believe $\mathbf{x}_j$ would be a scaler? But in that case, Hadamard product does not make sense. 
- What benefits do you get by using the loss form given in Equation (10) as opposed to minimizing other simpler forms such as $\mathbf{x}_j \odot (1-\mathbf{x}_j)$ which also try to push the values of $\mathbf{x}_j$ towards $0$ or $1$?
- Given that the overall optimization problem becomes an integer programming problem ($a, b \in {0,1}$),  how do you finally ensure that in the learned matrix M, the diagonal terms $a$ and $b$ are exactly $0$ or $1$. I suppose the inclusion of loss term (10) still does not guarantee to have the learned values of the parameters $a,b$ being $0$ or $1$. Do you round off the learned values for $a$ and $b$? 
- The projection matrix $M$ seems to be orthogonal projection matrix also because it appears that $S^{-1} = S^{\top}$ and hence $M=M^{\top}$. In this case, it may be better to explicitly state that you are focusing on the orthogonal projections of the entities.
-  In Figures 3(a), 3(b), and 3(c), what does the $y$-axis corresponds to? 
- Sentence in line # 238 needs some rewriting as it doesn’t read properly.
- In Figure 4(a), how does the pattern for Hits@1 score look like? Does it follow the same trend as Figure 4(a)? For KG embeddings, I feel Hits@1 is crucial to track.
- How about the expressivity of the Rat-Pro model? Is there any relation that can’t be model by the proposed approach? Can you add some discussion around that?

__Significance__

- Overall, this paper could have a noticeable impact in advancing the SOTA provided it satisfactorily addresses the doubts/concerns raised above. 
- The experimental section exposes a limitation saying that the presence of transitive relations dilutes the learning of other types of relations (as discussed in Section 4.5 and 4.6). These limitations kind of puts a question mark saying whether this paper managed to bridge the research gap completely. However, I feel, if not completely, this paper still managed to bridge the research gap and advance the SOTA to a decent and respectable level.
limitations_and_societal_impact: Yes. The authors have explicitly discussed the limitations of the proposed method which they observed during experimentation. There are a few minor clarifications I have in this regard which I have listed in my points above.
review_title: null

Meta-review: the reviewers agreed that the paper makes a solid contribution by proposing a model that can infer transitivity, together with convincing experimental results. Most of the reviewers' concerns have been cleared in the discussion phase. The authors should make sure to incorporate the necessary updates in the camera-ready version.


Decision: Accept (Poster)