Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
There has been a large family of knowledge base models developed in the recent years, aiming to encode both entities and relations in a latent space, so the entities can be “linked” via a relation-specific mapping.

The paper focuses on understanding these entity embeddings (and geometric embedding relationships), built on top of the connections with PMI-based word embeddings.

The paper categorizes all the relations into 3 types (1) related, (2) specialisation, and (3) context shift, and examines some relations in WordNet and NELL, and then empirically evaluates the performance of different types of models and draws the correlation of the results and intuitive understanding of different types of relations.

To me, this paper is more like providing some intuitive explanations of existing KG embeddings methods and their performance (not really theoretical justifications). It was an interesting read and I appreciate the authors trying to understanding the latent structure that has been encoded in these models. However, I am just not that sure how many take-aways we can get from this study. 

I am wondering how loose this categorization is , esp. for the important relations in practice. I’d be also interested in seeing more results on Freebase (and possibly Wikidata) as those KG embeddings are usually more useful. As indicated in the Appendix, the paper mentiosns most of FB15k-237 datasets are in type C, so I am just not sure how many R/S relations are actually there.

Also, according to Table 3 and Table 4, I am not sure if there are any surprising findings from there. It seems that there is some randomness/noise, but MuRE generally works better than the others. It is true that DistMult works well on the R-type relations but it is not consistent between WN and NELL.

It’d be useful to show results on more relations (and aggregated results in each category). 

It'd be really great if the paper actually provides some insights on we can further improve these entity embeddings according to this categorization. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes to provide a detailed study on the explainability of link prediction (LP) models by utilizing a recent interpretation of word embeddings. More specifically, the authors categorize the relations in KG into three categories (R, S, C) using the correlation between the semantic relation between two words and the geometric relationship between their embeddings. The authors utilize this categorization to provide a better understanding of LP models’ performance through several experiments.

This paper reads well and the results appear sound. I personally believe that works on better understanding KGC models are a very essential direction which is mostly ignored in this field of study. Moreover, the provided experiments support the authors’ intuition and arguments.

As for the drawbacks, I find the technical novelty of the paper is somewhat limited, as the proposed method consists of a mostly straightforward combination of existing methods. Further, I believe this work needs more experimental results and decisive conclusions identifying future directions to achieve better performance on link prediction. My concerns are as follows:

•    I am wondering about the reason for omitting Max/Avg path for two of the relations in WN18RR? Further, the average of 15.2 for the shortest path between entities with “also_see” relation appears to be a mistake?
•    Was there any specific reason in choosing WN18RR and NELL-995 KGs for the experiments?
•    It would be interesting to see the length of paths between entities for train and test data separately. 
•    I suggest providing a statistical significance evaluation for each experiment to better validate the conclusions.
•    I find the provided study in section 4.2 very similar to the triple classification task in KGs. Can you elaborate on the differences and potential advantages of your setting?
•    I am wondering how you identified the “Other True” triples for WN18RR KG in section 4.2 experiments?

On overall, although I find the proposed study very interesting and enlightening, I believe that the paper needs more experimental results and decisive conclusions.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:
The paper attempts to understand the latent structure underlying knowledge graph embedding methods. The work can be seen as an extension of understanding of PMI-based word embedding methods. They categorize knowledge graph relations into three categories based on their relation conditions: Relatedness (R), Specialisation (S), and Context-shift (C). For each category, they evaluate a representative of different types of knowledge graph embedding methods. Through results, they demonstrate that a model’s ability to represent a specific relation type depends on the limitations imposed by the model architecture with respect to satisfying the necessary relation conditions.

Questions:
1. The results in Tables 3 and 4 demonstrate that MuRE is the most effective method for handling different types of relations but then how come its performance on FB15k-237 (.336 MRR) is significantly lower than other methods like TuckER (.358 MRR). Can you provide an explanation?

2. In Section 3.2, the authors list 4 predictions (P1-4). It would be great if authors could provide some more reasoning behind coming with these predictions. 

3. In Section 4.2, it is stated that “ranking based metrics like MRR and hits@k are flawed if entities are related to more than k others”. It would be great if the authors could give an example to make it more clear. 

review_title: Official Blind Review #3

Meta-review: The paper proposes a set of conditions that enable a mapping from word embeddings to relation embeddings in knowledge graphs. Then, using recent results about pointwise mutual information word embeddings, the paper provides insights to the latent space of relations, enabling a categorization of relations of entities in a knowledge graph. Empirical experiments on recent knowledge graph models (TransE, DistMult, TuckER and MuRE) are interpreted in light of the predictions coming from the proposed set of conditions.

The authors responded to reviewer comments well, providing significant updates during the discussion period. Unfortunately, the reviewers did not engage further after their original reviews, and so it is hard to tell whether they agreed that the changes resolved all their questions.

Overall, the paper provides much needed analysis for understanding of the latent space of relations on knowledge graphs. Unfortunately, the original submission did not clearly present the ideas, and it is unclear whether the updated version addresses all the concerns. The paper in its current state is therefore not yet suitable for publication at ICLR.
Decision: Reject