Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to establish a theoretical basis for geometric properties of knowledge graph relations and embedded entities by comparing knowledge graph embeddings with word embeddings. Using the insight that the semantic properties of PMI-based word embeddings manifest as linear geometric relationships, they view and compare the relationship embeddings derived from different knowledge graph embedding schemes in this way. 
The analysis claims to show that when the KG architecture conforms to the presented relation types and conditions (divided into similarity, relatedness, and context shift types), it has better performance of link prediction for that embedding scheme. 

The empirical evaluation focuses on comparison of 4 embedding schemes that have linear transformation score functions (additive, multiplicative, both) on WN18RR and NELL-995 relations for link prediction on several examples of the relation types.

Overall, the paper is well-motivated, cites relevant literature in the theory behind word embeddings, and is generally clearly written. It has a useful proposal for the types and conditions of the three relation types and clear hypothesis for the performance of knowledge graph relation transformations that have certain properties.

However, the empirical evaluation does not seem to completely support the claims. TransE is obviously lower performing across relations, but MureI seems quite close in most cases to models that involve multiplacative relatedness, so it’s not obvious to me that MureI performs worse. Further, the summery suggests that DistMult is preferable for type R, but MuRe appears to do equally well or better on most cases, thus it’s not clear under what circumstances (what dataset dependent factor) would point to not choosing MuRe. 
I would expect to see a starker contrast between the performance of the different models per claim type to support the dataset dependent statement. Perhaps another experimental setting, like comparison on non-linear transformation, or other examples, would help support that claim. 
review_title: Clear exposition and hypothesis but underwhelming empirical validation

final_score: rating: 7: Good paper, accept
review_content: Recent works toward the understanding of word embeddings can explain how semantic word relationships, such as similarity, analogy and paraphrasing are encoded as low-rank projections of high dimensional vectors of co-occurrence statistics (Allen et al., 2019). Thus, the semantic relationships correspond to linear relationships of word embeddings. This paper builds on this understanding of (PMI-based) word embeddings aiming at the task of understanding the latent structure of low-rank knowledge graph representations. The authors draw a parallel between the embeddings of knowledge graphs and words under the premise that fundamentally the same structure of relations is captured in different ways. Strong evidence to this premise is provided by starting at encoded semantic relations of word embeddings generalizing them to three types (R,S,C) of knowledge graph relations. The authors analyse the performance of different state-of-the-art knowledge graph models and identify the best performing model per relation type. While a multiplicative model performs best for R-relations (highly related), an additive-multiplicative model should be used for S- (specialisation) or C-type (context-shift) relations. These results correspond to the predictions made beforehand and the theoretically derived loss functions based on the respective conditions of each relation type.

Pros:
•	The paper is technically sound, well written and organized and free of typographical errors
•	It focusses on the timely and interesting problem of understanding the latent structure of knowledge graph representations
•	The key strength of the paper is the idea to categorize relations between entities based on the geometric properties of relation representations of word embeddings
•	The theoretically derived loss functions are a strong evidence to why different kind of knowledge graph models perform unequally on the defined relation types

Minor comments:
•	The relation types in the text (four bullet points, p. 4) should perhaps mirror one-to-one the types in figure 1
•	Context-shift relations are explained as "subject to synonyms", a specialisation should perhaps be explained as hypernyms and context shift e.g. as meronyms

review_title: This paper contributes to understanding the latent structure of low-rank knowledge graph representations. The authors draw a parallel between the embeddings of knowledge graphs and words, in the sense that they capture the same structure of relations.

final_score: rating: 7: Good paper, accept
review_content: Summary and Contributions

The authors study the latent semantic properties of word representation models by categorising relations between entities.  The goal is to show that word embeddings and knowledge graph representations learn a common latent structure even if both types of models have different learning objectives.  The main contributions are the mapping of relations between subjects to object word embeddings, categorisation of such relations, and evaluation of the state-of-the-art knowledge graph representations. The study shows that knowledge representation models follow the defined relation conditions.

Strengths

- Clear description of background knowledge needed to understand the proposed study.  
- The authors perform a comprehensive comparison across different knowledge graph representations.
- The findings show that there is a connection in the lower dimensional space between word and graph representations.

Weaknesses

- It is not clear which particular framework is used to define the hierarchy of knowledge graph relations.
- It is not discussed the possible benefit of the learned latent structure of knowledge graph models for the performance on downstream tasks, e.g.  text classification, or natural language inference. 

Questions to the Authors

- How the main contributions relate to the experiments P1 and P2?
- Could you elaborate on how you define the hierarchy knowledge graph relations.
- Which is the relation between the used knowledge graph categorisation and the related work?

review_title: Interpreting Knowledge Graph Relation Representation From Word Embeddings

final_score: rating: 7: Good paper, accept
review_content: Based on PMI word embedding, the authors categorize the knowledge graph relations into three types, which serve as the foundation of knowledge analysis. This paper is not well-motived but presents the methodology, well. However, nothing in this paper surprised me, because this seems like a ````''regular'' research in this field.

Major Concerns:
1. The motivation is not crystally clear. I suggest the authors explain the motivation in detail, e.g., why different embedding are fundamentally the same (strong claim but no evidence).  I suppose the author would like to align the structure of linguistic semantics and knowledge semantics. However, there is no sufficient evidence for me to accept this alignment. Can you give me some examples or strong evidence to claim the joint learning is not incremental. 

2. This paper is indeed fine, except for motivation. Good presentation, the clear methodology, and promising results. However, there are plenty of researches to join the textual info and knowledge info together, which makes this paper not very novel. Besides, based on PMI embedding and relation categorization are still very regular in the field. Thus, I doubt the novelty of this paper. However, compared to these papers published in this field, I think this paper can be accepted. This is the reason why I rate 7 but I still got unsatisfactory about the novelty.

Minor Concerns:
1. Fig.1: Please specify the legend clearly. 
2. Table 2: For more beginning readers, please revise the examples to human-readable.

Discussion:

I think the categorization shall benefit the performance most when the difficulties between relation categories are balanced. Does this sound like a correct claim? because in your experiment, I found R is the easiest while S is hardest, and the difficulty different for the two types is large. I don't judge this question. I just want to provide a new idea for your paper to improve.

review_title: Nothing surprised me.

Meta-review: This paper extends the recent theoretical understanding on geometric properties for word embeddings to relations and entities of knowledge graph. It categorizes relations into different types and derive requirements for their representations. Empirically they experiment several graph embedding approaches and show that when the loss function is aligned with the requirement of the relation type, we can achieve better performance.  The reviewers generally find the paper to be solid, well executed and provides useful insights. The authors are encouraged to strengthen the discussion of the motivation of this work, and improve the presentation based on reviewers' comments. 
Decision: Accept (Poster)