Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors approach the problem of representing knowledge graphs from a top-down perspective, in which they argue for certain fundamental desiderata for hyper-relations (composition, inversion) followed by properties that define a mathematical group.

I found the paper extremely difficult to follow. As defined in Eq. 1, knowledge graph embeddings are a model family with a choice of domain for the entity and relation, and a choice of how that relation operates on a head entity. This means one can devise arbitrary properties and restrictions on that family. It's not clear to me what motivates selecting a (abelian) group, where inversion, closure, identity, associativity, and commutativity are demanded to be properties of knowledge graph embedding models. This seems more a definition of what models they consider, rather than a novel insight about knowledge graph embedding models itself (the authors claim "we proved for the first time the emergence of a group definition in the KG representation learning", which seems hard to wrap one head's around).

Given this overarching family of models, the authors proceed to identify existing models as certain choices of that family.  I see little use in inventing this abstraction as the authors do not show any practical insights, or interesting theoretical analysis that comes from this higher-level abstraction.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper start merely by studying the graph reconstruction problem and prove that the intrinsic structure of this task itself automatically produces the complete definition of groups. it seems to be a novel result. Based on this result, one could construct embedding models that naturally accommodate all possible local graph patterns, and the paper also shows a few simulations.

My main concern is that, while the focus on this work is the theoretical finding, there is no rigorous statement of it as a theorem. As a result, I am not exactly sure what the proofs in the appendix is trying to show. In addition, the proofs seems to be very trivial.

For the algorithm section, I feel that it is also lacking in the sense that there is still no automatic way to choose which group to embed. It is also unclear what is the purpose of the simulation section. While it says "As theoretically analyzed in Section 3.2, and empirically shown above, continuous nonabelian groups are more reasonable choices for general tasks", the advantage of continuous nonabelian groups are not so significant in the tables. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I would like to thank the authors for the detailed rebuttal. 

My concern is generalizing something that is not correctly describing the object we should model (KGs).

For instance, "all properties mentioned in the paper, including closure, identity, inversion, and associativity, are desired properties of relation embedding spaces, rather than existing properties of specific knowledge graphs. " => should we not care about the actual relations of knowledge graphs? Should we not build a theory that closely matches the properties of the actual objects we are modeling? The argument seems to be: forget the actual knowledge graph, here is a generalization of existing KGE methods. Even the proof provided was unrelated to my question: I gave an example: can you list all relations in FB15K and prove that they fit the group axioms? If they don't fit, what is the advantage of modeling the relations as a group? What is the representation error this causes? 


The authors misunderstood my comment about (Bloem-Reddy and Teh 2019). "Therefore the previous works mentioned by the reviewer above are irrelevant." => this is a weird statement. Permutation invariance is relevant to all objects that are graphs. I recommend reading the long history relating graph models and exchangeability (Persi Diaconis has a good overview). I was just mentioning that I don't see why KGs should have another associated permutation group besides permutation invariance. I would also recommend following the emerging literature in graph representation learning using group theory, of which KGs are but a special case.

"The (entity) manifold, however, means that the relations no longer form a group. I see no easy fix." => I clarified it, since I was talking about the entity embedding.

I did not see a revised version of the manuscript.


I will raise my score because the paper could be published as a niche paper. We generalize KGE but we acknowledge its shortcomings: And here is a way to measure the embedding error of modeling the relations as a group, when they are not actually a group. 

--------------

The paper is well written and an interesting read. I think it complements well the existing literature. 

Unfortunately, I think the paper overstate its claims. It is clear that permutation groups are the natural language of all graphs (Kondor and Trivedi, 2018) and (Bloem-Reddy and Teh, 2019). It is less clear that knowledge graphs also comply with another set of group axioms. Why would the set of relations be closed? Say, Owns \cdot Spouse_of = ? . I cannot see why and how this is necessary. Must all the relations be closed? I understand why this is true for KGE but this is not true for KGs, which only shows that KGEs may not be the right method to represent KGs moving forward. I see no easy fix. 

The rest of the paper is straightforward, just applying the definitions of groups. I found the classification of different methods interesting, and should be made more clear in the experiments. 

The experimental results are interesting but their significance is unclear. Please add standard deviations to all experiments. We cannot have a sense of the significance of the results without knowing how many runs were executed, how they were executed (e.g., k-fold cross-validation, bootstrapping), and their standard deviation.

I dispute that the paper proves that the definition of groups emerges purely from the nature of the knowledge graph embedding (KGE) task. I would actually say the opposite, it is clear to me that for most KGs, their relations do not fit the definition of a group. I am open to be (formally) proven wrong. E.g., can you list all relations in FB15K and show that they fit the group axioms?

Fixing the paper: Maybe the paper could be rewritten, constrained to Euclidean spaces? Then prove (formally) that the group axioms are a sufficient(?) and necessary(?) condition for such embeddings? 

Minor issues:
honestly could not make sense of the abstract:
The sentence choices
-  "which suggests that a group-based embedding framework is essential for model design" 
- "Our theoretical analysis explores merely the intrinsic property of the embedding problem itself without introducing extra design"
in the abstract are very strange. It means nothing to a reader at that point. What a model design? What is an extra design? A modeling assumption? A model prior? 

"Using the proposed framework, one could construct embedding models that naturally accommodate all possible local graph patterns, which are necessary for reproducing a complete graph from atomic knowledge triplets" => what is an "atomic knowledge triplet"? Why are they necessary to reproduce a complete graph? This is all very confusing.

" contradicts to the entity regularization" =>  contradicts the entity regularization

References:
Kondor, R. and Trivedi, S., 2018. On the generalization of equivariance and convolution in neural networks to the action of compact groups. ICML
Bloem-Reddy, B. and Teh, Y.W., 2019. Probabilistic symmetry and invariant neural networks. arXiv:1901.06082.

review_title: Official Blind Review #2

Meta-review: This paper presents a rigorous mathematical framework for knowledge graph embedding. The paper received 3 reviews. R1 recommends Weak Reject based on concerns about the contributions of the paper; the authors, in their response, indicate that R1 may have been confused about what the contributions were meant to be. R2 initially recommended Reject, based on concerns that the paper was overselling its claims, and on the clarity and quality of writing. After the author response, R2 raised their score to Weak Reject but still felt that their main concerns had gone unanswered, and in particular that the authors seemed unwilling to tone down their claims. R3 recommends Weak Reject, indicating that they found the paper difficult to follow and gave some specific technical concerns. The authors, in their response, express confusion about R3's comments and suggest that R3 also did not understand the paper. However, in light of these unanimous Weak Reject reviews, we cannot recommend acceptance at this time.  We understand that the authors may feel that some reviewers did not properly understand or appreciate the contribution, but all three reviewers are researchers working at highly-ranked institutions and thus are fairly representative of the attendees of ICLR; we hope that their points of confusion and concern, as reflected in their reviews, will help authors to clarify a revision of the paper for another venue. 


Decision: Reject