Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Paper summary**:
This paper tackles the problem of modeling monotonic and non-monotonic reasoning rules in different embeddings spaces, mostly vectors, but under different score functions used by the vector embeddings. The paper first provides the definition of monotonic reasoning, then iterates through a few vector-based embedding and label functions to check if they are able to model the monotonic reasoning part. Then the paper proposed a Relu based model that is able to model both monotonic and non-monotonic reasoning. 


**Strength**:
- This paper did a good theoretical analysis on monotonic and non-monotonic kinds of reasoning for word embeddings. 
- They iterate different embeddings and label functions for Euclidean vector embeddings. For cases that the proposed methods did not work, they give enough counterexamples and provide proof. 
- The paper also includes a Relu based model that is able to model both monotonic and non-monotonic reasoning.

**Weakness / Questions / Suggestions**
- There are many counterexamples in the paper to show that existing embedding and score functions can not model monotonic reasoning. Still, there are limited real-world examples to reflex the effect except for the one example after counterexample 2. If such counterexample does not or seldom exist in real data, the claim and the effect of the counterexample will be weakened. 
- The different combinations of the vector embedding model and label score function are limited. 
- The writing of the paper is a bit hard to follow, probably due to my limited familiarity with propositional logic. But it would be helpful to refine the writing, especially the part at the end of the introduction where monotonic reasoning and non-monotonic reasoning are first brought up. One specific suggestion is that it would be helpful to move the example of “parent, female and mother” right after the definition of the specific embedding and label functions to make it easier to make the connection of why unwanted dependencies are created due to these functions. 
- Echoing the first point, this is a theoretical analysis paper, in which a lot of the examples are hypothetical. Whether such reasoning exists in real data or how much it is there in real data remains a question. In addition, for models where you can model monotonic and non-monotonic reasoning, it remains a question of how easy it is to reach the perfect solution since none experiments are performed, not to mention the Relu function tends to have dead gradient issue which makes training harder. 

review_title: Interesting Theoretical Analysis on Different Models. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ### Summary
This paper provides a thorough analysis of the representational capacity of KB embedding models, analyzing the extent to which they can capture logical dependencies of a knowledge base. The authors consider a large range of common scoring functions, and both monotonic and non-monotonic dependencies. The authors make salient points as to the relevance of this sort of analysis - i.e. if a model is incapable of representing such dependencies then it cannot possible capture them from data - however there are a few limitations. For one, the authors are focusing on a setting where entity embeddings are pooled representations of attribute representations, which is not particularly standard (as acknowledged by the authors). Furthermore, there is a question of the extent to which the counterexamples as presented often manifest in real data, and how poorly a model which compromises to meet its objective would perform on various relevant queries. Both of these are reasonable limitations, however, in that they make the theoretical analysis possible and suggest conclusions that may reasonably apply to real-world data settings.

### Questions

After equation (5): "Here, we need to add this term, which amounts to imposing a Gaussian prior, to ensure that $Emb_{sig}$ is well-defined." Is it trivial to see that this actually ensures $Emb_{sig}$ is well-defined?


### Suggestions

For the counterexamples, it may be clearer to read if they are presented as statements and proofs, for example:

> Counterexample #: The knowledge base $K = \{\text{ (rules) }\}$ cannot be represented using $E_\text{something}$, $L_\text{something}$.
> **Proof:** (details here)

The statement that the set of $K$ which are representable using $Emb_{norm}$ are equivalent for $Lab_{dot}$ and $Lab_{dist}$ (following Counterexample 3) could be made into a proposition.

### Typos

After Counterexample 2: "result" -> "results"

review_title: Excellent paper exploring fundamental questions related to representational capacity

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper looks at entity embeddings and theoretically tries to answer which kinds of semantic dependencies can be modeled by them. Mainly focusing on settings where the entity embeddings are obtained by pooling the embeddings of its attributes (although this is a strong assumption), this paper presents various theoretical limitations of the embedding approaches in learning logical/structured attribute representations with known logical or structured dependencies in practice. The paper shows that some of the most popular embedding models are not able to capture basic logical rules.

I am not an expert in this topic. However, some of the assumptions made in the paper seem too strong to me e.g. that entity embeddings are obtained by pooling the embeddings of its attributes seems like a strong assumption. Most popular embedding approaches do not seem to follow this assumption. I also found the paper hard to follow. The authors can perhaps work a bit on making this paper accessible to those not working exactly on this problems. For example, the assumptions and the results and how it relate to well known entity embedding approaches can be made more clear. The introduction is too long and already presents technical details, so rewriting it might help.
review_title: Decent paper with a good contribution. However, the writing can be significantly improved.

Meta-review: This paper presented a theoretical analysis on the representation capacity of entity embeddings based on their attributes ( i.e. modeling the semantic dependencies between attributes), in particular monotonic and non-monotonic attribute dependencies. In general, the reviewers appreciate the theoretical contribution of this paper. We are glad to accept the paper in the AKBC program, though it is strongly recommended for the authors to improve the readability of the paper according to the suggestions by the reviewers.
Decision: Accept