Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a neural symbolic entangled framework for answering complex logical queries over knowledge graph. The framework firstly represents entities and relations via a neural projection and a symbolic projection. Then the symbolic based logical operator is defined to combine the neural and symbolic projection. Results show the proposed framework outperforms baselines on complex query answering benchmarks.
strengths_and_weaknesses: Pros:
-	The proposed neural symbolic entangled framework is novel, which combines neural representation and symbolic representation of entities and relations.
-	This paper is well written and easy to follow


Cons:
-	The proposed approach does not fully outperform the strongest baseline FuzzQE on benchmark datasets

questions: N/A
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors present a new method to answer complex logical queries on Knowledge Graphs. They include symbolic logic to answer FOL queries and take into account intermediate answers to the queries. By adding the neural symbolic model, the paper aims to tackle some of the main disadvantages of the best-known embedding methods: cascading error and KG incompleteness.
strengths_and_weaknesses: Strengths
- The authors clearly define their methodology and experimentation.
- The paper combines previous technologies in a novel way that achieves better results. The experiments prove the proposed method achieves better results as the queries get more complex.
- The proposed entangling framework can include any other embedding method. Thus, in theory, it adds a performance improvement to all previous KGE methods.
 
Weaknesses
- My main concern is the fact that the entangling framework adds more possible answers to the queries. I would like to see a comparison of how many of the possible answers added by the symbolic logic are actually correct, or else it is highly increasing the number of False Positives.
- According to the results, everything points in the direction that any other embedding method can have similar performance improvements. But I am missing some experimentation in that direction to show how each well-known method is improved by adding this entangling framework.
- Important baselines are missing. On many of the metrics, CQD[a] can achieve better results than that of the proposed model. The recent model GNN-QE[b] outperforms the proposed model almost on all metrics. These two models are closely related to this work but not discussed. 
- Performances on FB15k are missing.

[a] Complex query answering with neural link predictors, ICLR 2021
[b] Neural-Symbolic Models for Logical Queries on Knowledge Graphs, ICML 2022
questions: 1. Have you tested the entangling framework with other embedding methods? Why did you select rotatE?
2. As far as I understand, the entangling method only adds more possible answers to a query, have you checked in which percentage this number increases?
3. Have you manually checked some of the queries you are using for training and testing purposes?

limitations:  
The main advantage of this new entangling framework is that it can be added to any other existing framework. However, how much improvement it actually provides, in general, is still unclear.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a neural-symbolic approach (called ENeSy) for answering FOL-based queries over KGs. The main promise of the proposed approach lies in its ability to alleviate the problem of cascading error as it existed in previous approaches and also the incompleteness of KGs. 

As claimed in the paper, existing query embedding methods handle these complex queries over incomplete KB but they don’t pay any attention to intermediate entities which are important for answering multi-hop queries. The proposed approach ENeSy seems to bridge some of these gaps.

Overall, ENeSy comprises two parts (neural and symbolic)  to represent a given KG.
- For the neural part, it uses rotatE to embed entities and relations into a complex Space ${\mathbb{C}}^k$.
- For the symbolic part, an entity and a relation are encoded as a one-hot vector and an adjacent matrix, respectively. 
- The key operations involved in answering complex queries are _projection, intersection, union,_ and _negation_.  ENeSy proposes an entangled version of each of these operations in Equations (5) and (6). 

The experiments are performed on two datasets (FB15K-237 and NELL-995). 	The focus of the experiments is on answering FOL queries on incomplete KGs so as to evaluate the ability of models to obtain the answers that could not be discovered by traversing KGs. The experiment results (Table 1 and 2) and the insights (Figure 4) established the merits of the proposed ENeSy over the baselines. 
strengths_and_weaknesses: **Originality**

The idea of fusing neural and symbolic representations of the answer entities to cope with the KG incompletion is certainly novel. Neural and symbolic representations seem to be helping each other and thereby boosting the overall performance. 


**Quality**

The paper is really very well written. The entanglement ideas are novel. The experimentation evaluation plan laid out in lines # 205-217 is very clearly written and makes perfect sense.  The experiments seem to be convincing in terms of proving the merits of the proposed approach. The only reservation I have is that the experiments are performed only on two datasets. Authors could have included one more dataset. 


**Clarity**

By and large paper is easy to read and follow through. Below are some questions regarding the clarity of the matter.
- In line # 116, the symbol S’ seems a bit inconsistent. 
- In line # 139, the spelling of Hadamard is incorrect.
- In lines 162 and 168, why the word _possibility_ is used and not the word _probability_?
- How does MLP(.) function look like in equation (5)? Does it map a vector to a scalar? If it is a vector then I am not sure how this vector is getting multiplied with the other two vectors present in equation (5)? Is it an inner product between two vectors? Notations are a bit confusing as there is no transpose sign. 
- In equation (7), should it be $v_q$ instead of $v$?
- Caption of Table 1 seems to have typos. Please read it carefully once again and fix any typos.
- Just to make the paper self-contained, better explain the convention used behind assigning titles to the computation graphs in Figure 3. For example, in the case of 2i, why can’t we call it 2p2i. or why “inp” can’t be titled as “pin”. If need be, add this in the appendix. 
- In Tables 1 and 2, you should also tag the count of each query type in your test set over which these numbers are reported.

**Significance**

If we were to believe the numbers shown in Tables 1 and 2 then it clearly proves a significant improvement over the baselines methods. However, as I mentioned before, it would have been nicer to run experiments on one more dataset. Nonetheless, what is presented is still impressive.
questions: - In lines 162 and 168, why the word _possibility_ is used and not the word _probability_?
- How does MLP(.) function look like in equation (5)? Does it map a vector to a scalar? If it is a vector then I am not sure how this vector is getting multiplied with the other two vectors present in equation (5)? Is it an inner product between two vectors? Notations are a bit confusing as there is no transpose sign. 
- In equation (7), should it be $v_q$ instead of $v$?
- For baselines, did you recompute the numbers or picked from the existing papers? If picked from the existing papers, whether the existing papers followed the same evaluation steps as mentioned in lines # 205-217? Because, otherwise, it will not be an apple-to-apple comparison.

limitations: There is not much of a discussion on the limitations of the proposed approach. I would have loved to see some discussion on the memory footprints and time complexity-related issues and how they compare with baselines. Also, it may be worth discussing if there is any class of queries within FOL that are not being handled properly by your approach? By looking at Figure 4(b), it looks like negation queries are relatively hard to deal with. You can have some discussion about that as well.
review_title: null

Meta-review: metareview: This paper proposes a neural-symbolic approach (ENeSy) for answering FOL-based queries over KGs. The main promise of the proposed approach lies in its ability to alleviate the problem of cascading error as it existed in previous approaches and also the incompleteness of KGs. The experiments are performed on two datasets (FB15K-237 and NELL-995) which established the merits of the proposed ENeSy over the baselines. 

The reviewers are generally happy with the clarity and substance of the work. There are some suggested baselines in "3t6R"'s review which (to my understanding) are addressed in the authors' response.   The experiments on computation speed (in response to "7Bea") are also informative.  I hope all of these will be reflected in their revised draft.  Lastly, I hope the authors release the scripts necessary for reproducing their experiments. 
confidence: Less certain
Decision: Accept