Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new embedding method for knowledge reasoning on knowledge graphs.

main_review: In the introduction the description of the figure 1 could be improved. The understanding of what is depicted in the figure is not clear. From the introduction is not clear the embedding process of the queries and the entities.

In section 3 it is not clear how the input embedding for the entities are computed. Even the adoption of the MLP-Mixer in section 3.3 should be better explained. 

In section 3.4 it is not clear how the answer vector are computed. 

It is not clear the approach used to transform a FOL query to an embedding. Furthermore, how the graph is navigated in order to answer the query is not discussed. 

The obtained results seems to be promising when compared to other approaches. However, a discussion of the results should be included in order to explain the motivation of the improvements.



summary_of_the_review: Concluding, while the results seem to be interesting the clarity of the proposed approach should be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: New model for knowledge graph reasoning over complex logical queries.
Shows performance improvements over state of the art methods.

main_review: The paper describes a new approach for learning and inference in knowledge graphs when the queries have logical structure.
The main idea is to use neural networks to implement different operators to compute the queries over the knowledge graph. In particular, mixer-MLP networks have been used to implement these operations. The experiments using standard datasets show performance improvements over state of the art methods.
The main weakness here was that I found it hard to understand the key ideas in the paper. For instance, I am not sure what is being trained differently for the network to “learn” the logical operations. Sections 3.3 seems to indicate that the training is quite standard for MLP-mixer architectures. So essentially, I found it hard to understand why the results were better than the state of the art. I think a bit more of analysis instead of a brief description (3.3, 3.4) will help strengthen the claims of the paper. Right now the details seemed a little fuzzy to me. Also, the improved results in the 2-Vector Average Approach makes it hard to know if the performance improvements were a result of the embeddings actually learning the logical operations since using additional embeddings seems to be helping improve performance. I think the experiment section also needs to analyze the “why” questions more deeply than is being done currently. Overall, the paper shows good results but is perhaps a bit weak on justification for these good results.

After Author review
The authors did a good job in addressing some clarity concerns and some deeper insights I am raising the score accordingly.

summary_of_the_review: Overall this paper needs a bit more deep analysis and rewriting to back up the experiment results before it can be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to learn the projection, intersection, negation operators necessary for dealing with complex queries with neural networks. The neural networks are built on top of MLP-Mixer architecture and produce good empirical performance.
main_review: *Strengths*:
 + Good empirical performance
 + Good amount of information to reproduce the method given in the main text and appendix
 + Able to deal with negative queries

*Cons*:
 - Poor related work section: Most of the works cited are rather old (e.g. in KGR with logic rules the most recent is 2018, which in machine learning terms is old), there is no comparison of the method proposed and the baselines used, and classic KGR methods that answer queries of type (h,r,?) and complex queries should be clearly distinguished. The related work fails to explain the relation of this work to prior work in a meaningful way.
 - Clarity, the paper should outline in more detail the Graph Query Embedding method that this paper is based on.

*Questions*:
 1. Given more than 2 inputs to the intersection operator, how is the order of recursion decided? A different order will yield a different answer.
 2. What do the colour of the nodes mean in Figure 1?
 3. How is the subgraph (e.g. Figure 1 (C)) extracted from the Knowledge graph?
 4. How are the initial embeddings s_i for each entity and relation learned?
 5. How is the training objective dealing with several possible answers? (I think I know, but the main text should state it clearly)

*Minor*:
 - use \textit{NN} in mathmode to avoid the weird spacing between the two Ns
summary_of_the_review: The paper's empirical performance is solid, the technical novelty lies in making the Graph Query Embedding method work with Neural Networks operators. However, the paper lacks clarity and the related work is poor. Happy to increase my score, should my concerns be addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The article presents a new system for solving multi-hop queries on knowledge graphs. The system allows queries to be performed by handling operators using one- and two-input MLPs. This formulation also makes it possible to handle negative queries. 
The paper also presents a second version of the presented system that exploits an MLP Mixer model, usually used in computer vision.
The system was tested on three standard knowledge graphs: FB15K, FB15k-237 and NELL995. The results obtained are promising because the presented system is able to improve on the established baseline. The results obtained by the BetaE, Q2B and GQE systems on the same datasets were used as the baseline.
main_review: The results obtained are promising and the possibility of managing negation is a plus that few systems can boast of having. The experiments conducted are based on standard datasets, so the comparison with competitors is significant. I found the paper to be mathematically correct and formal enough. The description of the system is sufficiently detailed.
A shortcoming of the paper is the related work section, which presents a list of related systems without significantly discussing the differences, their pros and cons compared to the system presented.


As minor issues, on page one, in the sentence 'directed acyclic graph (DAG) graph which defines ...' the word graph is repeated. While on page 6, sigma is defined as margin, but I think that the margin is gamma while sigma could be the sigmoid function. Is it right?
summary_of_the_review: The results obtained are promising and the possibility of solving negative queries is a strong point in favour.
An appropriate discussion of related work is lacking.
review_title: null

Meta-review: This paper focuses on answering complex logical queries over an incomplete KG and use neural networks to do so flexibly handling multiple operations from FOL. Overall reviews agree that empirical performance is impressive. One reviewer gave a strong accept, one leaning to accept and two leaning to reject. Overall, the reviewers who are leaning to reject had mostly clarity issues which seem to have been addressed by the authors (without response from reviewers). 
Given this I recomment acceptance.
Decision: Accept (Poster)