Neural Methods for Logical Reasoning over Knowledge Graphs
Authors: Alfonso Amayuelas
Shuai Zhang
Xi Susie Rao
Ce Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: Knowledge Graphs
Knowledge Graph Reasoning
Graph Mining
Data Mining
Machine Learning
Artificial Intelligence
Deep Learning

Abstract: Reasoning is a fundamental problem for computers and deeply studied in Artificial Intelligence. In this paper, we specifically focus on answering multi-hop logical queries on Knowledge Graphs (KGs). This is a complicated task because, in real world scenarios, the graphs tend to be large and incomplete. Most previous works have been unable to create models that accept full First-Order Logical (FOL) queries, which includes negative queries, and have only been able to process a limited set of query structures. Additionally, most methods present logic operators that can only perform the logical operation they are made for. We introduce a set of models that use Neural Networks to create one-point vector embeddings to answer the queries. The versatility of neural networks allows the framework to  handle FOL queries with Conjunction, Disjunction and Negation operators. We demonstrate experimentally the performance of our models through extensive experimentation on well-known benchmarking datasets. Besides having more versatile operators, the models achieve a 10% relative increase over best performing state of the art and more than 30% over the original method based on single-point vector embeddings.

Published as a conference paper at ICLR 2022
NEURAL METHODS FOR LOGICAL REASONING OVER KNOWLEDGE GRAPHS
Alfonso Amayuelas Shuai Zhang', Susie Xi Rao & Ce Zhang EEPL
ETH Zurich
al fonso. . amayue asBa aas@almmin......mmmmmmm.mmm..mmm epffl.c
{shuazhang, raox, ce. .e.zaang 8 wwinff.thz.c.m
ABSTRACT
Reasoning is fundamental problem for computers and deeply studied in Arti- ficial Intelligence. In this paper, we specifically focus on answering multi-hop logical queries Knowledge Graphs (KGs). This is complicated task because, in real-world scenarios, the graphs tend to be large and incomplete. Most previ- ous works have been unable create models that accept full First-Order Logical (FOL) queries, which include negative queries, and have only been able to process limited set of query structures. Additionally, most methods present logic opera- tors that can only perform the logical operation they are made for. We introduce a set of models that use Neural Networks to create one-point vector embeddings to answer the queries. The versatility of neural networks allows the framework handle FOL queries with Conjunction (A), Disjunction (V) and Negation (-) operators. We demonstrate experimentally the performance of our model through extensive experimentation on well-known benchmarking datasets. Besides having more versatile operators, the models achieve 10% relative increase the best performing state the art and more than 30% over the original method based on single-point vector embeddings.
INTRODUCTION
Knowledge graphs (KGs) are type of data structure that can capture many kinds of relationships
between entities (e.g.: Moscow cityIn Russia) and have been popularized since the creation of the semantic web or its introduction into Google's search engine. They can contain many kinds of different information, and they be widely used in
systems, search engines, and recommender systems (Palumbo et al. 2017 Xiong et al. 2017a).
Reasoning is fundamental skill of human brains. For example, we can infer new knowledge based on known facts and logic rules, and discern patterns/relationshiis to make sense of seemingly unre- lated information. It is a multidisciplinary topic and being studied in psychology, neuroscience, and artificial intelligence (Fagin et al.) 2003). The ability to reason about the relations between objects central to generally intelligent behavior. We can define reasoning as the process of infer- ring new knowledge based known facts and logic rules. Knowledge graphs are structure used for storing many kinds of information, therefore the ability to answer complex queries and extract answers that are not directly encoded in the graph are of high interest the AI community.
To answer complex queries, the model receives query divided in logical statements. A full First- Order Logic (FOL) is necessary to process a wider range of queries, which includes negative queries. FOL includes the following logical operators: Existential (3), Conjunction (A), Disjunction (V), and Negation (-). The power of representation of our logical framework is the key to process complex queries. However, most frameworks have only been able to process Existential Positive First-Order Logic (EPFO), which means that negative queries cannot be processed.
For example, One could ask knowledge graph containing drugs and side effects the following ques- tion: " "What drug be used treat pneumonia and does not cause drowsiness?". The first step to answer such a query is to translate it into logical statements: q = VV Treat Pneumonia Vo)
Source code available http:///ahht...ooommmyyaaAaaaaaa

---

Published as a conference paper at ICLR 2022
q= V," V CitizenOf( Brazil, V) A
Winner(Ballon d'Or, V) A Team(V, V,)
Brazil
Projection
retrieved answers
Retrieved wrong answer
Intersection
Projection
Projection
Intersection
Projection
Projection
F.C. Barcelona
Ballon
Intersection
A.C. Milan
d'Or
Retrieved correct answers
Ballon
d'Or
Answer embedding
(A)
COMPUTATION GRAPH
(B) VECTOR SPACE
Figure 1: MLP Framework for KG Reasoning, Representation of a sample query: "List the teams
where Brazilian football players who were awarded a Ballon d'Or played". (A) Query represented by its logical statements and dependency graph. (B) 2D Representation of the answer entities in one-point vector space used by the reasoning framework.
Cause Drowwsinss,, V.J. Once the query is divided into logical statements, obtain the com- putation graph, a directed acyelic graph (DAG) which defines the order of operations. Afterwards, start traversing the graph. However, many real-world graphs are incomplete and therefore traversing them becomes very hard and even computationally impossible. There are many possible answer entities, and requires modeling sets of entities. As such, embedding methods become good solution to answer these queries. Previous works (Hamilton et al.) 2018 Ren et al.) 22200) Ren & Leskovee 2020) have created methods for embedding the query and the graph into vector space. The idea of graph embeddings reduces the problem to simply using nearest-neighht search to find the answers, without paying attention to the intermediate results.
The embedding approach solves many of the problems of query-answering knowledge graphs. In theory. we could answer the queries just by traversing the graph. In practice, graphs are large and incomplete, and answering arbitrary logical queries becomes complicated task. The graph incompleteness means that traversing its edges would not provide the correct answers.
This work aims to create some models that allow complex queries and extract the correct answers from large incomplete knowledge graphs. To this end, we present set of models based on Neural Networks that embed the query and the entities into one-point vector space. Then, it computes the distance between the query and the entities to rank the answers according to the likelihood answer the query. We use the versatility of Neural Networks to create the operators needed to process FOL. queries.
We conduct experiments using well-known datasets for KG Reasoning: FB15k, FB15-237, and NELL. The experiments show that our models can effectively answer FOL and provide noticeable improvement when compared with the sate-of-the-an baselines. Our models provide a relative improvement of 5% to 10% to the latest state-of-ar -frt method and about 30% to 40% when compared with the method that uses the same idea of one-point vector space embeddings (Hamilton et al.) 2018).
The main contributions of this work are summarized as: (1). New embedding- methods for logical reasoning over knowledge graphs: two new models, plus variants, for KG Reasoning. These methods embed the query and the entities in the same vector space with single-point vectors. Implementing the logical operators with neural networks provides versatility create any operator with virtually the same architecture. (2). Improved performance over the current state of the art. Experimental results show that the models presented this paper outperform the selected baselines: Graph Query Embeddings (GQE) (Hamilton al., 2018). Query2Box (Q2B) (Ren al.) 2020). and BetaE (Ren & Leskovec 2020). (3). Handling of negative queries. Modelling queries with negation has been open question in KG Reasoning until recently. BetaE (Ren & Leskovec, 2020) introduced the first method able to do so. This work takes advantages of the good relationship
inference capabilities of Neural Networks and uses them create the negation operator.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Traditional tasks on graphs include Link Prediction Liben-NNowlll Kleinberg 2007), Knowledge Base Completion (Wang et al.) 2015). or basic Query-Answering (one-hop). They are all different versions of the same problem: Is link (h,r.1) the KG? or Is an answer to query (h,r,)?, where only a variable is missing. However, we face more complicated problem, known as Knowledge Graph Reasoning, that may involve several unobserved edges or nodes over massive and incomplete KGs. In this case, queries can be path queries, conjunctive queries, disjunctive queries combination of them. A formal definition of KG Reasoning can be found in Chen et al. (2020). as stated in Definition
Definition 2.1 (Reasoning over knowledge graphs). Defining knowledge graph as: (E, R, T), where E, T represent the set of entities, R the set of relations, and the edges in R. link two nodes to form triple as r,, 1) TT Then, reasoning over KG is defined creating triplet that does not exist in the original KG, g' = {(h, r, h) E & R, E T, (h, r, $ 9}
Most related to our work embedding approaches for multi-hop queries over KGs: (Hamilton ctt al., 2018), (Ren et al.] (020). (Ren & Leskoved 2020) and (Das et al. 2016), as well as models for question answering Yasunaga et al.) (021). (Feng al. 2020). The main differences with these methods rely on the ability to handle full First-Order Logical Queries and using Neural Networks define all logical operators, including the projection. We also deliver more extensive range of network implementations.
On a broader outlook, we identify a series of works that aim to solve Knowledge Graph Reasoning with several different techniques, such as Attention Mechanisms (Wang et al. (2018), Reinforcement Learning like DeepPath (Xiong et al. 2017b) or DIVA (Chen et al. 2018), 8), or Neural Logic Networks (Shi et al. 2020), (Qu & Tang 2019)
MODELS
Both models presented here follow the idea behind Graph Query Embedding - GQE (Hamilton et al., 2018):
Learning embed the queries into low dimensional space. Our models differ from it in the point that logical query operations are represented by geometric operators. In our case, we do not follow the direct geometric sense and these operators are all represented by Neural Networks, instead of just the Intersection operator in GQE. Similarly, however, the operators are jointly optimized with the node embeddings to find the optimal representation.
In order answer query, the system receives query q, represented DAG, where the nodes the entities and the edges the relationships. Starting with the embeddings e....., es,, of its anchor nodes and apply the logical operations represented by the edges finally obtain an embedding q of the query (Guu et al. 2015).
3.1 FORMAL PROBLEM DEFINITION
A Knowledge Graph (g) is heterogeneous graph with set of entities nodes (V) and set of relations edges (R). In heterogeneous graphs, there can be different kinds of relations, which are defined as binary functions r: VX V + {True, False} that connect two entities with directed edge. The goal to answer First-Order Logical (FOL) Queries. We can define them as follows:
Definition 3.1 (First-Order Logical Queries). A first-order logical query is formed by an anchor entity set V. C V, an unknown target variable V, and series of existentially quantified variables Vi,.., Vg, In its disjunctive normal form (DNF), it written as a disjunction conjunctions:
(1) where e; represents conjunctive query of one several literals: 6 ea A e2 A... Cim- And the literals represent relation or its negation: eij = r(v,, v,) or -e(e,, Uz) where Us uy are entities and r e R.
q(V3] = V/- ...3...... V/. V..
The entity embeddings are initialized to zero and later learned as part of the training process, along with the operators' weights.
3

---

Published as a conference paper at ICLR 2022
Entity 2
Embedded vector
Entity
Entity 1
Embedded vector
(a) Representation MLP for 2 input operators: Pro- jection, Intersection.
(b) Representation of MLP for input operator:
Negation.
Figure 2: Multi-Later Perceptron Model (MLP) Network Architecture.
Computation Graph. The Computation Graph can be defined as the Direct Acyclic Graph (DAG) where the nodes correspond to embeddings and the edges represent the logical operations. The computation graph can be derived from query by representing the relations as projections, inter- sections as merges and negation as complement. This graph shows the order of operations to answer the queries. Each branch can be computed independently and then merged until the sink node is reached, Each node represents a point in the embedding space and each edge represents a logical operation, computed via Neural Network in our case. The representation of FOL as compu- tation graph can be seen as a heterogeneous tree where each leaf node corresponds to the anchor entities and the root is the final target variable, which is set of entities. The logical operations corresponding to the edges are defined below:
Projection. Given an entity E V and a relation type E R. It aims to return the set of adjacent entities with that relation. Being P, iv, r) the set of adjacent entities through r, we define the projection as: P,i(Us, r) (v' E V (v, v') True).
Intersection. The intersection can be defined as: (w:) = Mi=1%
Negation. It calculates the complement of set of entities T C V: N(T) = = where the set can either be embedding corresponding to an entity or another embedding in between which represents set of them.
A Union operation is unnecessary, it will be later discussed in Sections 3.5 Query2Box (Ren et al. 2020) shows that union operator becomes intractable in distance-based metrics.
3.2 MULTI-LAYER PERCEPTRON MODEL (MLP)
Based on the good results of Neural Tensor Networks (NTN) (Socher et al. 2013) for knowledge base completion, we have extended similar approach multi-hop reasoning.
We introduce three logical operators compute the queries. Each of them is represented by simple neural network: a multilayer perceptron. Each perceptron contains feed-forward network: a linear layer plus ReLu rectifier. The number of layers remains show what the model looks like.
seree-parammter. Figures 2a and 2b
Neural operators. We define the operators with a multi-layer perceptron. The model will take as an input the embedding representation of the input entities and will return approximate embedding of the answer. Defining the operators with a Neural Network has the advantage of generalization. Thus, distinguish between 2-input operators, Projection and Intersection and 1-input operator, Negation.
2-innut operator (Figure Projection P. and Intersection The operator is composed by multi-layer perceptron that takes 2 inputs and returns 1 as embedding the output. The training process will make the networks learn the weights to represent each operation. Equation expressees formally:

---

Published as a conference paper at ICLR 2022
HLL--Mixer
Fully-connected
Perpatcch fully-commettdd
N Mixer Layer
Average Pooling
connected
Fully
GELU
Fully-coneeceed
Mixer
Patches
Layer
Layer norm
Dimensions Entiy:
Patches
Dimensions lonnss
MLP1
Layer norm
MLP1
Endiy
MLP1
MLP1
Figure 3: MI.e-Mxxe. At the top, we show the block diagram the MLP-Mixer Architecture. is formed by per-patch fully connected module, N Mixer Modules, an average pooling and last fully connected module. The bottom figure shows the Mixer Module, which contains one channel- mixing MLP, each consisting of 2 fully-connected layers and ReLu nonlinearity. It also includes skip-connections, dropout and layer norm on the channels.
P(s,, rj) = (N,,s,, Vs; E S,Vs, E T(s1, 85) NNv 8j), Vs,, 8j E
(2)
where E S is an embedding in the vector space S, ry E R is relation and NN₂ is multi-layer perceptron with layers.
The intersection can take more than two entities as an input, for instance the 3i query structure. In this case we do a recursive operation, we use the result of the previous intersection to compute the next one.
Negation N. The goal of this operator represent the negation of set of entities. Following the same neural network approach, we can represent it as in the equation below (Equation 3).
I-imput operator (Figure 2bh:
((:)) = (NN=(s:), Vs; E
where 8; S vector the embedding space, can an entity or the result previous operation. NN₃ is multi-layer perceptron with & layers and the same number of inputs outputs.
3.3 MUL.TI-LAYER PERCEPTRON MIXER MODEL. (MLP P-X-ER)
The MLP-Mixer (Tolstikhin et al. (021 Neural Architecture originally built for computer vision applications, which achieves competitive results when compared to Convolutional Neural Networks (CNNs) and Attention-baser networks.
The MLP-Mixer is a model based exclusively on multilayer perceptrons (MLPs). It contains two types of layers: (1) one with MLPs applied independently to patches and (2) another one with MLPs applied across patches. Fi a diagram of the architecture.
Mixer operators. We the same procedure in the MLP model. We full MLP-Mixer block to train each of the 2 operators with inputs: projection and intersection. Since negation only has input, the architecture cannot be accommodated for this use so far.
2- 2-input operator (Figure 2a Represents Projection P or Intersection I with MLP-Mixer architec- ture.
P(sGry) MLP mix (85, Ty). Vs, S, [(s,, (()),;)) ) = MLP min (St, 9;), Vsi, 8j E S
(4)
where ML represent the mixer architecture, s; an embedding in the entity vector space S; and ry R. relation.

---

Published as a conference paper at ICLR 2022
3.4 TRAINING OBJECTIVE, DISTANCE AND INFERENCE
Training Objective. The goal is to jointly train the logical operators and the node embeddings, which are learning parameters that are initialized randomly. Our training objective is to minimize the distance between the query and the query vector, while maximizing the distance from the query to incorrect random entities, which can be done via negative samples. Equation 5 expresses this training objective in mathematical terms.
& - Σ k j=l
C = - log o( Dist(e; g))
log o Dist(r); -7)
(5)
where q is the query, E [g] is an answer of query (the positive sample); v/ & [q] represents a random negative sample and y refers to the margin. Both, the margin y and the number of negative samples k remain as hyperparameters of the model.
Distance measure. When defining the training objective, we still need to specify the distance mea- sure to compare the entity vectors. Unlike in previous works, we do not need measure that com- pares between boxes or distributions. The Euclidean distance is enough for this purpose, as it calcu- lates the distance between two points in a Euclidean space: Dist(v, q) |0 q/-
Inference. Each operator provides an embedding. Following the query's computation graph, obtain the final answer embedding (or query representation). Then, all entities are ranked according to the distance value of this embedding to all entity embeddings via neig-ne search in constant time using Locality Sensitivity Hashing (Indyk & Motwani 1998).
3.5 DISCUSSION ON ANSWERING FOL QUERIES
The aim of this work to answer wide set of logical queries, specifically, be able to answer first-order logical queries (FOL), which includes: conjunctive A, disjunctive V, existential 3 and negation operations. Notice that we will not consider universal quantification (V) as does not apply to real-world knowledge graphs since entity will ever be connected to all other entities in the graph.
Theorem 1 in Query2Box shows that any embedding- based method that retrieves entities using a distance-based method is not able to handle arbitrary disjunctive queries. To overcome this problem, they transformed the queries into its Disjunctive Normal Form (DNF). By doing so, the disjunction is placed the end of the computational graph and can be easily aggregated. The transformed computational graphs are equivalent to answering N conjunctive queries. N is meant to be small in practice, and all the N computations can be parallelized. As expressed in (Davey & Priestley] 2002), all First-Order Logical Queries can be transformed into its DNF form. We refer readers to (Ren al. 2020) to understand the transformation process.
4 EXPERIMENTS AND RESULTS
4.1 DATASETS
We perform experiments on three standard datasets in KG benchmarks. These are the same datasets used in Query2Box (Ren et al. 2020) and BetaE (Ren & Leskovec 2020): FB15k (Bordes et al., 2013 FB15k-237 (Toutanova al. 2015) and NELL999 (Xiong et al. (0176).
In the experiments, split into training, test and validation sets. After augmenting the KG to also include inverse relations and double the number of edges in the graph, we effectively create graphs: G trin for training; g valid, which contains Gunin plus the validation edges; and Geest which contains G valid and the test edges. Some statistics about the datasets can be found in Appendix A
the standard evaluation scheme for Knowledge Graphs, where edges
4.2 QUERY/ANSWER GENERATION
To obtain the queries from the datasets and its ground truth answers, consider the 9 query basic structures from Query2box (Ren et al. 2020). shown in Figure 4 The training, validation, and test
graphs were created as: train C 9 val test: therefore the generated queries are also: [4] tain C

---

Published as a conference paper at ICLR 2022
Training Evaluation Queries
Further Evaluation Queries
o 0+0+0 2p
1p
3p
lp
pi
2u
up
Figure 4: Training and evaluation queries represented with their graphical structures and abbrevia- tion of their computation graphs. We consistently use the following nomenclature: p projection, i intersection, n negation, and union.
Additional Training Evaluation Querles
Transformation to negative queries
2in
inp
21
2in
ip
inp
pni
pin
pi
pni
pin
Figure 5: Queries with negation used in the experiment used for both, training and testing. On the right side, we show the transformation process from the original queries to its negative structure.
14]l val S [gg] seu. Thus, we evaluate and tune hyperparameters on [q] val \ [g] train and report the results on [g] test [g] val. We always evaluate on queries and entities that were not part of the already seen dataset used before.
For the experiments, we have used the train/valid/test set of queries-answers used for training and evaluating BetaE (Ren & Leskovec 2020). This query-generation system differs from the original in the fact that limits the number of possible answers to a specific threshold since some queries in Query2box had above 5000 answers, which is unrealistic. More information about how queries are created in BetaE can be found in Appendix
To include FOL queries, BetaE (Ren & Leskovee 2020) created some transformations include additional query structures with negation, shown in the right panel of Figure
4.3
EVALUATION
Given the rank of answer entities v E V to non-trivial test query g. compute the evaluation metrics defined according to Equation 6 below. Then, we average all queries with the same query format.
Metric(g) = [[g] \. Igllaal
fmento (rank(e;)) (v;))
(6)
where is the set of answers V C Iglbest [g] fimnti the specific metric function and rank(v;) the rank of answer entities returned by the model.
Mean Reciprocal Rank (MRR): : It is statistic measure used Information Retrieval evaluate the systems that returns list of possible responses ranked by their probability to be correct. Given an answer, this measure is defined as the inverse the rank for the first correct answer, averaged over the sample of queries Q. Equation express this measure formally.
MRR O
rank;
where rank; refers to the rank position of the first correct/relevant answer for the i-th query.
Hits rate at K (H@ K): This measure considers how many correct answers ranked above K. It directly provides an idea of how the system performing. Equation defines this metric mathemat- ically.
7

---

Published as a conference paper at ICLR 2022
HaK 1ngk
4.4 BASELINES AND MODEL VARIANTS
To compare our results, we have selected the state-of-the-art baselines from some previous papers on
KG Logical Reasoning: Graph Query Embeddings (Hamilton al. 2018). Query2Box (Ren al.) 2020) and BetaE (Ren & Leskovee 2020). Only the last method, BetaE, accepts negative queries. We have modified the simpler model, MIP, to potentially improve the initial results. Below, we describe the modifications:
Heterogeneous Hyper-Graph Embeddings (Sun et al., 2021). The Heterogeneous Hypergraph Embeddings creates graph embeddings by projecting the graph into a series of snapshots and taking the Wavelet basis to perform localized convolutions. In essence, it is embedding transformation that aims to capture the information of related nodes. We have added this transformation to our MLP model right before computing the distance, and then calculate the distance measure defined specifically for this new hyper space.
Attention mechanism (Vaswani et al. 2017). The goal of attention layers is to enhance the "impor- tant" parts of the input data and fade the rest. This allows the modelling of dependence without regard to the length of the input. This method has proven a performance increase in many Deep Learning Applications. We have implemented attention mechanism for the intersection operator in the MIP model.
2- Vector Average Approach. In our models, we create an embedding of the query and the graph entities to point in hyperdimensional space. Since
using Neural Networks to create these embeddings, the resulting embeddings will depend on the training process and the optimization of the weights. We have assurance the embedding is correct. To add more robustness to the computation, we have decided to calculate the embedding twice with two separate networks and average the results between the two networks.
RESULTS
Table I shows the results of MRR for our models MLP and MLP-Mixer whee compared to the baselines- - GQE, Q2B, BetaE As listed the table, our methods yield promising results that improve those from the state of the art across all query forms and datasets. Additionally, Table 2 shows the results for the model variants previously described in Section 4.4 We observe the Hyper Embedding space does not provide an improvement when compared the basic version of the model. On the other side, the other two variants do show a significance improvement. Nonetheless, worth noting the 2-vector approach may improve the result at the of more computer power, as it mainly computes each embedding twice. Finally, in Table 3 we also show results for experiments on full FOL queries, which include negation. Additional results for H@ can be found in Appendix
5.1 ANALYSIS OF RESULTS
In general terms, observe increased performance over the selected baselines. In this section we discuss some of the implications:
Implementation of Logical Operators. One of the main differences with other approaches such as GQE (Hamilton et al.) 2018) or Q2B (Ren et al.) 2020) is the use of Neural Networks to represent all logical operations: Projection, Intersection and Negation. In comparison to GQE that only uses Q2B that creates set of geometrical operations represent the logical
it on the Intersection, operations. In light of these results, it seems that the geometrical implications of the operations can be restraining some of the possible solutions.
Correctly learning the logical operations. It hard clearly find out if the operators are learning the logical operations correctly. At least, we can assure the Neural Networks do a better job at learning the approximate solution. Neural Networks that implement directly logical operation already currently under research (Shi et al. 2019). A too constrained implementation of a Neural Logic Network can be found in Appendix
8

---

Published as a conference paper at ICLR 2022
Performance of model variants. We observe that both Attention Mechanism and the 2- V-eettor Approach manage to improve the results from the original MLP model. This indicates that Hyper- Graph Embeddings the improvement original from the 2-vector approach seems to indicate the optimal solution was not yet reached and there is still room for improvement learning the correct logical operations.
not correctly transforming the embedding space in case. Additionally,
Table MRR Results (%) of baselines (GQE, Q2B, BetaE) and our models (MLP, MLP-Mixer) on EPFO A,,,, V) queries.
Dataset
Model
1p
2p
3p
2i
ip
pi
2u
up
avg
MLPMix
69.7
27.7
23.9
58.7
69.9
30.8
46.7
38.2
24,8
43.4
MLP
67.1
31.2
27.2
57.1
66.9
33.9
45.7
38.0
28.0
43.9
FB15k
BetaE
65.1
25.7
24.7
55.8
43.9
25.2
41.6
Q2B
68.0
21.0
14.2
66.5
39.4
16.7
38.0
GQE
54.6
15.3
10.8
39.7
51.4
19.1
27.6
22.1
11.6
28.0
MLPMix
42.4
11.5
9.9
33,5
46.8
14.0
25.4
14.0
9.2
22.9
MLP
42.7
12.4
10.6
31.7
43.9
14.9
24.2
13.7
22.6
FB15k-237
BetaF
39.0
10.9
10.0
28.8
42.5
12.6
22.4
12.4
9.7
20.9
Q2B
40.6
9.4
6.8
29.5
42,3
12.6
21.2
11.3
7.6
20.1
GQE
35.0
7.2
5.3
23.3
34.6
10.7
16.5
8.2
5.7
16.3
MLPMix
55.4
16.5
13.9
39.5
51.0
18.3
25.7
14.7
11.2
27.4
MLP
55.2
16.8
14.9
36.4
48.0
18.2
22.7
14.7
11.3
26.5
NELL995
BetaE
53.0
13.0
11.4
37.6
47.5
14.3
24.1
12.2
8.5
24.6
Q2B
42.2
14.0
11.2
33.3
44.5
16.8
22.4
11.3
10.3
22.9
GQE
32.8
11.9
9.6
27.5
35.2
14.4
18.4
8.5
8.8
18.6
Table 2: MRR Results (%) of the model variants of MLP EPFO (3, A, V) queries: Hyper Embed- ding space, Attention mechanism and 2-vector approach.
Dataset
Model
1p
2p
3p
ip
pi
up
avg
2-vector
71.9
32.1
27.1
59.9
70.5
33.7
48.4
40.4
28.4
45.8
FB15k
Attention
70.0
29.5
25.4
58.6
70.0
29.8
47.2
36.8
26.5
43.8
HyperE
64.2
23.2
20.1
50.7
63.4
19.6
39.2
29.6
20.6
36.8
2-vector
43.4
12.6
10.4
33.6
47.0
14.9
25.7
14.2
10.2
23.6
FB15k-237
Attention
42.7
11.9
10.2
33.3
46.7
14.2
25.2
14.1
9.7
23.1
HyperE
41.1
10.6
9.1
28.5
41.6
11.0
21.8
13.1
8.8
20.6
2-vector
55.6
16.3
14.9
38.5
49.5
17.1
23.7
14.6
11.0
26.8
NELL995
Attention
55.6
16.2
14.4
38.0
49.0
17.9
22.3
14.7
11.0
26.6
HyperF.
54.6
14.5
12.1
34.6
45.8
13.9
21.7
12.3
9.1
24.3
Table 3: MRR Results (%) of MLP model on full FOL queries (3, V. -), including negative queries. We only use show results for negative query structures.
Datasef
Model
2in
3in
inp
pin
pni
avg
MIP
17.2
17.8
13.5
9.1
15.2
14.5
FB15K
BetaE
14.3
14.7
11.5
6.5
12.4
11.8
MI.
10.7
4.4
6.9
FB15-237
BetaF
5.1
7.9
7.4
3.6
3.4
5.4
MLP
5.1
8.0
10.0
3.6
3.6
6.1
NELL995
BetaF.
5.1
7.8
10.0
3.1
3.5
5.9
CONCLUSIONS
In this work, we have introduced a competitive embedding framework for Logical Reasoning over Knowledge Graphs. It presents flexible approach build logical operators through Neural Net- works. This method accepts queries in its First-Order Logical form, and it is one the first models to accept negative queries. Extensive experimental results show significance performance im- provement when compared to other state-of of-the----rrrr methods built for this purpose.

---

Published as a conference paper at ICLR 2022
REFERENCES
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Neural Information Processing Systems (NIPS), pp. 1-9, 2013.
Wenhu Chen, Wenhan Xiong, Xifeng Yan, and William Wang. Variational knowledge graph reason- ing. arXiv preprint arXiv: 0188888300888/8 2018.
Xiaojun Chen, Shengbin Jia, and Yang Xiang. A review: Knowledge reasoning knowledge graph. Expert Systems with Applications, 11148, 2020.
Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. Chains rea- soning over entities. relations, and text using recurrent neural networks. arXiv:1607.01426 2016.
arXiv preprint
Brian A Davey and Hilary Priestley. Introduction to lattices and order. Cambridge university press, 2002.
Ronald Fagin, Yoram Moses, Joseph Y Halpern, and Moshe Y Vardi. Reasoning about knowledge. MIT press, 2003.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. Scalable multi-hop relational reasoning for knowledge-aware question answering. CoRR, abs/2005.00646 2020. /Rttpt:ss//axxiiii
Kelvin Guu, John Miller, and Percy Liang. Traversing knowledge graphs in vector space. In Pro- ceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp- 318-327, 2015.
Will Hamilton, Payal Bajaj. Marinka Zitiik Dan Jurafsky, and Jure Leskovee. Embedding log- ical queries on knowledge graphs. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Besa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL / ngs neurips cc/ pdf
paper/2018 htt.f//eeeffff3033eeem 6
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp. 604-613, 1998.
David Liben-Nowell and Jon Kleinberg. The link- preedcceeitioo problem for social networks. Journal of the American society for information science and technology. 5877)1019--031, 2007.
Enrico Palumbo, Giuseppe Rizzo, and Raphaël Troncy. Entity2rec: Learning user-item relatedness from knowledge graphs for top-n item recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys '17, pp. 32-36, New York, NY, USA, 2017. Asso- ciation for Computing Machinery. ISBN 9781450346558. doi: 10. 00.144//119859931099 URL
htttps://di..... wrt/10. 11T.000.1111111111111 1144//31999999199999 3109889
Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. In Proceedings of the
33rd International Conference on Neural Information Processing Systems, pp. 7712-7722, 2019.
Hongyu Ren and Jure Leskovec. Beta embeddings for multi-hop logical reasoning in knowledge graphs. CoRR, abs/2010.11465, 2020. URL
wttp://ww...o//aab//........ 11465
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. CoRR, abs/2002.05969, 2020. URL https:///rxxvv..
org/abs/20022 05969
Shaoyun Shi, Hanxiong Chen, Min Zhang, and Yongfeng Zhang. Neural logic networks. CoRR, abs/1910.08629, 2019. URL wttpxxxxvvvo..oooo abs/ 1 1910. 08629
Shaoyun Shi. Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang, Neural logic reasoning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1365-1374, 2020.
10

---

Published as a conference paper at ICLR 2022
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems. pp. 926 934. Citeseer, 2013.
Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Jiuxin Cao, Yingxia Shao, and Nguyen Quoc Viet Hung. Heterogeneous hypergraph embedding for graph classification. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 725-733, 2021.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. CoRR, abs/2105.01601,
2021. URL/ttps: htttsp::/aaxiv.ooo org./ abs/ 221..2155.0000 2105.01601111
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1499-1509, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10.18653/v1/ D15-1174. URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and lllia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Quan Wang, Bin Wang, and Li Guo. Knowledge base completion using embeddings and rules. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Zikang Wang, Linjing Li, Daniel Dajun Zeng, and Yue Chen. Attention-based multi-hop reason- ing for knowledge graph. In 2018 IEEE International Conference on Intelligence and Security Informatics (ISI), pp. 211-213. IEEE, 2018.
Chenyan Xiong, Russell Power, and Jamie Callan. Explicit semantic ranking for academic search via knowledge graph embedding. In Proceedings of the 26th international conference world wide web, pp. 1271-1279, 2017a.
Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learning method for knowledge graph reasoning, arXiv preprint arXiv:1707.06649 2017b.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint 063.078, 2021.

---

Published as a conference paper at ICLR 2022
A QUERY GENERATION AND STATISTICS
Statistics about the datasets be found in Table However, it is useful to understand how they have been created. Given the datasets, graphs are created: G Guain Gval, Gest for training, testing and validation, respectively. As explained in Section 4.2 Grain contains the training edges, while Gal contains edges from training + valid and Geest from valid + test. Then they apply pre-order traversal the target nodes until all anchor entities are instantiated, and use post-order traversal to find the answers. The difference with the query generation process used in Query2Box is a limit on the number of queries, since the original dataset had queries with over 5000 answers, nearly 1/3 of some datasets. The average number of queries and answers are shown in Tables and
Table 4: Datasets statistics.
Dataset
Entities
Relations
Training Edges
Val Edges
Test Edges
Total Edges
FBI5k
14,951
1,345
483,142
50,000
59,071
592,213
FB15K-237
14,505
237
272,115
17,526
20,438
310,079
NELL995
63,361
200
114,213
14,324
14,267
142,804
Table 5: Number of queries divided by training, validation and test sets, and query structure.
Queries
Training
Validation
Test
Dataset
1p/2p/3p/2/33
2im/3in/inp/pin/pmi
1p
Rest
1p
Rest
FB15k
273,710
27.371
59,097
8,000
67,016
8,000
FB15k-237
149,689
23,714,968
20,101
5,000
22,812
5,000
NELL995
107,982
10,798
16,927
4,000
17,034
4,000
Table 6: Average number of answers divided by query structure.
Dataset
1p
2p
3p
3i
ip
pi
2u
up
2in
3in
inp
pin
pni
FB15k
1.7
19.6
24.4
8.0
5.2
18.3
12.5
18.9
23.8
15.9
14.6
19.8
21.6
16.9
FB15k-237
17.3
24.3
4.5
17.7
19.6
16.3
13.4
19.5
21.7
18.2
NELL995
1.6
14.9
17.5
3.7
6.0
17.4
11.9
14.9
19.0
12.9
11.1
12.9
16.0
13.0
B EXPERIMENTAL DETAILS
Our code is implemented using PyTorch. For the baselines, we have used the implementation of the baselines and the testing framework from BetaF (Ren Leskovee [2020), this also includes the code for Query2Box (Ren et al. (2020) and GQE (Hamilton al. 2018 8) For fair comparison with the models in the paper, we have selected the same hyperparameters listed in the paper. This also includes the consideration of the single-point vector methods GQE, MLP, MLPMixer having embedding dimensions 2n. Since Query2Box needs to model the offset and the center of the box and BetaE has parameters per distribution: and 3, and the embedding dimensions is set to
All our models and GQE use the following parameters: Embed dim 800, learning rate 0.0001, negative sample size 128, batch size = 512, margin 24, num. iterations = 300,000/450,,000 and BetaE differ from the previous configuration in Embed. dim 400 and margin 30/60.
Q2B
All experiments have been computed on independent processes on NVIDIA GPUs, either the GeForce GTX Titan Pascal (12 GB) or the Tesla T4 (16 GB).
12

---

Published as a conference paper at ICLR 2022
C ADDITIONAL RESULTS
Hits at K (H@K) is another metric of evaluation that captures the of precision of model to retrieve the correct entities. is described in the Section 4.3 In Table we show the results of H@l for all models (MLP-Mixer, MIP) and baselines (BetaF, Q2B, GQE). In Table
8 we show the H@ results for our model variants: HyperE, Attention Layers and 2-vector average. Finally, Table
9 shows the results of BetaE and MLP on negative queries.
Table H@! I Results (%) of baselines (GQE, Q2B, Betaß) and our models (MLP, MLP-Mixer) on EPFO (3,,, V) queries.
Dataset
Model
1p
2p
3p
3i
ip
pi
2u
up
avg
MLPMix
59.0
18.6
16.2
47.5
59.8
21.3
35.8
26.8
16.3
33.5
MLP
56.0
22.0
19.2
46.3
56.4
24.0
35.1
26.9
19.1
33.9
FB15k
BetaE
52.0
17.0
16.9
43.5
55.3
19.3
32.3
28.1
16.9
31.3
Q2B
52.0
12.7
7.8
40.5
53.4
16.7
26.7
22.0
9.4
26.8
GQE
34.2
8.3
5.0
23.8
34.9
11.2
15.5
11.5
5.6
16.6
MLPMix
31.9
6.0
4.7
22.7
36.5
8.2
16.7
7.7
4.3
15.4
MLP
32.5
6.4
5.3
21.4
33.4
8.9
16.0
7.5
4.3
15.1
FB15k-237
BetaE
28.9
5.5
4.9
18.3
31.7
6.7
14.0
6.3
4.6
13.4
Q2B
28.3
4.1
3.0
17.5
29.5
7.1
12.3
5.2
3.3
12.3
GQE
22.4
2.8
2.1
11.7
20.9
5.7
8.4
3.3
2.1
8.8
MLPMix
45.3
10.8
9.1
28.5
40.0
12.1
18.2
8.4
6.3
19.8
MLP
45.6
11.2
10.0
25.3
36.7
12.4
15.4
8.6
6.5
19.0
NELL995
BetaE
43.5
8.1
7.0
27.1
36.5
9.3
17.4
6.9
4.7
17.8
Q2B
23.8
8.7
6.9
20.3
31.5
10.7
14.2
5.0
6.0
14.1
GQE
15,4
6.7
5.0
14.3
20.4
9.0
10.6
2.9
5.0
9.9
Table 8: H@l Results (%) of model variants: HyperEmbedddings, Attention Mechanism and 2-
vector average.
Dataset
Model
1p
2p
3p
2i
3i
ip
pi
2u
up
avg
2-vector
71.9
32.1
27.1
59.9
70.5
33.7
48.4
40.4
28.4
45.8
FB15k
Attention
70.0
29.5
25.4
58.6
70.0
29.8
47.2
36.8
26.5
43.8
HyperE
64.2
23.2
20.1
50.7
63.4
19.6
39.2
29.6
20.6
36.8
2-vector
43.4
12.6
10.4
33.6
47.0
14.9
25.7
14.2
10.2
23.6
FB15k-237
Attetion
42.7
11.9
10.2
33.3
46.7
14.2
25.2
14.1
9.7
23.1
HyperE
41.1
10.6
9.1
28.5
41.6
11.0
21.8
13.1
8.8
20.6
2-vector
55.6
16.3
14.9
38.5
49.5
17.1
23.7
14.6
11.0
26.8
NELL9955
Attention
55.6
16.2
14.4
38.0
49.0
17.9
22.3
14.7
11.0
26.6
HyperE
54.6
14.5
12.1
34.6
45.8
13.9
21.7
12.3
9.1
24.3
Table 9: H@ I Results (%) of baselines BetaE and our model MLP negative queries.
Dataset
Model
2in
3in
inp
pin
pni
avg
MLP
8.3
8.6
6.9
3.6
7.4
6.9
FB15K
BetaE
6.4
6.6
5.5
2.2
5.2
5.2
MLP
2.2
4.2
3.4
1.4
1.2
2.5
FB15-237
BetaF
1.5
2.8
2.8
0.7
0.9
1.7
MLP
1.4
2.6
4.2
0.9
1.1
2.0
NELL995
BetaE
1.6
2.6
4.4
0.9
1.1
2.1
13

---

Published as a conference paper at ICLR 2022
D APPENDIX: ADDITIONAL MODELS
Table 10: MRR Results (%) of additional models: CNN, NLN on EPFO (3, A, V) queries.
Dataset
Model
1p
2p
3p
2i
ip
pi
2u
up
avg
CNN
41.1
11.1
9.6
29.6
41.4
11.9
21.4
13.0
9.3
20.9
FB15k-237
NLN
9.6
2.4
2.6
5.5
7.8
1.0
4.4
0.8
1.9
0.4
During this research, we have additionally explored other models, like Convolutional Neural Net- works and Neural Logic Networks. We show the preliminary results of these models in the appendix for informative reasons, available Table IO They have been tested on FB15k -37.
Convolutional Neural Networks (CNNs) CNNs manage to capture the intrinsic dependencies that happen between close items. This is the reason why they are well suited for images, where closed pixels tend to have similar colors. This is the case we find in Knowledge Graph embeddings. and we can intuitively its poor performance.
We have adapted a simple model a CNN to compute the logic operators in our method. As previously, we have created two different CNNs, one with 2 inputs (Intersection and Projection) and another one with input (Negation).
I-impul operator. Represented by CNN with 2 convolutional layers (Ist: in_channels 1, out_channels 10, kernelsiize 6, 2nd: in_channels 10, out-chaneels 10, kernel_size 6) + maxPool (kernel size = 6). followed by fully connected layers + ReLu, except for the last one which does not have ReLu. Input and output sizes the same.
2-input operator. It uses the same architecture as the 1-input operator, but applies the layers Conv+MaxPool to the 2 input entities separately and concatenates their outputs to feed the fully connected layers.
Neural Logic Networks (NLN) Neural Logic Networks (Shi et al. 2019) are kind of network architecture created to conduct logical inference. They use vectors to represent the variables, and each logic operation is learned as neural model with some predefined logic regularizers. The logic regularizers constraint the neural module to complete the tasks they are conceived for. They have defined the required regularizers for the most common operations, like NOT, AND, and OR.
In this model, we have implemented the Intersection operator and used the same Projection operator from GQE (Hamilton al. 2018): P(g.r) R, where R Rdxd is a trainable parameter for edge type and entity.
Intersection operator The intersection operator is implemented as the AND logical operation. Below, we show the formal definition of the AND (Equation 9) operation a basic multilayer per- ceptron and its corresponding regularizer in Table
[(, 0z) AND(U;,
Us; 1z) H / (Ha1 (milvy) + ba)
where Ha1 E Rdx2d Ho2 E Rdxd and by Rd are the parameters of the Neural Network.
Table 11: Logical regularizer for the AND operation. Sim(-) is measure of the similarity between two items, Euclidean distance in our case.
Operation Logic Rule
Equation
Logic Regularizer
Identity
A T
nEew 1 -Sim(AND(w: T), w)
Annihilator
W A = F
T2 = wEE SiimAND(w: F), F)
AND
Idempotence
w
T3 "EW -Sim(AND(w: w), w)
Complementation
W A W
F
BEW 1 - SSim(AND(w, NOT(W))
Comment. Neural Logic Networks could be good solution for Reasoning in KGs, especially after they have already been proven to be useful in other reasoning tasks (Shi al. 2020). There are many reasons why it might have not worked this case: a wrong implementation, unit mismatch between our loss and the regularizer values, or the regularizer being too constraining for the task.
14


Reviews:
[{"review_id": "yRF13_T6Nbp", "reviewer": "Reviewer_ukvD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes a new approach for learning and inference in knowledge graphs when the queries have logical structure.\nThe main idea is to use neural networks to implement different operators to compute the queries over the knowledge graph. In particular, mixer-MLP networks have been used to implement these operations. The experiments using standard datasets show performance improvements over state of the art methods.\nThe main weakness here was that I found it hard to understand the key ideas in the paper. For instance, I am not sure what is being trained differently for the network to “learn” the logical operations. Sections 3.3 seems to indicate that the training is quite standard for MLP-mixer architectures. So essentially, I found it hard to understand why the results were better than the state of the art. I think a bit more of analysis instead of a brief description (3.3, 3.4) will help strengthen the claims of the paper. Right now the details seemed a little fuzzy to me. Also, the improved results in the 2-Vector Average Approach makes it hard to know if the performance improvements were a result of the embeddings actually learning the logical operations since using additional embeddings seems to be helping improve performance. I think the experiment section also needs to analyze the “why” questions more deeply than is being done currently. Overall, the paper shows good results but is perhaps a bit weak on justification for these good results.\n\nAfter Author review\nThe authors did a good job in addressing some clarity concerns and some deeper insights I am raising the score accordingly.\n", "summary_of_the_paper": "New model for knowledge graph reasoning over complex logical queries.\nShows performance improvements over state of the art methods.\n", "summary_of_the_review": "Overall this paper needs a bit more deep analysis and rewriting to back up the experiment results before it can be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tax-B65-LRF", "reviewer": "Reviewer_zRD3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The results obtained are promising and the possibility of managing negation is a plus that few systems can boast of having. The experiments conducted are based on standard datasets, so the comparison with competitors is significant. I found the paper to be mathematically correct and formal enough. The description of the system is sufficiently detailed.\nA shortcoming of the paper is the related work section, which presents a list of related systems without significantly discussing the differences, their pros and cons compared to the system presented.\n\n\nAs minor issues, on page one, in the sentence 'directed acyclic graph (DAG) graph which defines ...' the word graph is repeated. While on page 6, sigma is defined as margin, but I think that the margin is gamma while sigma could be the sigmoid function. Is it right?", "summary_of_the_paper": "The article presents a new system for solving multi-hop queries on knowledge graphs. The system allows queries to be performed by handling operators using one- and two-input MLPs. This formulation also makes it possible to handle negative queries. \nThe paper also presents a second version of the presented system that exploits an MLP Mixer model, usually used in computer vision.\nThe system was tested on three standard knowledge graphs: FB15K, FB15k-237 and NELL995. The results obtained are promising because the presented system is able to improve on the established baseline. The results obtained by the BetaE, Q2B and GQE systems on the same datasets were used as the baseline.", "summary_of_the_review": "The results obtained are promising and the possibility of solving negative queries is a strong point in favour.\nAn appropriate discussion of related work is lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pcQ_uBcEA23", "reviewer": "Reviewer_rkaY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the introduction the description of the figure 1 could be improved. The understanding of what is depicted in the figure is not clear. From the introduction is not clear the embedding process of the queries and the entities.\n\nIn section 3 it is not clear how the input embedding for the entities are computed. Even the adoption of the MLP-Mixer in section 3.3 should be better explained. \n\nIn section 3.4 it is not clear how the answer vector are computed. \n\nIt is not clear the approach used to transform a FOL query to an embedding. Furthermore, how the graph is navigated in order to answer the query is not discussed. \n\nThe obtained results seems to be promising when compared to other approaches. However, a discussion of the results should be included in order to explain the motivation of the improvements.\n\n\n", "summary_of_the_paper": "The paper proposes a new embedding method for knowledge reasoning on knowledge graphs.\n", "summary_of_the_review": "Concluding, while the results seem to be interesting the clarity of the proposed approach should be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "84b6uj2kSSe", "reviewer": "Reviewer_vpE2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*:\n + Good empirical performance\n + Good amount of information to reproduce the method given in the main text and appendix\n + Able to deal with negative queries\n\n*Cons*:\n - Poor related work section: Most of the works cited are rather old (e.g. in KGR with logic rules the most recent is 2018, which in machine learning terms is old), there is no comparison of the method proposed and the baselines used, and classic KGR methods that answer queries of type (h,r,?) and complex queries should be clearly distinguished. The related work fails to explain the relation of this work to prior work in a meaningful way.\n - Clarity, the paper should outline in more detail the Graph Query Embedding method that this paper is based on.\n\n*Questions*:\n 1. Given more than 2 inputs to the intersection operator, how is the order of recursion decided? A different order will yield a different answer.\n 2. What do the colour of the nodes mean in Figure 1?\n 3. How is the subgraph (e.g. Figure 1 (C)) extracted from the Knowledge graph?\n 4. How are the initial embeddings s_i for each entity and relation learned?\n 5. How is the training objective dealing with several possible answers? (I think I know, but the main text should state it clearly)\n\n*Minor*:\n - use \\textit{NN} in mathmode to avoid the weird spacing between the two Ns", "summary_of_the_paper": "The paper proposes to learn the projection, intersection, negation operators necessary for dealing with complex queries with neural networks. The neural networks are built on top of MLP-Mixer architecture and produce good empirical performance.", "summary_of_the_review": "The paper's empirical performance is solid, the technical novelty lies in making the Graph Query Embedding method work with Neural Networks operators. However, the paper lacks clarity and the related work is poor. Happy to increase my score, should my concerns be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on answering complex logical queries over an incomplete KG and use neural networks to do so flexibly handling multiple operations from FOL. Overall reviews agree that empirical performance is impressive. One reviewer gave a strong accept, one leaning to accept and two leaning to reject. Overall, the reviewers who are leaning to reject had mostly clarity issues which seem to have been addressed by the authors (without response from reviewers). 
Given this I recomment acceptance.
Decision: Accept (Poster)