Year: 2021

Abstract:
Reasoning is a fundamental problem for computers and deeply studied in Artificial Intelligence. In this paper, we specifically focus on answering multi-hop logical queries on Knowledge Graphs (KGs). This is a complicated task because, in real-world scenarios, the graphs tend to be large and incomplete. Most previous works have been unable to create models that accept full First-Order Logical (FOL) queries, which include negative queries, and have only been able to process a limited set of query structures. Additionally, most methods present logic operators that can only perform the logical operation they are made for. We introduce a set of models that use Neural Networks to create one-point vector embeddings to answer the queries. The versatility of neural networks allows the framework to handle FOL queries with Conjunction (∧), Disjunction (∨) and Negation (¬) operators. We demonstrate experimentally the performance of our model through extensive experimentation on well-known benchmarking datasets. Besides having more versatile operators, the models achieve a 10% relative increase over the best performing state of the art and more than 30% over the original method based on single-point vector embeddings.

heading: 1 INTRODUCTION
text: Knowledge graphs (KGs) are a type of data structure that can capture many kinds of relationships between entities (e.g.: Moscow cityIn−−−→ Russia) and have been popularized since the creation of the semantic web or its introduction into Google’s search engine. They can contain many kinds of different information, and they can be widely used in question-answering systems, search engines, and recommender systems (Palumbo et al., 2017; Xiong et al., 2017a). Reasoning is a fundamental skill of human brains. For example, we can infer new knowledge based on known facts and logic rules, and discern patterns/relationships to make sense of seemingly unrelated information. It is a multidisciplinary topic and is being studied in psychology, neuroscience, and artificial intelligence (Fagin et al., 2003). The ability to reason about the relations between objects is central to generally intelligent behavior. We can define reasoning as the process of inferring new knowledge based on known facts and logic rules. Knowledge graphs are a structure used for storing many kinds of information, therefore the ability to answer complex queries and extract answers that are not directly encoded in the graph are of high interest to the AI community. To answer complex queries, the model receives a query divided in logical statements. A full FirstOrder Logic (FOL) is necessary to process a wider range of queries, which includes negative queries. FOL includes the following logical operators: Existential (∃), Conjunction (∧), Disjunction (∨), and Negation (¬). The power of representation of our logical framework is the key to process complex queries. However, most frameworks have only been able to process Existential Positive First-Order Logic (EPFO), which means that negative queries cannot be processed. For example, One could ask a knowledge graph containing drugs and side effects the following question: “What drug can be used to treat pneumonia and does not cause drowsiness?”. The first step to answer such a query is to translate it into logical statements: q = V? · ∃V : Treat(Pneumonia, V?) ∧ ¬ Cause(Drowsiness,V?). Once the query is divided into logical statements, we obtain the computation graph, a directed acyclic graph (DAG) which defines the order of operations. Afterwards, we can start traversing the graph. However, many real-world graphs are incomplete and therefore traversing them becomes very hard and even computationally impossible. There are many possible
answer entities, and it requires modeling sets of entities. As such, embedding methods become a good solution to answer these queries. Previous works (Hamilton et al., 2018; Ren et al., 2020; Ren & Leskovec, 2020) have created methods for embedding the query and the graph into a vector space. The idea of graph embeddings reduces the problem to simply using nearest-neighbor search to find the answers, without paying attention to the intermediate results. The embedding approach solves many of the problems of query-answering in knowledge graphs. In theory, we could answer the queries just by traversing the graph. In practice, graphs are large and incomplete, and answering arbitrary logical queries becomes a complicated task. The graph incompleteness means that traversing its edges would not provide the correct answers. This work aims to create some models that allow complex queries and extract the correct answers from large incomplete knowledge graphs. To this end, we present a set of models based on Neural Networks that embed the query and the entities into a one-point vector space. Then, it computes the distance between the query and the entities to rank the answers according to the likelihood to answer the query. We use the versatility of Neural Networks to create the operators needed to process FOL queries. We conduct experiments using well-known datasets for KG Reasoning: FB15k, FB15-237, and NELL. The experiments show that our models can effectively answer FOL and provide a noticeable improvement when compared with the state-of-the-art baselines. Our models provide a relative improvement of 5% to 10% to the latest state-of-art method and about 30% to 40% when compared with the method that uses the same idea of one-point vector space embeddings (Hamilton et al., 2018). The main contributions of this work are summarized as: (1). New embedding-based methods for logical reasoning over knowledge graphs: two new models, plus variants, for KG Reasoning. These methods embed the query and the entities in the same vector space with single-point vectors. Implementing the logical operators with neural networks provides versatility to create any operator with virtually the same architecture. (2). Improved performance over the current state of the art. Experimental results show that the models presented in this paper outperform the selected baselines: Graph Query Embeddings (GQE) (Hamilton et al., 2018), Query2Box (Q2B) (Ren et al., 2020), and BetaE (Ren & Leskovec, 2020). (3). Handling of negative queries. Modelling queries with negation has been an open question in KG Reasoning until recently. BetaE (Ren & Leskovec, 2020) introduced the first method able to do so. This work takes advantages of the good relationship inference capabilities of Neural Networks and uses them to create the negation operator.

heading: 2 RELATED WORK
text: Traditional tasks on graphs include Link Prediction (Liben-Nowell & Kleinberg, 2007), Knowledge Base Completion (Wang et al., 2015), or basic Query-Answering (one-hop). They are all different
versions of the same problem: Is link (h,r,t) in the KG? or Is t an answer to query (h,r,)?, where only a variable is missing. However, we face a more complicated problem, known as Knowledge Graph Reasoning, that may involve several unobserved edges or nodes over massive and incomplete KGs. In this case, queries can be path queries, conjunctive queries, disjunctive queries or or a combination of them. A formal definition of KG Reasoning can be found in Chen et al. (2020), as stated in Definition 2.1. Definition 2.1 (Reasoning over knowledge graphs). Defining a knowledge graph as: G = 〈E ,R, T 〉, where E , T represent the set of entities,R the set of relations, and the edges inR link two nodes to form a triple as (h, r, t) ∈ T . Then, reasoning over a KG is defined as creating a triplet that does not exist in the original KG, G′ = {(h, r, t)|h ∈ E , r ∈ R, t ∈ T , (h, r, t) 6∈ G}
Most related to our work are embedding approaches for multi-hop queries over KGs: (Hamilton et al., 2018), (Ren et al., 2020), (Ren & Leskovec, 2020) and (Das et al., 2016), as well as models for question answering (Yasunaga et al., 2021), (Feng et al., 2020). The main differences with these methods rely on the ability to handle full First-Order Logical Queries and using Neural Networks to define all logical operators, including the projection. We also deliver a more extensive range of networks implementations. On a larger outlook, we identify a series of works that aim to solve Knowledge Graph Reasoning with several different techniques, such as Attention Mechanisms (Wang et al., 2018), Reinforcement Learning like DeepPath (Xiong et al., 2017b) or DIVA (Chen et al., 2018), or Neural Logic Networks (Shi et al., 2020), (Qu & Tang, 2019).

heading: 3 MODELS
text: Both models presented here follow the idea behind Graph Query Embedding – GQE (Hamilton et al., 2018): Learning to embed the queries into a low dimensional space. Our models differ from it in the point that logical query operations are represented by geometric operators. In our case, we do not follow the direct geometric sense and these operators are all represented by Neural Networks, instead of just the Intersection operator in GQE. Similarly, however, the operators are jointly optimized with the node embeddings to find the optimal representation. In order to answer a query, the system receives a query q, represented as a DAG, where the nodes are the entities and the edges the relationships. Starting with the embeddings ev1 , ..., evn of its anchor nodes and apply the logical operations represented by the edges to finally obtain an embedding q of the query (Guu et al., 2015).

heading: 3.1 FORMAL PROBLEM DEFINITION
text: A Knowledge Graph (G) is a heterogeneous graph with a set of entities – nodes – (V) and a set of relations – edges – (R). In heterogeneous graphs, there can be different kinds of relations, which are defined as binary functions r : V × V → {True, False} that connect two entities with a directed edge. The goal is to answer First-Order Logical (FOL) Queries. We can define them as follows: Definition 3.1 (First-Order Logical Queries). A first-order logical query q is formed by an anchor entity set Va ⊆ V , an unknown target variable V? and a series of existentially quantified variables V1, ..., Vk. In its disjunctive normal form (DNF), it is written as a disjunction of conjunctions:
q[V?] = V? · ∃V1, ...Vk : c1 ∨ c2 ∨ ... ∨ cn (1)
where ci represents a conjunctive query of one or several literals: ci = ei1 ∧ ei2 ∧ ...∧ eim. And the literals represent a relation or its negation: eij = r(vi, vj) or ¬v(vi, vj) where vi, vj are entities and r ∈ R.
The entity embeddings are initialized to zero and later learned as part of the training process, along with the operators’ weights. Computation Graph. The Computation Graph can be defined as the Direct Acyclic Graph (DAG) where the nodes correspond to embeddings and the edges represent the logical operations. The computation graph can be derived from a query by representing the relations as projections, intersections as merges and negation as complement. This graph shows the order of operations to answer the queries. Each branch can be computed independently and then merged until the sink node is
reached. Each node represents a point in the embedding space and each edge represents a logical operation, computed via a Neural Network in our case. The representation of a FOL as a computation graph can be seen as a heterogeneous tree where each leaf node corresponds to the anchor entities and the root is the final target variable, which is a set of entities. The logical operations corresponding to the edges are defined below:
• Projection. Given an entity vi ∈ V and a relation type r ∈ R. It aims to return the set of adjacent entities with that relation. Being Pri(vi, r) the set of adjacent entities through r, we define the projection as: Pri(vi, r) = {v′ ∈ V : (v, v′) = True}.
• Intersection. The intersection can be defined as: I(vi) = ∩ni=1vi. • Negation. It calculates the complement of a set of entities T ⊆ V: N(T ) = T = V \ T ,
where the set can either be the embedding corresponding to an entity or another embedding in between which represents a set of them.
A Union operation is unnecessary, as it will be later discussed in Sections 3.5. Query2Box (Ren et al., 2020) shows that a union operator becomes intractable in distance-based metrics.

heading: 3.2 MULTI-LAYER PERCEPTRON MODEL (MLP)
text: Based on the good results of Neural Tensor Networks (NTN) (Socher et al., 2013) for knowledge base completion, we have extended a similar approach to multi-hop reasoning. We introduce three logical operators to compute the queries. Each of them is represented by a simple neural network: a multilayer perceptron. Each perceptron contains a feed-forward network: a linear layer plus a ReLu rectifier. The number of layers remains as a hyper-parameter. Figures 2a and 2b show what the model looks like. Neural operators. We define the operators with a multi-layer perceptron. The model will take as an input the embedding representation of the input entities and will return an approximate embedding of the answer. Defining the operators with a Neural Network has the advantage of generalization. Thus, we distinguish between 2-input operators, Projection and Intersection and 1-input operator, Negation. - 2-input operator (Figure 2a): Projection P , and Intersection I . The operator is composed by a multi-layer perceptron that takes 2 inputs and returns 1 as embedding as the output. The training process will make the networks learn the weights to represent each operation. Equation 2 expresses it formally:
P (si, rj) = NNk(si, rj),∀si ∈ S,∀sj ∈ R I(si, sj) = NNk(si, sj),∀si, sj ∈ S
(2)
where si ∈ S is an embedding in the vector space S, rj ∈ R is a relation and NNk is a multi-layer perceptron with k layers.
The intersection can take more than two entities as an input, for instance the 3i query structure. In this case we do a recursive operation, we use the result of the previous intersection to compute the next one. - 1-input operator (Figure 2b): Negation N . The goal of this operator is to represent the negation of a set of entities. Following the same neural network approach, we can represent it as in the equation below (Equation 3).
N(si) = NNk(si),∀si ∈ S (3)
where si ∈ S is a vector in the embedding space, it can be an entity or the result of a previous operation. NNk is a multi-layer perceptron with k layers and the same number of inputs as outputs.

heading: 3.3 MULTI-LAYER PERCEPTRON MIXER MODEL (MLP-MIXER)
text: The MLP-Mixer (Tolstikhin et al., 2021) is a Neural Architecture originally built for computer vision applications, which achieves competitive results when compared to Convolutional Neural Networks (CNNs) and Attention-based networks. The MLP-Mixer is a model based exclusively on multilayer perceptrons (MLPs). It contains two types of layers: (1) one with MLPs applied independently to patches and (2) another one with MLPs applied across patches. Figure 3 presents a diagram of the architecture. Mixer operators. We use the same procedure as in the MLP model. We use a full MLP-Mixer block to train each of the 2 operators with 2 inputs: projection and intersection. Since negation only has 1 input, the architecture cannot be accommodated for this use so far. - 2-input operator (Figure 2a). Represents Projection P or Intersection I with MLP-Mixer architecture.
P (si, rj) = MLPmix(si, rj),∀si ∈ S,∀sj ∈ R I(si, sj) = MLPmix(si, sj),∀si, sj ∈ S
(4)
where MLPmix represent the mixer architecture, si an embedding in the entity vector space S; and rj ∈ R a relation.

heading: 3.4 TRAINING OBJECTIVE, DISTANCE AND INFERENCE
text: Training Objective. The goal is to jointly train the logical operators and the node embeddings, which are learning parameters that are initialized randomly. Our training objective is to minimize
the distance between the query and the query vector, while maximizing the distance from the query to incorrect random entities, which can be done via negative samples. Equation 5 expresses this training objective in mathematical terms.
L = − log σ(γ − Dist(v; q))− k∑
j=1
1 k log σ( Dist(v′j ; q)− γ)) (5)
where q is the query, v ∈ [q] is an answer of query (the positive sample); v′j 6∈ [q] represents a random negative sample and γ refers to the margin. Both, the margin γ and the number of negative samples k remain as hyperparameters of the model. Distance measure. When defining the training objective, we still need to specify the distance measure to compare the entity vectors. Unlike in previous works, we do not need a measure that compares between boxes or distributions. The Euclidean distance is enough for this purpose, as it calculates the distance between two points in a Euclidean space: Dist(v, q) = |v − q|. Inference. Each operator provides an embedding. Following the query’s computation graph, we obtain the final answer embedding (or query representation). Then, all entities are ranked according to the distance value of this embedding to all entity embeddings via near-neighbor search in constant time using Locality Sensitivity Hashing (Indyk & Motwani, 1998).

heading: 3.5 DISCUSSION ON ANSWERING FOL QUERIES
text: The aim of this work is to answer a wide set of logical queries, specifically, be able to answer first-order logical queries (FOL), which includes: conjunctive ∧, disjunctive ∨, existential ∃ and ¬ negation operations. Notice that we will not consider universal quantification (∀) as it does not apply to real-world knowledge graphs since no entity will ever be connected to all other entities in the graph. Theorem 1 in Query2Box shows that any embedding-based method that retrieves entities using a distance-based method is not able to handle arbitrary disjunctive queries. To overcome this problem, they transformed the queries into its Disjunctive Normal Form (DNF). By doing so, the disjunction is placed at the end of the computational graph and can be easily aggregated. The transformed computational graphs are equivalent to answering N conjunctive queries. N is meant to be small in practice, and all the N computations can be parallelized. As expressed in (Davey & Priestley, 2002), all First-Order Logical Queries can be transformed into its DNF form. We refer readers to (Ren et al., 2020) to understand the transformation process.

heading: 4 EXPERIMENTS AND RESULTS

heading: 4.1 DATASETS
text: We perform experiments on three standard datasets in KG benchmarks. These are the same datasets used in Query2Box (Ren et al., 2020) and BetaE (Ren & Leskovec, 2020): FB15k (Bordes et al., 2013), FB15k-237 (Toutanova et al., 2015) and NELL995 (Xiong et al., 2017b). In the experiments, we use the standard evaluation scheme for Knowledge Graphs, where edges are split into training, test and validation sets. After augmenting the KG to also include inverse relations and double the number of edges in the graph, we effectively create 3 graphs: G train for training; G valid, which contains G train plus the validation edges; and G test which contains G valid and the test edges. Some statistics about the datasets can be found in Appendix A.

heading: 4.2 QUERY/ANSWER GENERATION
text: To obtain the queries from the datasets and its ground truth answers, we consider the 9 query basic structures from Query2box (Ren et al., 2020), shown in Figure 4. The training, validation, and test graphs were created as: G train ⊆ G val ⊆ G test, therefore the generated queries are also: JqK train ⊆ JqK val ⊆ JqK test. Thus, we evaluate and tune hyperparameters on JqK val \ JqK train and report the results on JqK test \ JqK val. We always evaluate on queries and entities that were not part of the already seen dataset used before.
For the experiments, we have used the train/valid/test set of queries-answers used for training and evaluating BetaE (Ren & Leskovec, 2020). This query-generation system differs from the original in the fact that it limits the number of possible answers to a specific threshold since some queries in Query2box had above 5000 answers, which is unrealistic. More information about how queries are created in BetaE can be found in Appendix A. To include FOL queries, BetaE (Ren & Leskovec, 2020) created some transformations to include 5 additional query structures with negation, shown in the right panel of Figure 5.

heading: 4.3 EVALUATION
text: Given the rank of answer entities vi ∈ V to a non-trivial test query q, we compute the evaluation metrics defined according to Equation 6 below. Then, we average all queries with the same query format.
Metric(q) = 1 |JqKtest \ JqKval| ∑ vi∈V fmetric(rank(vi)) (6)
where vi is the set of answers V ⊂ JqKtest \ JqKval, fmetric the specific metric function and rank(vi) the rank of answer entities returned by the model. Mean Reciprocal Rank (MRR): It is a statistic measure used in Information Retrieval to evaluate the systems that returns a list of possible responses ranked by their probability to be correct. Given an answer, this measure is defined as the inverse of the rank for the first correct answer, averaged over the sample of queries Q. Equation 7 express this measure formally.
MRR = 1
|Q| |Q|∑ i=1 1 ranki (7)
where ranki refers to the rank position of the first correct/relevant answer for the i-th query. Hits rate at K (H@K): This measure considers how many correct answers are ranked above K. It directly provides an idea of how the system is performing. Equation 8 defines this metric mathematically.
H@K = 1vi≤K (8)

heading: 4.4 BASELINES AND MODEL VARIANTS
text: To compare our results, we have selected the state-of-the-art baselines from some previous papers on KG Logical Reasoning: Graph Query Embeddings (Hamilton et al., 2018), Query2Box (Ren et al., 2020) and BetaE (Ren & Leskovec, 2020). Only the last method, BetaE, accepts negative queries. We have modified the simpler model, MLP, to potentially improve the initial results. Below, we describe the 3 modifications: Heterogeneous Hyper-Graph Embeddings (Sun et al., 2021). The Heterogeneous Hypergraph Embeddings creates graph embeddings by projecting the graph into a series of snapshots and taking the Wavelet basis to perform localized convolutions. In essence, it is an embedding transformation that aims to capture the information of related nodes. We have added this transformation to our MLP model right before computing the distance, and then we calculate the distance measure defined specifically for this new hyper space. Attention mechanism (Vaswani et al., 2017). The goal of attention layers is to enhance the “important” parts of the input data and fade out the rest. This allows the modelling of dependence without regard to the length of the input. This method has proven a performance increase in many Deep Learning Applications. We have implemented an attention mechanism for the intersection operator in the MLP model. 2-Vector Average Approach. In our models, we create an embedding of the query and the graph entities to a point in a hyperdimensional space. Since we are using Neural Networks to create these embeddings, the resulting embeddings will depend on the training process and the optimization of the weights. We have no assurance the embedding is correct. To add more robustness to the computation, we have decided to calculate the embedding twice with two separate networks and average the results between the two networks.

heading: 5 RESULTS
text: Table 1 shows the results of MRR for our models – MLP and MLP-Mixer – when compared to the baselines – GQE, Q2B, BetaE –. As listed in the table, our methods yield promising results that improve those from the state of the art across all query forms and datasets. Additionally, Table 2 shows the results for the model variants previously described in Section 4.4. We observe the Hyper Embedding space does not provide an improvement when compared to the basic version of the model. On the other side, the other two variants do show a significance improvement. Nonetheless, it is worth noting the 2-vector approach may improve the result at the cost of more computer power, as it mainly computes each embedding twice. Finally, in Table 3 we also show results for experiments on full FOL queries, which include negation. Additional results for H@1 can be found in Appendix C.

heading: 5.1 ANALYSIS OF RESULTS
text: In general terms, we observe an increased performance over the selected baselines. In this section we discuss some of the implications: - Implementation of Logical Operators. One of the main differences with other approaches such as GQE (Hamilton et al., 2018) or Q2B (Ren et al., 2020) is the use of Neural Networks to represent all logical operations: Projection, Intersection and Negation. In comparison to GQE that only uses it on the Intersection, or Q2B that creates a set of geometrical operations to represent the logical operations. In light of these results, it seems that the geometrical implications of the operations can be restraining some of the possible solutions. - Correctly learning the logical operations. It is hard to clearly find out if the operators are learning the logical operations correctly. At least, we can assure the Neural Networks do a better job at learning the approximate solution. Neural Networks that implement directly a logical operation are already currently under research (Shi et al., 2019). A too constrained implementation of a Neural Logic Network can be found in Appendix D. - Performance of model variants. We observe that both Attention Mechanism and the 2-Vector Approach manage to improve the results from the original MLP model. This indicates that HyperGraph Embeddings are not correctly transforming the embedding space in our case. Additionally,
the improvement original from the 2-vector approach seems to indicate the optimal solution was not yet reached and there is still room for improvement on learning the correct logical operations.

heading: 6 CONCLUSIONS
text: In this work, we have introduced a competitive embedding framework for Logical Reasoning over Knowledge Graphs. It presents a flexible approach to build logical operators through Neural Networks. This method accepts queries in its First-Order Logical form, and it is one of the first models to accept negative queries. Extensive experimental results show a significance performance improvement when compared to other state-of-the-art methods built for this purpose.

heading: A QUERY GENERATION AND STATISTICS
text: Statistics about the datasets can be found in Table 4 However, it is useful to understand how they have been created. Given the 3 datasets, 3 graphs are created: G train,G val,G test for training, testing and validation, respectively. As explained in Section 4.2, G train contains the training edges, while G val contains edges from training + valid and G test from valid + test. Then they apply pre-order traversal to the target nodes until all anchor entities are instantiated, and use post-order traversal to find the answers. The difference with the query generation process used in Query2Box is a limit on the number of queries, since the original dataset had queries with over 5000 answers, nearly 1/3 of some datasets. The average number of queries and answers are shown in Tables 5 and 6.

heading: B EXPERIMENTAL DETAILS
text: Our code is implemented using PyTorch. For the baselines, we have used the implementation of the baselines and the testing framework from BetaE (Ren & Leskovec, 2020), this also includes the code for Query2Box (Ren et al., 2020) and GQE (Hamilton et al., 2018). For a fair comparison with the models in the paper, we have selected the same hyperparameters listed in the paper. This also includes the consideration of the single-point vector methods – GQE, MLP, MLPMixer – having embedding dimensions 2n. Since Query2Box needs to model the offset and the center of the box and BetaE has 2 parameters per distribution: α and β, and the embedding dimensions is set to n. All our models and GQE use the following parameters: Embed dim = 800, learning rate = 0.0005, negative sample size = 128, batch size = 512, margin = 60, num. iterations = 300,000/450,000. Q2B and BetaE differ from the previous configuration in Embed. dim = 400 and margin = 30. All experiments have been computed on independent processes on NVIDIA GPUs, either the GeForce GTX Titan X Pascal (12 GB) or the Tesla T4 (16 GB).

heading: C ADDITIONAL RESULTS
text: Hits at K (H@K) is another metric of evaluation that captures the sense of precision of our model to retrieve the correct entities. It is described in the Section 4.3. In Table 7 we show the results of H@1 for all models (MLP-Mixer, MLP) and baselines (BetaE, Q2B, GQE). In Table 8 we show the H@1 results for our model variants: HyperE, Attention Layers and 2-vector average. Finally, Table 9 shows the results of BetaE and MLP on negative queries.

heading: D APPENDIX: ADDITIONAL MODELS
text: During this research, we have additionally explored other models, like Convolutional Neural Networks and Neural Logic Networks. We show the preliminary results of these models in the appendix for informative reasons, available in Table 10. They have been tested on FB15k-237. Convolutional Neural Networks (CNNs) CNNs manage to capture the intrinsic dependencies that happen between close items. This is the reason why they are well suited for images, where closed pixels tend to have similar colors. This is not the case we find in Knowledge Graph embeddings, and we can intuitively its poor performance. We have adapted a simple model of a CNN to compute the logic operators in our method. As previously, we have created two different CNNs, one with 2 inputs (Intersection and Projection) and another one with 1 input (Negation). - 1-input operator. Represented by CNN with 2 convolutional layers (1st: in channels = 1, out channels = 10, kernel size = 6, 2nd: in channels = 10, out channels = 10, kernel size = 6) + maxPool (kernel size = 6), followed by 3 fully connected layers + ReLu, except for the last one which does not have ReLu. Input and output sizes are the same. - 2-input operator. It uses the same architecture as the 1-input operator, but applies the layers Conv+MaxPool to the 2 input entities separately and concatenates their outputs to feed the 3 fully connected layers. Neural Logic Networks (NLN) Neural Logic Networks (Shi et al., 2019) are a kind of network architecture created to conduct logical inference. They use vectors to represent the variables, and each logic operation is learned as a neural model with some predefined logic regularizers. The logic regularizers constraint the neural module to complete the tasks they are conceived for. They have defined the required regularizers for the most common operations, like NOT, AND, and OR. In this model, we have implemented the Intersection operator and used the same Projection operator from GQE (Hamilton et al., 2018): P (q, r) = Rrq, where Rd×dr is a trainable parameter for edge type r and q an entity. - Intersection operator The intersection operator is implemented as the AND logical operation. Below, we show the formal definition of the AND (Equation 9) operation – a basic multilayer perceptron – and its corresponding regularizer in Table 11.
I(vi, vj) = AND(vi, vj) = Ha2f(Ha1(vi|vj) + ba) (9)
where Ha1 ∈ Rd×2d, Ha2 ∈ Rd×d and ba ∈ Rd are the parameters of the Neural Network.
Comment. Neural Logic Networks could be a good solution for Reasoning in KGs, especially after they have already been proven to be useful in other reasoning tasks (Shi et al., 2020). There are many reasons why it might have not worked in this case: a wrong implementation, a unit mismatch between our loss and the regularizer values, or the regularizer being too constraining for the task.

Review summaries:
The article presents a new system for solving multi-hop queries on knowledge graphs. The system allows queries to be performed by handling operators using one- and two-input MLPs. This formulation also makes it possible to handle negative queries.  The paper also presents a second version of the presented system that exploits an MLP Mixer model, usually used in computer vision. The system was tested on three standard knowledge graphs: FB15K, FB15k-237 and NELL995. The results obtained are promising because the presented system is able to improve on the established baseline. The results obtained by the BetaE, Q2B and GQE systems on the same datasets were used as the baseline.