Neural-Symbolic Entangled Framework for Complex Query Answering
Authors: Zezhong Xu
Wen Zhang
Peng Ye
Hui Chen
Huajun Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural and symbolic
complex query answering
knowledge graph

Abstract: Answering complex queries over knowledge graphs (KG) is an important yet challenging task because of the KG incompleteness issue and cascading errors during reasoning. Recent query embedding (QE) approaches embed the entities and relations in a KG and the first-order logic (FOL) queries into a low dimensional space, making the query can be answered by dense similarity searching. However, previous works mainly concentrate on the target answers, ignoring intermediate entities' usefulness, which is essential for relieving the cascading error problem in logical query answering. In addition, these methods are usually designed with their own geometric or distributional embeddings to handle logical operators like union, intersection, and negation, with the sacrifice of the accuracy of the basic operator -- projection, and they could not absorb other embedding methods to their models. In this work, we propose a Neural and Symbolic Entangled framework (ENeSy) for complex query answering, which enables the neural and symbolic reasoning to enhance each other to alleviate the cascading error and KG incompleteness. The projection operator in ENeSy could be any embedding method with the capability of link prediction, and the other FOL operators are handled without parameters. With both neural and symbolic reasoning results contained, ENeSy answers queries in ensembles. We evaluate ENeSy on complex query answering benchmarks, and ENeSy achieves the state-of-the-art, especially in the setting of training model only with the link prediction task.

Neural-Symbolic Entangled Framework for Complex Query Answering
Zezhong Xu Wen , Peng Ye', Hui Chen Huajun Chen 1.2t 'Zhejiang Univensity & AZFT Joint Lab for Knowledge Engine, China Hangghou Innovation Center, Zhejiang University, *Alibaba Group {xuzezhong, zhang.went wen, yep, huajunsir) @zju. edu. en, weidu. ch@al baaabbaabbeee inc. com
Abstract
Answering complex queries over knowledge graphs (KG) is an important yet chal- lenging task because of the KG incompleteness issue and cascading errors during reasoning. Recent query embedding (QE) approaches embed the entities and rela- tions in a KG and the first-order logic (FOL. queries into a low dimensional space, answering queries by dense similarity search. However, previous works mainly concentrate on the target answers, ignoring intermediate entities' usefulness, which is essential for relieving the cascading error problem in logical query answering. In addition, these methods are usually designed with their own geometric or distribu- tional embeddings to handle logical operators like union(V). intersection(A), and negation(-), with the sacrifice of the accuracy of the basic operator projection, and they could not absorb other embedding methods to their models. In this work, we propose a Neural and Symbolic Entangled framework (ENeSy) 9) for complex query answering, which enables the neural and symbolie reasoning enhance each other to alleviate the cascading error and KG incompleteness. The projection operator in ENeSy could be any embedding method with the capability of link prediction, and the other FOL operators are handled without parameters. With both neural and symbolic reasoning results contained, ENeSy answers queries in ensembles. ENeSy achieves the SOTA performance on several benchmarks, especially in the setting of training model only with the link prediction task.
Introduction
People built different Knowledge Graphs, such as Freebase [4], YAGO [13], and Wordnet [11], to store complex structured information and knowledge. The facts in KG are usually represented in the form of triplets, e.g., isCityOf(Nen) York, USA). KGs have been widely applied in various intelligent systems such as question answering and natural language understanding. One of the key tasks on KG reasoning is complex query answering which involves answering FOL query with logical operators including existential quantification (3), conjunction(A), disjunction(V (V, and negation(-).
Given question "Who won the Turing Award in developing countries?". illustrated in Figure could be converted to FOL query, and a computation graph can be generated with the query. Each node in the computation graph represents an entity or an entity set, while each edge represents logical operation. Answering these queries is challenging since not all the answers could be directly identified by traversing the KG because of the incompleteness of KG. To address this problem, several
"Equal Contribution. i Corresponding Author. 'Souree code of ENeSy is available at https//gittub.com///kkk///E/
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

V, V: CoContainnDeeeelppirg countrrppppppocooor..oooo V) Citizen(V Winnuuriin Award,V.)
q Vs. www.oonnnnnneeapppnnnnnn wtwp:lpppppnnooooty.y.o.o Country,))
V, = (China, India, Sri Lanka,
Turing
ry Win
Intersection
4 Award
A
A, Developing Country
ri Contain
Py Citizen
embedding space
Anchor Entity
Variable
Predicted Entryy
Embedding Projection Neural Symbolic Entangled
(A) Computation Graph
(B) Neural Symbol Entangled Reasoning
Figure 1: An example of answering a complex graph query by using the ENeSy. (A): FOL query and its computation graph for the question 'Who won the Turing Award in developing countries?' (B): ENeSy uses neural and symbolic ways to handle projection separately, and the results are entangled enhance each other alleviate the problem of cascading error and incompleteness of KG. The logic operator A, V, and - are supported with symbolic reasoning.
QE methods [9, 14, 15, 6] are proposed which encode entities and the query to a low-dimensional embedding space and the logical operators are also parameterized. Following the corresponding computation graph, the query embedding could be computed step by step. The target answers are selected according the distance between the final embedding and candidate entity embedding.
Although the query embedding methods are effective for solving the incompleteness of KG, several limitations still exist. Firstly, the role of intermediate entities remains largely unexamined which we argue plays significant role in obtaining the target answers since besides incompleteness, cascading етrоr also influences the accuracy of complex queries. Existing works only pay attention to the target answers, but for multi-hop reasoning, the intermediate entities could not ignored. Secondly, previous works propose different geometric shapes or distribution embedding to support more logical operators. For instance, Query2Box (Q2B) embed queries to boxes and can handle intersection(A)) while ConE [26] and BetaE [15] embed query with cone embedding and beta distribution to further support negation(-)) Although the logical operators are supported with their own embedding shape, the performance of the basic operator, projection, is not satisfying. For example, the accuracy of Q2B on one projection query is worse than TransE [5] according to the Q2B paper [14]. Few studies have investigated generalizing existing embedding models, such TransE and RotatE which achieve good performance on link prediction, to the complex query.
In this paper, we propose neural symbolic entangled model, named ENeSy, which enables embed- ding and symbolic reasoning to enhance each other. The embedding results influence the symbolic results to solve the KG incompleteness while symbolic results could revise embedding results by alleviating the problem of cascading error in multi-hop reasoning. Our approach has the following important advantages: (1) ENeSy can generalize KG embedding methods to answering complex queries, and could not only use the classic KGE algorithm like TransE, but also the projection operator of existing query embedding methods could be employed. (2) The logical operators except for projection (including A, V.-) are not parameterized so ENeSy could be trained with only link prediction task since meaningful complex query data might be hard to obtain in the real world.
The experimental results prove that our model outperforms existing complex query answering methods over standard knowledge graphs: FB FB15K-237 [18] and NELL-995 [22]. The performance, which is trained with link prediction, also competitive or better than the baselines trained with complex queries. The analysis of the model proves the effectiveness of neural and symbolic entanglement in solving the problem cascading error and KG incompleteness with our framework.
Related Work
2.1 Knowledge Graph Embedding
great deal of previous research into KG has focused on using machine learning to reason with the embedding method. The effects have been shown in TransE [5], TransH [20], TransG [21],

---

ComplEx [19], ConvE [8]. DistMult [23] and RotatE [17]. These approaches aim to embed the entities and relations continuous vector space so that one-hop reasoning can be answered with link prediction. Different methods map the entities and relations into vector space with different distributions. Meanwhile, the rule and path-based models [10, 25, 24, 16], try to use learned patterns of path or rules to do multi-hop reasoning.
However, KGE models lack the ability to handle complex logical operators like conjunction (A), disjunction (V), and negation (-), and they are not easy to generalize to multi-hop queries. In contrast, our framework can alleviate the problem of cascading error during multi-hop reasoning and also support all of the logical operators above so that any KGE models could be used to answer the complex query.
2.2 Embedding for Complex Query Answering
To answer logical queries, some works [9, 14, 26, 2, 15, 6] aims to encode the complex query into space that proposes various geometric shape to represent the entity set and support complex query reasoning. Generally, a FOL query is converted to computation graph with a directed acyclic graph (DAG) structure with which the query representation could be iteratively computed using the logical operation in embedding space. GQE (graph-query embedding) [9] consider conjunction operator (A) with vector representation. Q2B (query to box) [14] replace the vector embedding with hyper-rectangles since it holds the idea that box embedding can represent sets of entities better and define the disjunctive norm form (DNF) to support the disjunction (V) operator. More recently, BetaE [15] models the query and entities with beta distributions which could support the negation (-) operator. CQD [1] applics beam search to an embedding model. ConE [26] proposes a new geometry model that embeds entities with cones embedding. FuzzQE [6] satisfies the axiomatic system of fuzzy logic to reason. Q2P (Quere2Particles) [2] encodes each query into multiple vectors. GNN-QE [27] concentrates on the interpretation of the variables along the query path.
Most of these methods usually design their geometric or distribution embedding to support logical operators but the effectiveness of the projection operator, which is the most basic operator, has not been discussed. Meanwhile, they only concentrate on the final answers as labels but ignore the intermediate entities during the query process which are also important for reasoning.
3 Preliminaries
In this section, we introduce the task of complex query answering on KG. Given set of entities V and set of relations R, a knowledge graph 9 is defined as (V. R. T) where T is the set of triplets. A triplet is defined r(e,, ez) where e, V.e E V.r E R if the relation exists between e; and ej.
First-Order Logic. The purpose of complex query answering on KG is obtaining the target answer of FOL query which is defined with existential quantifiers("), conjunction (A), disjunction (V), and negation (-). In a FOL query q, the anchor nodes are represented as set Va C V, existential quantified variables nodes are represented as V1, V2,... V2 and the target answer nodes is variable V4. Following the betaE [15], we use the FOL query in its disjunctive norm form, with which the query can be represented disjunction of several conjunctions. Finally, the query can be formulated as:
V = V32 V1, V2, V& a 02 V... V...
where c; is conjunction of several literals ag, i.e., c az A ... aim, and ai is atom or negation of an atom: r(c,, V) or -r(ea, V) or r(V', V) or -VV' V) where e, E Va. V, V e (V1, V2 V....., Ve, Vi) and V * V in atom.
Computation Graph and Logical Operators As illustrated in figure 1, for given FOL query, we can represent the whole process of reasoning as computation graph. Each node corresponds to variable V or an anchor node e, and each edge represents a logical operation over the entity sets which includes the following operators:
Relational Projection: Given entity set S C V and a relation E R, the projection operator return a new entity set S' that contains the entities related to least one of entity in S: S' = {e' E V| e S}.
3

---

Intersection: Given sets of entities 1S1.S2. .......,,y, S. where S; S V, the intersection operator returns the intersection of these sets N S..
Union:
Given sets of entities {S1, .p...... where S; S V, the union operator returns the union of these sets U.=1 St.
Complement: Given set of entities S, the complement operator returns its complement S' S.
4
Methodology
In this section, we first introduce the neural and symbolic entangled projection operator in detail. Then we define symbolic-based logical operators and describe ensemble prediction using embedding and symbolic results. Finally, the objective function and the learning procedure will be represented.
4.1 Neural and Symbolic Entangled Reasoning
Symbolic Embedding
Blstanne vel
PN
aggregate
Intersection(A)
Union(v)
integrate
Projection(E)
Negation(-)
Figure 2: ENeSy's logical operators and the details about neural symbolic entanglement. PN means neural projection and Ps means symbolic projection.
In our work, there are two ways to represent entities and relations. The neural part is just like the KGE query embedding method adopt, and rotatE [17] is chosen in this paper which embeds entities and relations to a complex space Ck For the symbolic part, an entity and relation are encoded one-hot vector and an adjacent matrix, respectively.
Specifically, given the KG G, entity set and relation set R, an entity e's embedding representation a vector V. Ck and its symbolic representation is encoded one-hot vector Pe {0. 1)1 Each relation modeled
vector v, E C and the corresponding symbolic representation an adjacent matrix M, E {0, 1] [V where M if r(e,, ey) G, else M = 0.
Neural projection follows the embedding method, we use rotatE [17] here as an example. For projection query rh r, ?), the functional mapping with relation is an element-wise rotation from to the target answer t:
Ve VA Vr, where v,
and o means Hadamard product. This step gets the predicted embedding which could be used to search for the target answers.
Symbolic projection is conducted with matrix multiplications followed TensorLog [7]:
M,)
(2)
where p. is multi-hot vector that represents the entities linked with h via relation r, T means transposing the vector and g a normalization function. Particularly, we consider the function as g(x) = x/ x/sum(x). This step gets the target entities by traversing KG.

---

Entangled projection tries to take advantage of these above two results. Since embedding prediction suffers from the cascading error but could obtain the answer not linked with h, while traversing KG could not get the answers which lack the edges to the head entity but the searched answers are convincing, ENeSy combines them an entanglement way as illustrated in Figure Specifically, the similarity between the predicted embedding and all the entities are calculated first. We define the similarity function as:
S(x, - ||x -yll
(3)
where y denotes the margin. The norm function can be changed to any distance function. We define new vector representation with these similarity values. After softmax function, we get inferred vector p E [o, 1] which is generated from Vz. With p/, a new symbolic vector p is obtained by the following step:
P g(P: P!)
(4)
This new symbolic vector P integrates the information from the embedding V; with symbolic reasoning results Pt- This procedure adds more entities, which might be the answer to the query which are not linked with to p²" This enhancement eliminates the limitation of KG incompleteness of symbolic reasoning to some extent. Each element of P could be regarded as the probability of the corresponding entity. Let's assume the entity set with non-zero probability as St, aggregation function is employed to transfer the symbolic vector p and the embedding of entities in S; to a new embedding vector vद with an MI. function:
V/
Σ Pi MLL, E
(5)
where p is the corresponding probability of E; in pt- This new embedding v/ aggregates the symbolic answer, Note that although we use the rotatE as the neural function, any other sufficiently expressive KG embedding model or the projection operator of any other query embedding models could be employed with our framework in theory.
4.2 Neural and Symbolic Ensemble Answering
With the symbolic vector p. the logical operator intersection(p) A p2). union(pi V P2) and negation (-p) could be defined as follows:
a
Pi A P2 g(P1 o P2), P1 V P2 g(P: + P1 o P2), p: :p:g g(
p)
V
where o is the Hadmard product and a is hyperparameter. After getting symbolic vector with these logical operators, the MLP function used in Equation (5) employed to get an aggregated
embedding.
The embedding and symbolic vector can both be used to get the final answers. For the neural part, the similarity between the embedding vector v and all the entities E V is computed with Equation (3), which is used to rank the candidate answers. For the symbolic part, since the vector p represents the probability of each entity, the answers can be directly obtained with ranked elements of p. To make the ensemble using the two type answers, set get a combined result with and p as:
a Ap + (1 A) Softmax CaaiaaaxCContttSSyS ve)))
VeEV
where À is the weight to balance the influence of and p and Concat is function mapping the similarity between all entities e E V and v to a vector. The final answers can be determined with g(a) E 1]
4.3 Learning Procedure
Given query q with answer entity set S,,, after getting the final answer embedding v, and symbolic vector Py, we construct two following objective loss functions:
a Σ
L -logo vo))
logo ve))
(7)
TI
La = -logo log(p, 0)-)
(8)
5

---

where e E Sq is an answer of 9, c' & Sq is negative answer which is sampled randomly. a is the sigmoid d function, is dot-product and xx f denotes the maximum value between each element of x and 0, which is a threshold.
Meanwhile, the MLP function in Equation (5) needs to be pretrained individually, Since this function is employed to convert a symbolic vector to an embedding vector, and in the projection operator, we have p/ which is generated from Vz, MLP could be used convert pf back to v,. Based on this, we also design a loss function to train this function as follows:
| L3 = -logo Σ P DYMLP(va)aa)).ee
where St is the corresponding entity set whose probabilities are not zero in p/. In the first step of the training process, the symbolic part is not included, and only the embedding of entities and relations will be trained with the link prediction task. Next, in the second step, the completed projection operator will be trained still by link prediction and the loss function is L = L1 + L2 + L3. In theory, the model could answer complex queries after the above two steps, but it could also be fine-tuned with complex query data using the loss function L = L1 + L2.
5 Experiment
In this section, we evaluate the ability of ENeSy on answering the complex query on several KG benchmark datasets. The experiment results demonstrate that: 1) The performance of ENeSy is excellent; 2) We can train ENeSy with only link prediction task to answering complex queries; 3) The embedding and symbolic parts of our model can enhance each other.
5.1 Dataset and Experiment Setting
5.1.1 Datasets and Evaluation Protocol
We perform the experiments on two benchmarks, FB15K-237 [18] and NELL-995 [22]. FB15K-237 a subset from Freebase [3] and removes the inverse relation. NELL-995 is a dataset constructed from high-confidence facts of NELL [12].
The focus of our experiment is answering FOL queries on incomplete KGs we only evaluate the ability of models to obtain the answers that could not be discovered by traversing KGs. Specifically, with the standard training, validation, and testing set, the edges in KG could be divided into three parts, which are training edges, validation edges, and test edges. The corresponding graph Gerain- Guatid and Gtest build with training edges, training + validation edges and training validation test edges, respectively, so we have Strain C Gvalid C Stext- We only use the queries whose answer sets Atrain, Avalid and Atest different graph have Atrain C Avalid C Ateat- The answers in Avatid - Atrain could be used to tune the hyper-parameters and results are reported with the answer entities in Arest Amatid. This means we only evaluate on entities that not the answers to the training query set and the model has not seen them. Meanwhile, these answers could not be found with graph traversal on Strasn/Saisee For each answer of a test query, we calculate the Mean Reciprocal Rank (MRR) as evaluation metrics, and the results are reported with filter setting as TransE [5], in which all other correct answers are filtered out before calculating the rankings of answers.
The queries sampled from these two benchmarks are provided by BetaE [15] which is an expansion of the version provided by Q2B [14]. The query set contains 14 different types of query structures are shown in Figure 3. In order to further verify the generalization ability of the models, only conjunctive queries (Ip/2p/3p/2i/3i) and 5 query types with negation (2in/3in/nmp/pnippin are used to train the model, while the other four query types (2u/up/ip/pi) do not appear in the training process and are directly evaluated when testing, which makes this task more challenging.
5.1.2 Baseline
We consider four baselines as the compared methods the following sections: Graph Query Embed- ding (GQE) [9] embeds the query into vectors, which could handle the projection and conjunction

---

1p
2p
3p
2i
3i
inp
pin
2in
3im
ip
pi
2u
up
pni
Anchor Entity
Variable
Answer
Projection
Intersection
Union
Negation
The query structure of all queries used for training and evaluation. Namely, the i. u and stands for the projection, intersection, union and negation, respectively.
Figure 3:
Table The MRR results FOL queries on FB15K-237 and NELL-9955, and the models are trained with only link prediction task. The Avg, and Avg, are the average MRR of Existential Positive First
Order (EPFO) queries (query with 3, or A and without -) and queries with respectively. N/A means not available.
Model
Avgp
Avgn
1p
2p
3p
2i
3i
pi
ip
2u
up
2in
3in
inp
pin
pni
FB15K-237
GQE
17.7
N/A
41.6
7.9
5.4
25.0
33.6
16.3
10.9
11.9
6.2
N/A
N/A
N/A
N/A
N/A
Q2B
18.2
N/A
42.6
6.9
4.7
27.3
36.8
17.5
11.1
11.7
5.5
N/A
N/A
N/A
N/A
N/A
BetaF
15.8
0.5
37.7
4.4
23.3
34.5
15.1
7.8
9.5
4.5
1.1
FuzzQE
21.8
6.6
44.0
10.8
8.6
32.3
41.4
22.7
15.1
13.5
8.7
7.7
9.5
7.0
4.1
4.7
ENeSy
23.4
8.1
44.5
10.8
7.7
33.2
48.4
25.8
18.8
13.4
7.6
9.6
10.2
7.1
5.8
7.8
NELL-995
GQE
21.7
N/A
47.2
12.7
9.3
30.6
37.0
20.6
16.1
12.6
9.6
N/A
N/A
N/A
N/A
N/A
Q2B
21.6
N/A
47.6
12.5
8.7
30.7
36.5
20.5
16.0
12.7
9.6
N/A
N/A
N/A
N/A
N/A
BetaE
19.0
0.4
53.1
6.0
3.9
32.0
37.7
15.8
8.5
10.1
3.5
0.1
1.4
0.1
0.1
0.1
FuzzQE
27.1
7.3
57.6
17.2
13.3
38.2
41.5
27.0
19.4
16.9
12.7
9.1
8.3
8.9
4.4
5.6
ENeSy
28.7
9.4
58.8
17.4
12.8
39.1
48.9
29.1
24.1
16.0
12.4
10.9
8.2
11.0
8.4
8.6
queries. With DNF setting, it could also support the union operator. Query2Box (Q2B) [14] uses box embedding to represent the query and entity sets and could answer existential positive first-order (EPFO) logic queries. Beta Embedding (BetaE) [15] models query as Beta Distributions which enables it to support negation(-) operation. CQD [1] applies beam search to embedding model but it could not support the negation operator. FuzzQE [6] uses fuzzy logic to embed the query.
The MRR results of these baselines are from the BetaE [15] and FuzzQE paper [6]. The first two methods can't handle negation operation and among these models, only FuzzQE can be trained with only link prediction tasks and answer the complex query.
5.1.3 Training Procedure and Experiment Settings
To train the model, we first only train the embedding relations and entities with Ip queries which is similar to the pure KG embedding training. Second, the projection operator of ENeSy is trained on 1p queries. Meanwhile, the queries of all structures can be used to fine-tune the model.
We implement our model with Pytorch framework and train our model on RTX3090 GPU. The ADAM optimizer was used to parameter tune with a learning rate of 0.0001 that will decrease during the training process. In the second step, the learning rate is set to be 10-5 to train the model with 1p queries, and in the fine-tuning process, the learning rate starts with dimension of the entity and relation to 1024, respectively. The hidden state dimension of MLP is 1024. The training batch size [64, 16) for FB15K-237 and NELL-995, while the negative sample size is [128, 32). The margin y used in similarity computation is 24. used as a threshold is 10 10 set to be The choices of À which used for ensemble prediction are based on the results of valid set for each query type. The best hyperparameter setting is selected by the MRR metric on the valid set. We run the experiment several times and find random seed has almost no effect on the result, but in the fine-tuning process, the learning rate influence the accuracy.
10-7 We set the embedding

---

Table 2: The average MRR results of FOL queries on FB FB15K-231 and NELL-995, and the models are trained with complex query data.
Model
Avg,
Avg..
1p
2p
3p
2i
31
pi
ip
2u
up
2in
3in
inp
pin
pni
FB15K-237
GQE
16.3
N/A
35.0
7.2
5.3
23.3
34.6
16.5
10.7
8.2
5.7
N/A
N/A
N/A
N/A
N/A
Q2B
20.1
N/A
40.6
9.4
6.8
29.5
42.3
21.2
12.6
11.3
7.6
N/A
N/A
N/A
N/A
N/A
BetaB
20.9
5.5
39.0
10.9
10.0
28.8
42.5
22.4
12.6
12.4
9.7
5.1
7.9
7.4
3.5
3.4
CQD
21.7
N/A
46.3
9.9
5.9
31.7
41.3
21.8
15.8
14.2
8.6
N/A
N/A
N/A
N/A
N/A
FuzzQE
24.2
8.5
42.2
13.3
10.2
33.0
47.3
26.2
18.9
15.6
10.8
9.7
12.6
7.8
5.8
6.6
ENeSy
24.5
8.5
44.7
11.7
8.6
34.8
50.4
27.6
19.7
14.2
8.4
10.1
10.4
7.6
6.1
8.1
NELL
GQE
18.6
N/A
32.8
11.9
9.6
27.5
35.2
18.4
14.4
8.5
8.8
N/A
N/A
N/A
N/A
N/A
Q2B
22.9
N/A
42.2
14.0
11.2
33.3
44.5
22.4
16.8
11.3
10.3
N/A N/A
N/A
N/A
N/A
BetaE
24.6
5.9
53.0
13.0
11.4
37.6
47.5
24.1
14.3
12.2
8.5
5.1
7.8
10.0
3.1
3.5
CQD
28.4
N/A
60.0
16.5
10.4
40.4
49.6
28.6
20.8
16.8
12.6
N/A
N/A
N/A
N/A
N/A
FuzzQE
29.3
8.0
58.1
19.3
15.7
39.8
50.3
28.1
21.8
17.3
13.7
8.3
10.2
11.5
4.6
5.4
ENeSy
29.4
9.8
59.0
18.0
14.0
39.6
49.8
29.8
24.8
16.4
13.1
11.3
8.5
11.6
8.6
8.8
5.2 Trained with Link Prediction
Since meaningful FOL queries are usually not available in real scenes, we first report the results of models which are trained with only /p query data, which also can be seen link prediction task. The MRR results are shown in Table 1.
As the table shows, compared with pure embedding methods like GQE, Q2B, and BetaE, the performance of ENeSy significantly improves. Since the operators except for the projection of these methods parameterized, the ability to handle complex queries is limited, but our model can be generalized to more complex query structures. Even though, the improvement on /p query proves the advantages of generalizing other KG embedding models solve this problem. Compared with FuzzQE, which could also be trained with link prediction, our model improves the average MRR of EPFO by about 2.6 (relatveyy 11.9%) on FB15K-237 and 1.6% (relatively 5.9%) on NELL-995. For the queries with negation, ENeSy provides more absolute improvement, which is (relatiyy 24.6% and 2. (relatively 28. 3)) on FB15K-237 and NELL-995. We believe the reason for this enhancement is that the symbolic representation can indicate the probability of each entity better.
5.3 Trained with Complex Query
The results of models trained with query data are also reported in Table 2. Compared with the average MRR in Table 1, the performance of most baselines improves lot because of the labeled data of complex queries, but ENeSy still achieves the best performance on the average MRR EPFO query and negation query. Moreover, model could get closer results with different training settings. Specifically, the average MRR decreases by about 4.7% and 2.4% for EPFO and 5% and 4.1% for negation queries on FB155-233 and NELL-995 respectively when trained with link prediction. For FuzzQE, the results decline about 9.9%, 7.5% for EPFO, and 22.4%, 8.8% for query with negation on FB15K-233 and NELL-995, respectively. This proves the stronger robustness of our methods. Note that though the 2p/3p results seem to be worse than FuzzQE, we can replace RotatE with the projection operator of FuzzQF, and the results should be the same in theory. All the models do not get good results on the negation query. We think it's because, after the negation operation, most of the entities in KG will be included, which makes the reasoning hard. Maybe better way only considering the entities which belong to the same type as the entities before the negation operation.
5.4 Analysis of ENeSy
To get deep insights into the neural and symbolic reasoning part of ENeSy, we investigate their impact in this section. In what follows, we first explore how symbolic affects the embedding results. We then examine the influence of embedding on symbolic. Finally, the effect of ensemble prediction is discussed. The results are based on FB 15K-2377, and the situation on NELL-995 is similar.

---

MRR
KGE
%
MRR
Travasse ENeRys
MRR
EN
ENeS/s
ENesy
42.
130
31.2
38.2
30
20
20
131
105
10
101
75
25
7.1
33
100
81
01 01
11(1) 2u(2) 2p(2)
up(3) 3p(3)
93
93
9.
(a)
(b)
Figure 4: The figure of model analysis. (a): The MRR results and increase scale of ENeSy g and pure KGE models. The query types sorted by the query length. (b): The average MRR results of E and traversing. The queries are grouped by their operator. 93 includes all of the query type, 9^ includes 2i/3//pi/ip. 9v includes 2u/up and q- includes Ein/iin/ip/ppinpput (c): The average MRR results of ENeSy p. ENeSys and ENeSy, queries are grouped in the same way as (b).
5.4.1 Q1: Do symbolic results assist neural reasoning in cascading error?
We compare the pure KE embedding model, which is RotatE in this experiment, with the embedding results of ENeSy without ensemble using, which we denote as ENeSy g. For fairness of training data, after the KGE model has converged which trained with /p query, ENeSy, 1p data rather than complex query data based on the KGE model. Since the KGE method could not support logical operators, the results of Ip/2p/3p query and the 2u/up with DNF are reported. The MRR results and the ratio of improvement are shown in Figure 4 (a). The query types are listed below the horizontal axis and we sort them by the length of the query which is the longest distance from the anchor nodes to the target node in the computation graph, and it's marked after the query type. Based on the similar performance on Ip query, the MRR results of more complex queries significantly improve with query length increases. This comparison demonstrates that the cascading error, which is the main limitation of multi-hop embedding reasoning, has been alleviated with the symbolic assistant.
trained with
5.4.2 Q2: Does embedding results assist symbolic reasoning in KG incompleteness?
The symbolic MRR results of ENeSy without ensemble, denoted ENeSy g. and pure symbolic results are shown in Figure (b). We divided the query types into four groups according to the operator they have. Since we only evaluate the generalization ability of models with answers that could not be found by simply traversing KG, the traversing results are nearly zero (since the result is MRR, the number won't be an absolute zero). while the ENeSy, achieve better results than most baselines. The reason for this significant improvement from zero to almost SOTA performance is in the entangled process, ENeSy successfully captures the information from embedding which makes symbolic results include the answers that could not be obtained directly. In summary, the experiment proves that the KG incompleteness problem for symbolic reasoning can be solved in our framework.
5.4.3 Q3: Is ensemble prediction of neural and symbolic results useful?
Ensemble prediction enables us fuse the symbolic and reasoning results. To verify its effectiveness, we compare the performance of ENeSy v. E and ENeSy trained with Ip queries. The MRR results are shown Figure 4 (c). The queries are grouped in the same way as Q2. As the figure illustrates, all the results of different group queries improve with ensemble using. Specifically, the average MRR of the different groups increased by about 14.0%, 15.3%, 4.0%, 28.6% and 2.3%, 1.3%, 2.9%, 8.0% compared with ENeSy g and ENeSys, respectively, which certifies that the two parts could not only enhance each other in the reasoning process but also can be combined in the final results.

---

Conclusion
In this paper, we proposed ENeSy, a neural-symbolic entangled framework for answering complex logical queries over KGs. This model could generalize any embedding methods to the complex query and use the symbolic reasoning results to alleviate cascading error, while the symbolic part also benefits from neural reasoning to solve the problem of KG incompleteness. The ENeSy supports all the FOL operations and can trained with only link prediction tasks, Experimental results show that our model achieves state-of-the-ar performances in answering FOL queries with strong robustness, and each part tightly entangled to enhance the other.
Acknowledgements
This work is funded by NSFCUI192022/9118400 1846204.
10

---

References
[1] Erik Arakelyan, Daniel Daza, Pasquale Minervini, and Michael Cochez. Complex query answer- ing with neural link predictors. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[2] Jiaxin Bai, Zihao Wang, Hongming Zhang, and Yangqiu Song. Query2particles: Knowledge graph reasoning with particle embeddings. CoRR, abs/2204. 12847, 2022.
[3] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250, 2008.
[4] Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Jason Tsong- Li Wang, editor. Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pages 1247-1250. ACM, 2008.
[5] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795, 2013.
[6] Xuelu Chen, Ziniu Hu, and Yizhou Sun. Fuzzy logic based logical query answering on knowledge graph. CoRR, abs/2108 0.0390,, 2021.
[7] William W. Cohen, Fan Yang, and Kathryn Mazaitis. Tensorlog: A probabilistic database implemented using deep-learning infrastructure. J. Artif. Intell. Res., 67:285-325. 2020.
[8] Tim Dettmers, Pasquale Minervini. Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[9] William I.. Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. Embedding logical queries on knowledge graphs. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018 NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2030-2041 2018.
[10] Yankai Lin, Zhiyuan Liu, Huan-Bo Luan, Maosong Sun, Siwei Rao, and Song Liu. Modeling relation paths for representation learning of knowledge bases. In Lluís Mårquez, Chris Callison- Burch, Jian Su, Daniele Pighin, and Yuval Marton, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 705-714. The Association for Computational Linguistics, 2015.
[11] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39 41, 1995.
[12] Tom M. Mitchell, William W. Cohen, Estevam R. Hruschka Jr., Partha Pratim Talukdar, Justin Betteridge, Andrew Carlson, Bhavana Dalvi Mishra, Matthew Gardner, Bryan Kisiel, Jayant Krishnamurthy, Ni Lao, Kathryn Mazaitis, Thahir Mohamed, Ndapandula Nakashole, Em- manouil A. Platanios, Alan Ritter, Mehdi Samadi, Burr Settles, Richard C. Wang, Derry Wijaya, Abhinay Gupta, Xinlei Chen, Abulhair Saparov, Malcolm Greaves, and Joel Welling. Never- ending learning. In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2302-2310. AAAI Press, 2015.
[13] Thomas Rebele, Fabian M. Suchanek, Johannes Hoffart, Joanna Biega, Erdal Kuzey, and Gerhard Weikum. YAGO: A multilingual knowledge base from wikipedia, wordnet, and geonames. Paul Groth, Elena Simperl, Alasdair J. G. Gray, Marta Sabou, Markus Krötzsch, Freddy Lécué, Fabian Flöck, and Yolanda Gil, editors, The Semantic Web ISWC 2016 15th International Semantic Web Conference, Kobe, Japan, October 17-21, 2016, Proceedings, Part II, volume 9982 of Lecture Notes in Computer Science, pages 177-185 2016.

---

[14] Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In 8th International Conference on Learning Represen- lations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[15] Hongyu Ren and Jure Leskovec. Beta embeddings for multi hop logical reasoning in knowledge graphs. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
[16] Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. DRUM: end-to-end differentiable rule mining on knowledge graphs. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, 2019.
Canada, pages
[17] Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph em- bedding by relational rotation in complex space. In 7th International Conference on Learning Representations, 2019.
[18] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases, In EMNLP, pages 1499-1509. The Association for Computational Linguistics, 2015.
[19] Théo Trouillon, Johannes Welbi, Sehastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference Machine Learning, pages 2071-2080, 2016.
[20] Zhen Wang, Jianwen Zhang, Jianlin Feng. and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
[21] Han Xiao, Minlie Huang, Yu Hao, and Xiaoyan Zhu. Transg: A generative mixture model for knowledge graph embedding. CoRR, 0111550905488 2015.
[22] Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learning method for knowledge graph reasoning. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Demmark, September 9-11, 2017, pages 564-573. Association for Computational Linguistics, 2017.
[23] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations, 2015.
[24] Fan Yang. Zhilin Yang, and William W. Cohen. Differentiable learning of logical rules for knowledge base reasoning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
[25] Wen Zhang, Bibek Paudel, Liang Wang, Jiaoyan Chen, Hai Zhu, Wei Zhang, Abraham Bernstein, and Huajun Chen. Iteratively learning embeddings and rules for knowledge graph reasoning. In The World Wide Web Conference, pages 1366-2377 ACM, 2019.
[26] Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. Cone: Cone embeddings for multi-hop reasoning over knowledge graphs Mare Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 19172-19183, 2021.
12

---

[27] Zhaocheng Zhu, Mikhail Galkin, Zuobai Zhang, and Jian Tang. Neural-symboll models for logical queries on knowledge graphs. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 27454-27478 PMLR, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section and abstract.
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [No] There is no potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
5.2, 5.3 and 5.4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 5.2, 5.3 and 5.4.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section
5.1.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 5.1
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 5.1
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tBVZkcZXL7f", "reviewer": "Reviewer_QWSg", "summary": "This paper proposes a neural symbolic entangled framework for answering complex logical queries over knowledge graph. The framework firstly represents entities and relations via a neural projection and a symbolic projection. Then the symbolic based logical operator is defined to combine the neural and symbolic projection. Results show the proposed framework outperforms baselines on complex query answering benchmarks.", "questions": "N/A", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n-\tThe proposed neural symbolic entangled framework is novel, which combines neural representation and symbolic representation of entities and relations.\n-\tThis paper is well written and easy to follow\n\n\nCons:\n-\tThe proposed approach does not fully outperform the strongest baseline FuzzQE on benchmark datasets\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L181BnFYFmI", "reviewer": "Reviewer_7Bea", "summary": "This paper proposes a neural-symbolic approach (called ENeSy) for answering FOL-based queries over KGs. The main promise of the proposed approach lies in its ability to alleviate the problem of cascading error as it existed in previous approaches and also the incompleteness of KGs. \n\nAs claimed in the paper, existing query embedding methods handle these complex queries over incomplete KB but they don’t pay any attention to intermediate entities which are important for answering multi-hop queries. The proposed approach ENeSy seems to bridge some of these gaps.\n\nOverall, ENeSy comprises two parts (neural and symbolic)  to represent a given KG.\n- For the neural part, it uses rotatE to embed entities and relations into a complex Space ${\\mathbb{C}}^k$.\n- For the symbolic part, an entity and a relation are encoded as a one-hot vector and an adjacent matrix, respectively. \n- The key operations involved in answering complex queries are _projection, intersection, union,_ and _negation_.  ENeSy proposes an entangled version of each of these operations in Equations (5) and (6). \n\nThe experiments are performed on two datasets (FB15K-237 and NELL-995). \tThe focus of the experiments is on answering FOL queries on incomplete KGs so as to evaluate the ability of models to obtain the answers that could not be discovered by traversing KGs. The experiment results (Table 1 and 2) and the insights (Figure 4) established the merits of the proposed ENeSy over the baselines. ", "questions": "- In lines 162 and 168, why the word _possibility_ is used and not the word _probability_?\n- How does MLP(.) function look like in equation (5)? Does it map a vector to a scalar? If it is a vector then I am not sure how this vector is getting multiplied with the other two vectors present in equation (5)? Is it an inner product between two vectors? Notations are a bit confusing as there is no transpose sign. \n- In equation (7), should it be $v_q$ instead of $v$?\n- For baselines, did you recompute the numbers or picked from the existing papers? If picked from the existing papers, whether the existing papers followed the same evaluation steps as mentioned in lines # 205-217? Because, otherwise, it will not be an apple-to-apple comparison.\n", "limitations": "There is not much of a discussion on the limitations of the proposed approach. I would have loved to see some discussion on the memory footprints and time complexity-related issues and how they compare with baselines. Also, it may be worth discussing if there is any class of queries within FOL that are not being handled properly by your approach? By looking at Figure 4(b), it looks like negation queries are relatively hard to deal with. You can have some discussion about that as well.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\n\nThe idea of fusing neural and symbolic representations of the answer entities to cope with the KG incompletion is certainly novel. Neural and symbolic representations seem to be helping each other and thereby boosting the overall performance. \n\n\n**Quality**\n\nThe paper is really very well written. The entanglement ideas are novel. The experimentation evaluation plan laid out in lines # 205-217 is very clearly written and makes perfect sense.  The experiments seem to be convincing in terms of proving the merits of the proposed approach. The only reservation I have is that the experiments are performed only on two datasets. Authors could have included one more dataset. \n\n\n**Clarity**\n\nBy and large paper is easy to read and follow through. Below are some questions regarding the clarity of the matter.\n- In line # 116, the symbol S’ seems a bit inconsistent. \n- In line # 139, the spelling of Hadamard is incorrect.\n- In lines 162 and 168, why the word _possibility_ is used and not the word _probability_?\n- How does MLP(.) function look like in equation (5)? Does it map a vector to a scalar? If it is a vector then I am not sure how this vector is getting multiplied with the other two vectors present in equation (5)? Is it an inner product between two vectors? Notations are a bit confusing as there is no transpose sign. \n- In equation (7), should it be $v_q$ instead of $v$?\n- Caption of Table 1 seems to have typos. Please read it carefully once again and fix any typos.\n- Just to make the paper self-contained, better explain the convention used behind assigning titles to the computation graphs in Figure 3. For example, in the case of 2i, why can’t we call it 2p2i. or why “inp” can’t be titled as “pin”. If need be, add this in the appendix. \n- In Tables 1 and 2, you should also tag the count of each query type in your test set over which these numbers are reported.\n\n**Significance**\n\nIf we were to believe the numbers shown in Tables 1 and 2 then it clearly proves a significant improvement over the baselines methods. However, as I mentioned before, it would have been nicer to run experiments on one more dataset. Nonetheless, what is presented is still impressive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2Ek3z2_BMz", "reviewer": "Reviewer_3t6R", "summary": "In this paper, the authors present a new method to answer complex logical queries on Knowledge Graphs. They include symbolic logic to answer FOL queries and take into account intermediate answers to the queries. By adding the neural symbolic model, the paper aims to tackle some of the main disadvantages of the best-known embedding methods: cascading error and KG incompleteness.", "questions": "1. Have you tested the entangling framework with other embedding methods? Why did you select rotatE?\n2. As far as I understand, the entangling method only adds more possible answers to a query, have you checked in which percentage this number increases?\n3. Have you manually checked some of the queries you are using for training and testing purposes?\n", "limitations": " \nThe main advantage of this new entangling framework is that it can be added to any other existing framework. However, how much improvement it actually provides, in general, is still unclear.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors clearly define their methodology and experimentation.\n- The paper combines previous technologies in a novel way that achieves better results. The experiments prove the proposed method achieves better results as the queries get more complex.\n- The proposed entangling framework can include any other embedding method. Thus, in theory, it adds a performance improvement to all previous KGE methods.\n \nWeaknesses\n- My main concern is the fact that the entangling framework adds more possible answers to the queries. I would like to see a comparison of how many of the possible answers added by the symbolic logic are actually correct, or else it is highly increasing the number of False Positives.\n- According to the results, everything points in the direction that any other embedding method can have similar performance improvements. But I am missing some experimentation in that direction to show how each well-known method is improved by adding this entangling framework.\n- Important baselines are missing. On many of the metrics, CQD[a] can achieve better results than that of the proposed model. The recent model GNN-QE[b] outperforms the proposed model almost on all metrics. These two models are closely related to this work but not discussed. \n- Performances on FB15k are missing.\n\n[a] Complex query answering with neural link predictors, ICLR 2021\n[b] Neural-Symbolic Models for Logical Queries on Knowledge Graphs, ICML 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept