Relation-Aware Question Answering for Heterogeneous Knowledge Graphs
Authors: Haowei Du
Quzhe Huang
Chen Li
Chen Zhang
Yang Li
Dongyan Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Question Answering
Knowledge Graph
Heterogeneous

Abstract: Multi-hop Knowledge Base Question Answering(KBQA) aims to find the answer entity in a knowledge graph (KG), which requires multiple steps of reasoning. Existing retrieval-based approaches solve this task by concentrating on the specific relation
at different hops and predicting the intermediate entity within the reasoning path. 
However, these models fail to utilize information from head-tail entities and the semantic connection between relations to enhance the current relation representation, which undermines the information capturing of relations in KGs. To address this issue, we construct a \textbf{dual relation graph} where each node denotes a relation in the original KG (\textbf{primal entity graph}) and edges are constructed between relations sharing same head or tail entities. Then we iteratively do primal entity graph reasoning, dual relation graph information propagation, and interaction between these two graphs. In this way, the interaction between entity and relation is enhanced, and we derive better entity and relation representations. Experiments on two public datasets, WebQSP and CWQ, show that our approach achieves a significant performance gain over the prior state-of-the-art.

Relation-Aware Question Answering for Heterogeneous Knowledge Graphs
Haowei Du Quzhe Huang Chen Li Chen Zhang! Yang Li Dongyan Zhao
'Wnnxuuan Institute of Computer Technology, Peking University
The National Key Laboratory of Cross-Media General Artificial Intelligence
3 Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China Institute for Artificial Intelligence, Peking University Ant Group duhaowei@stu. pku. edu. cn, (huangquzhe, zhangch, zhaodongyan]{@pk edu. cn wenyou. www.ntgroru......omommmmmo com, 1y2001 7 70 iwaia-baaneeee inc. com
Abstract
Thrregh which overmes of Sabal Nigit flo?
Acener city
Saini River
Multi-hop Knowledge Base Question Answer- ing(KBQA) aims to find the answer entity in knowledge graph (KG), which requires mul- tiple steps of reasoning. Existing retrieval- based approaches solve this task by concen- trating on the specific relation at different hops and predicting the intermediate entity within the reasoning path. During the reasoning pro- cess of these methods, the representation of relations are fixed but the initial relation rep- resentation may not be optimal. We claim they fail to utilize information from head-tail entities and the semantic connection between relations to enhance the current relation rep- resentation, which undermines the ability to capture information of relations in KGs. To address this issue, we construct a dual rela- tion graph where each node denotes rela- tion in the original KG (primal entity graph) and edges are constructed between relations sharing same head or tail entities. Then we iteratively do primal entity graph reasoning, dual relation graph information propagation, and interaction between these two graphs. In this way. the interaction between entity and relation enhanced, and we derive better en- tity and relation representations. Experiments on two public datasets, WebQSP and CWQ, show that our approach achieves a significant performance gain over the prior state-of-the- art. Our code is available on https: / thub. commyyanmenxue/RRe /RAH-KBQAA
Contancd by
Guanca
Consiee
Type
Coctzinal by
Thpp
Couztry
Nuk
Niger
Carzital
Cerivrn
Subdl
Book szbjod
Alrici
Type
Certioee
Waler Macic
Figure 1: An example from WebQSP dataset. The yel- low arrow denotes the reasoning path and the orange arrows denote two constraints "of the Sahel" and "coun- try".
of reasoning are required to identify the answer en- tity (Atzeni et al., 2021; Mavromatis and Karypis, 2022). Such complicated task is referred to as multi-hop KBQA.
There are two groups of methods in KBQA; in- formation retrieval (IR) and semantic parsing (SP), SP-based methods need executable logical forms as supervision, which are hard to obtain in real scenes. IR- based approaches, which retrieves relevant enti- ties and relations establish the question specific subgraph and learn entity representation by reason- ing over question-aware entity graph with GNN (Zhou et al., 2020), are widely used to solve multi- hop KBQA
et al., 2020; He et al., 2021; Shi et al., 2021; Sen et al., 2021). These systems are trained without logical forms, which is more realistic setting as these are hard to obtain. We fo- the retriever-reasoner approach to multi-hop KBQA without the addition of logical form data.
1 Introduction
Knowledge Base Question Answering (KBQA) is a challenging task that aims to answer natural lan- guage questions with a knowledge graph. With the fast development of deep learning, researchers leverage end-to-end neural networks (Liang et al., 2017; Sun al., 2018) to solve this task. Recently there has been more and more interest in solv- ing complicated questions where multiple steps
During the reasoning process of existing meth- ods, the representation of relations are fixed but the initial relation representation may not be opti- mal. We claim the semantic connection between relations which is co-relation and the semantic in- formation from head and tail (head-tail) entities to build relation representation is neglected.
We take one case in WebQSP dataset (Yih et al., 2015) as an example, where the question
"Corresponding Author

---

is "Through which countries of the Sahel does the Niger river flow?". In Figure 1, to answer the question, the model has to hop from the topic entity "Niger River" by the reasoning path "Con- tained_by" byd On the other hand, the candidate en- tity "Niger"(country) should connect to "Sahel" with relation "Contains" to satisfy the constraint "of the Sahel" to be the answer. So one crucial point for the model to retrieve the answer is un- derstand the meaning of relation "Contained_ie by" and "Contains". On one hand, the relation "Con- tained_by by" and "Contains" have head-tail entities of type "location" " like "Niger River", "Guinea", "Africa" and so on. The semantic information of entity type narrows down the meaning scope of relation "Contained_by" and "Contains" into loca- tive relations. On the other hand, the massive intersection between head-tail entity sets of rela- "ContCnte by" and "Contains" implics the high semantic connection of these two relations, and we assume the representation learning for one of relations can enhance the other. If the model discovers the reverse relationship between "Con- tained_ by" and "Contains", the understanding of one relation can help construct the representation of the other.
folds:
We are the first incorporate semantic in- formation of head-tail entities and co-relation among relations to enhance the relation repre- sentation for KBQA.
2. We create an iterative unified reasoning frame- work on dual relation graph and primal en- tity graph, where primal graph reasoning, dual graph information propagation, and dual- primal graph interaction proceed iteratively. This framework could fuse the representation of relations and entities in deeper way.
3.
Our approach outperforms existing methods two benchmark datasets this field, be- coming the new state-of-the-art of retrieval- reasoning methods.
Related Work
Over the last decade, various methods have been developed for the KBQA task. Early works utilize machine-learnee or hand-crafted modules like en- tity recognition and relation linking to find out the answer entity (Yih et al., 2015; Berant et al., 2013; Dong et al., 2015; Ferrucci et al., 2010). With the popularity of neural networks, recent researchers utilize end-to-end neural networks to solve this task. They can be categorized into two groups: semantic parsing based methods (SP-based) (Yih et al., 2015; Liang et al., 2017: Guo et al., 2018; Saha et al., 2019) and information retrieval based methods (IR-based) (Zhou al., 2018; Zhao et al., 2019; Cohen et al., 2020; Qiu et al., 2020). SP- based methods convert natural language questions into logic forms by learning a parser and predict- ing the query graph step by step. However, the predicted graph is dependent on the prediction of last step and if one step the model inserts the incorrect intermediate entity into the query graph, the prediction afterward will be unreasonable. SP- based methods need the ground-truth executable queries as supervision and the ground-truth exe- cutable query forms bring much more cost to anno- tate in practical. IR- bR-based methods first retrieve rel- evant Knowledge Graph(KG) elements to construct the question-speeiff subgraph and reason over it to derive answer distribution by GNN. Zhang et al. (2018) introduce variance reduction in retrieving answers from the knowledge graph. Under the set- ting of supplemented corpus and incomplete knowl- edge graph, Sun et al. (2019) propose PullNet to
To utilize the semantic information from head- tail entities and the co-relation among relations, we propose our framework of unified reasoning of primal entity graph and dual relation graph. We construct the dual relation graph by taking all rela- tions in the primal entity graph as nodes and linking the relations which share the same head or tail enti- ties. In each hop, first we reason over primal entity graph by attention mechanism under question in- structions, and then we do information propagation on dual relation graph by GAT (Veličković al., 2017) to incorporate neighbor relation information into the current relation, at last we merge the seman- tic information of head-tat tail entities by the assump- tion of TransE (Bordes et al., 2013) to enhance the relation representation. We conduct the iteration for multiple rounds to ensure that the representation of relations and entities can stimulate each other. We do experiments on two benchmark datasets in the field of KBQA and our approach outperforms the existing methods by 0.8% Hits @ score and 1.7% Fl score WebQSP, 1.5% Hits@I score and 5.3% F1 score in CWQ (Talmor and Berant, 2018), 8), becoming the new state-of-the-ard
Our contributions can be summarized as three

---

learn what retrieve from a corpus. Different from EmbedKGQA (Saxena et al., 2020), which utilizes KG embeddings over interaction between the question and entities to reduce KG sparsity, we apply TransF incorporating the head-tail entities information to enhance the relation representation. Shi et al. (2021) propose TransferNet to support both label relations and text relations in unified framework, For purpose of enhancing the supervi- sion of intermediate entity distribution, He al. (2021) adopt the teacher-student network in multi- hop KBQA. However, ignores the information from head-tail entities and semantic connection with neighbor relations to build the representation of current relation.
rencody
graph
iteration
Dusl-primal Interaction
Veccoder
grapa
Dual graph propagation
srrlyy dwar Idkazo updaing
Figure Method Overview. Our approach consists of four main components: question encoder, primal entity graph reasoning, dual relation graph propagation, and dual-pric primal graph interaction. The four components communicate mutually in an iterative manner.
Methodology
3 Preliminary
4.1 Overview
this part, we introduce the concept of the knowl- edge graph and the definition of multi-hop knowl- edge base question answering taBQA..
As illustrated in Figure 2 our approach consists of 4 modules: question instruction generation, primal entity graph reasoning, dual relation graph informa- tion propagation, and dual-primal graph interaction. We first generate n instructions based on the ques- tion, and reason over the primal entity graph under the guidance of instructions, and then we do infor- mation propagation over the dual relation graph by GAT network, finally we update the representation of relations based on the representation of head-tail entities by TransE assumption.
3.1 KG
A knowledge graph contains a series of factual triples and each triple (fact) is composed of two entities and one relation. A knowledge graph can be denoted G (e,,r,e)le, E E,r R}. where G denotes the knowledge graph, E denotes the entity set and R denotes the relation set. fact (c, vrrce e') means relation exists between the two entitics E and e'.
4.2 Question Encoder
The objective of this component is to utilize the question text to construct a series of instructions [ik) where in denotes the k-th instruction to guide the reasoning over the knowledge graph. We encode the question by SentenceBERT (Reimers and Gurevych, 2019) to obtain a contextual rep- resentation of each word in the question, where the representation of j-th word denoted as hj. We utilize the first hidden state ho corresponding to "(CLS)" token as the semantic representation of the question,i.e.g = he- We utilize the attention mechanism to attend to different parts of the query and choose the specific relation at each step. Fol- lowing He et al. (2021), we construct instructions as follows:
3.2 Multi-hop KBQA
Given natural language question q that is answer- able using the knowledge graph, the task aims to find the answer entity. The reasoning path starts from topic entity(i.e., entity mentioned in the ques- tion) to the answer entity. Other than the topic en- tity and answer entity, the entities in the reasoning path are called intermediate entity. If two entities are connected by one relation, the transition from one entity to another called one hop. In multi- hop KBQA, the answer entity is connected to the topic entity by several hops. For each question, a subgraph of the knowledge graph is constructed by reserving the entities that are within hops away from the topic entity to simplify the reasoning pro- cess, where n denotes the maximum hops between the topic entity and the answer. To distinguish from the dual relation graph in our approach, we denote this question- aware subgraph as the primal entity graph.
i(k)
(10
hj
(1)
(A)
softmax, ( (Q( hy))
(2)
qth (k) wik ikk 1) (k)
(3)

---

where 1 denotes the question length, i is initial- ized as q, W W(k), b(k) are learnable parame- ters.
gation on dual relation graph:
p(k)
Σ FEN(T)
(k-1)
(7)
4.3 Primal Entity Graph Reasoning
exp (r ( att
Oyr
(8)
EVENG) exp (r W Watt!")
The objective of this component to reason over entity graph under the guidance of the instruction obtained by the previous instructions generation component. First, for each fact (e, e') in the subgraph, we learn matching vector between the fact and the current instruction ;(k);
F(k) = a (W, ( 1) + b,)
(9)
where F denotes the representation of neighbour lation r, f(k) denotes the representation of relation after integrating neighborhood relation informa- tion, (r)) denotes the neighbor set of relation r in dual relation graph, Watt. Wr and by are parame- ters to learn.
k)) (e.r.e')
(iikk) C WRF)
(4)
where denotes the embedding of relation and WR R are parameters to learn. Then for each entity e in the subgraph, we multiply the activating prob- ability of each neighbor entity e' by the matching (k)
4.5
Dual-Primal Graph Interaction
4.5.1 Entity-Aware Relation Updating
The objective of this part is to incorporate the head- tail entity representation from the primal graph to enhance the representation of relations. Based on Transl method, believe the subtraction between head-tail entity representation composes seman- tic information of the relation. So we use head entity representation ésait to minus tail entity rep- resentation éhead as the representation Efact. We encode the information of one relation from adja- cent entities by passing all the fact representation fact which contains it average pooling layer. Then we concatenate the neighbor entity informa- tion along with the previous representation and project into the new representation. The process above can be described as follows:
vector m (erre') I and aggregate them as the representation of information from its neighbor- hood:
ē(k)
Σ (e,ne')EG
(k-1) (k) Pe
(5)
(e.x,e))
The activating probability of entities pe rived from the distribution predicted by decoding module the previous step. We concatenate the previous entity representation with its neighbor- hood representation and pass into a MLP network to update the entity representation:
(k-1) is de-
(k) (k 1)
(k 1)
(kk (L
(6)
Pact
etail
h
(10)
(k)
((k) Ejarter: efct
(11)
where @kk) denotes the representation of entity after incorporating neighbor entity information.
(k) (F + be)
(12)
where F, (fls (c, E G. E F, c E}. n denotes the number of facts containing re- lation r, a denotes activating function, W. and be are parameters to learn.
4.4 Dual Relation Graph Propagation
The objective of this part is to model and learn the connection between relations. To incorporate the semantic information from neighborhood relations to enhance the representation of current relation, we construct a dual graph for reasoning. The graph is constructed as follows: each node in dual graph denotes one relation in the primal graph. If two relations in primal graph share same head or tail entity, we believe they should have semantic com- munication and we connect them in the dual graph. We utilize GAT network to do information propa-
4.5.2 Relation-Aware Entity Updating
In this part, we capture the information from ad- jacent relations to update entity representation by projecting the relation representation into entity space. Considering the semantic roles of head en- tity and tail entity in one relation are different, we
adopt two separate projectors to handle the rela- tions which hold the entity e as head entity or tail

---

entity respectively:
Datasets
train
dev
test
entities
WebQSP 2,848
250
1,639
1,429.8
@(k)
Σ
W Whadd
Σ
W Wtairr' ntaii
CWQ
27,639
3,519
3,531
1,305.8
nhead
rEllhoad
Table 1: Statistics about the datasets. The column "en- tities" denotes the average number of entities in the subgraph for each question
(13)
((k) = =o(MPi(el))
(14)
where Rhoad = {rl(e, r, E G}. Rtail {ri(e',r,e) E G), n head and n lail denote the num- ber of relations in Ichood and Reail W Whead and Wtall two separate groups of parameters to learn.
WebQSP
CWQ
Models
Hits
Hits@1
FI
GraftNet
67.8
62.8
32.8
PullNet
68.1
45.9
EmbedKGQA
66.6
TransferNet
71.4
48.6
4.6 Decoding and Instruction Adaption
Rigel
73.3
48.7
NSM
74.3
67.4
48.8
44.0
For simplicity we leverage MLP to decode the en- tity representation into distribution:
SQALER
76.1
REAREV
76.4
70.9
52.9
p (k) softmax(MLP((
(15)
UniKGQA
77.2
72.2
51.2
49.0
Ours
77.2
72.6
54.4
49.3
where ple denotes the activating probability of the k-th intermediate entity, and each column of E(k) is the updated entity embedding (lk) E IRd. Con- sidering the number of candidate entities is large and the positive sample is sparse among the candi- dates, adopt the Focal loss function (Lin al.. 2017) enhance the weight of incorrect prediction. Moreover we update the representations of instruc- tions to ground the instructions to underlying KG semantics and guide the reasoning process in an it- erative manner following Mavromatis and Karypis (2022). The representations of topic entities are incorporated by gate mechanism:
Table 2: Results on two benchmark datasets compared with several competitive methods proposed in recent years. The baseline results from original papers. Our approach significantly outperforms NSM, where p-values of Hits@ and are 0.01 and 0.0006 on We- bQSP, 0.002 and 0.0001 on CWQ.
(Yih et al., 2015) includes 4,327 natural language questions that are answerable based Freebase knowledge graph (Bollacker et al., 2008), which contains millions of entities and triples. The answer entities of ques- tions in WebQSP are either 1 hop or 2 hops away from the topic entity. Following Saxena al. (2020), we prune the knowledge graph to contain the entities within 2 hops away from the mentioned entity. On average, there are 1,430 entities in each subgraph.
WeboueettionsSP
(k) = (1 (k) & i(k) + g (k)
c (k)
(16)
h (k) (i( he (k) i(k) he
(k)
1k))
he
(k)
(17)
h
(k)
Σ 1E(e)
k)
(18)
Complex WebQuestions(CWO) (Talmor and Berant, 2018) is expanded from WebQSP by ex- tending question entities and adding constraints to answers. It has 34.689 natural language questions which are up 4 hops of reasoning over the graph. Following Sun al. (2019), we retrieve subgraph for each question using PageRank algorithm. There are 1,306 entities in each subgraph on average.
where [c], denotes the topic entities in the ques- tion, W; denote learnable parameters and (8) dente the output gate vector computed by a standard GRU (Cho et al., 2014).
5 Experiments
5.1 Datasets
5.2 Implementation Details
We evaluate our method on two widely used datasets, WebQuestionSSP and Complex WebQues- tion. Table shows the statistics about the two datasets.
We optimize models with Adam optimizer, where the batch size is set to and the learning rate set to 7e-4. The reasoning steps are set to 3.

---

WebQSP
CWQ
TransferNet (Shi al., 2021) unifies two forms of relations, label form and text form reason over knowledge graph and predict the distribution of entities.
Models
Hits @
Fl
Hits@ 1
Fl
P-Transfer
74.6
76.5
58.1
RnG-KBQA
75.6
DECAF*
74.7
49.8
50.5
54.7
(Sen et al., 2021) utilizes differentiable knowledge graph to learn intersection of entity sets for handling multiple-entity questions.
Rigel
DECAF(1 80.7
77.1
67.0
79.0
Ours
77.2
72.6
54.4
49.3
Table 3: Comparing with recent semantic parsing meth- ods. DeCAF* DeCAR()) denote the DeCAF only with answer supervision and DeCAF with T5-large as backbone.
SQALER (Atzeni et al., 2021) shows that multi- hop and more complex logical reasoning can be accomplished separately without losing expressive power.
The hidden size of GRU and GNN is set 128.
REAREV (Mavromatis and Karypis, 2022) per- form reasoning in an adaptive manner, where KG- aware information is used to iteratively update the initial instructions.
5.3 Evaluation Metrics
For each question, we select set of answer en- tities based on the distribution predicted. We uti- lize two evaluation metrics Hits@1 and F1 that are widely applied in the previous work (Sun et al., 2018, 2019; Wang et al., 2020; Shi et al., 2021; Sax- ena et al., 2020; Sen et al., 2021). Hits@I measures the percent of the questions where the predicted answer entity is in the set of ground-truth answer entities. F1 is computed based on the set of pre- dicted answers and the set of ground-truth answers. @its@I focuses on the top entity and Fl focuses on the complete prediction set.
(Jiang al., 2022) unifies the trieval and reasoning in both model architecture and learning for the multi-hop KGQA.
UniKGQA
We also compare with several competitive SP- based methods as follows:
P- P-Transfer (Cao al., 2021) leverages the valu- able program annotations the rich-resourced KBs external supervision signals to aid program induction for the low-resourced KBs that lack pro- gram annotations.
RnG-KBQA (Ye et al., 2021) first uses con- trastive ranker rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a tailored generation model conditioned on the question and the top- ranked candidates to compose the final logical form
5.4 Baselines to Compare
As an IR-based method, we mainly compare with recent IR-based models as follows:
GraftNet (Sun et al., 2018) uses a variation of GCN to update the entity embedding and predict the answer.
DECAF (Yu et al., 2022) Instead of relying on only either logical forms direct answers, DE- CAF jointly decodes them together, and further combines the answers executed using logical forms and directly generated ones to obtain the final an- swers.
PullNet (Sun et al., 2019) improves GraftNet by retrieving relevant documents and extracting more entities to expand the entity graph.
(Saxena et al., 2020) incorporates pre-trained knowledge embedding to predict an- swer entities based on the topic entity and the query question.
EmbedKGQA
5.5 Results
The results of different approaches are presented in Table 2, by which we can observe the follow- ing conclusions: Our approach outperforms all the existing methods two datasets, becoming the new state-of-the-art of IR-based methods. It is ef- fective introduce the dual relation graph and iteratively do primal graph reasoning, dual graph information propagation, and dual- -oilml graph in- teraction. Our approach outperforms the previous
(He et al., 2021) makes use of the teacher framework to provide the distribution of interme- diate entities as supervision signs for the student network. However, NSM fails utilize the seman- tie connection between relations and information from head-tail entities to enhance the construction current relation representation.
NSM

---

Quantile
0-0.25
0.25-0.5
0.5-0.75
0.75-1
Quantile
0-0.25
0.25-0.5
0.5-0.75
0.75-1
NSM 65.4
70.5
68.2
63.1
NSM
65.6
66.7
69.3
65.6
Ours 71.0
74.0
74.2
71.3
Ours
69.7
72.6
75.5
72.8
Table 4: Average Fl score on 4 groups of cases in WebQSPTabbl 5: Average Fl score on 4 groups of cases in WebQSP
classified by relation number.
classified by fact number.
Quantile
0-0.25
0.25-0.5
0.5-0.75
0.75-1
Quantile 0-0.25
0.25-0.5
0.5-0.75
0.75-1
NSM
71.4
66.5
66.3
63.0
NSM 72.3
68.9
66.8
59.2
Ours
77.0
72.2
72,4
68.9
Ours 77.3
73.4
71.6
68.3
Table 6: Average Fl score on groups of cases WebQSPTable 7: Average F1 score on 4 groups of cases in WebQSP classified by the ratio between relation number and entity classified by the ratio between fact number and entity number.
number.
state-of-the-art REAREV by 0.8% Hits @ score and 1.7% Fl score in WebQSP as well as 1.5% Hits score in CWQ. Compared to the Hits@ metric, the metric considers the prediction of the whole answer set instead of the answer that has the maximum probability. Our performance in Fl shows our approach can significantly improve the prediction of the whole answer set. Our signif- icant improvement the more complex dataset CWQ shows our ability address complex relation information.
ules in our approach and explore our performance with different reasoning steps n.
6.1 Size and Connectivity of Dual and Primal Graph
In this part, we classify the cases of WebQSP dataset into 4 groups respectively based on the quantile of metrics to ensure the same number of different groups: the number of relations, the num- ber of facts, the ratio of relation number by entity number, the ratio of fact number by entity number. The number of relations and facts reflects the size and connectivity of dual relation graph, while the ratio of fact number by entity number reflects the connectivity of primal graph. The connectivity be- tween primal entity graph and dual relation graph is embodied in the ratio of relation number by entity number.
Comparing with SP-based methods in table we can derive the following findings: 1. SP-based methods utilize ground-truth executable queries as the stronger supervision, which will signifi- cantly improves the KBQA performance with IR- based methods. 2. If losing the ground-truth -ouuh exe- cutable queries supervision signal, the performance of SP-based methods will drop by a large degree. The SOTA SP-method DecAF, drops by and 18 Hits @1 score on WebQSP and CWQ with answer as the only supervision. In the setting, our method outperforms DecAF by 3 and 4 Hits @ score on WebQSP and CWQ. The ground-truth executable query forms bring much more cost and difficulty annotate in practical. So the reasoning methods are important in real scenes where the ground-truth logical forms are hard obtain. the ground executable queries are at hand, SP-based methods deserve to be take into consideration.
From Table 3-6, overall, our approach derives better improvement over NSM on cases with larger size and high connectivity of dual relation graph and primal entity graph. With the larger size and higher connectivity, our dual graph propagation module incorporates more neighbor relation infor- mation to enhance the representation of current relation and the dual-primal interaction module ex- ploits more abundant information from head-tail entities into relation representation.
6.2 Evaluation of Relation and Entity Representation
6 Analysis
In this part, we explore the quality of the rep- resentation of relations and entities. By the as-
In this section, we probe our performance on cases with different sizes and connectivity of dual re- lation graph and primal entity graph. Then explore the benefit on relation and entity represen- tation by introducing dual-prine dual-primal graph interaction along with dual graph propagation. Furthermore, we do ablation studies to evaluate different mod-
sumption of TransE, the representation of relation should be translation operating the embedding of its head entity and tail entity, which satisfies Choad + r Stail So we choose the inner product of relation representation and head-tai entity em- bedding subtraction as the metric to evaluate the

---

quality of relation and entity representation:
can see the performance drops by 0.1% score on Hits@ I and 0.6% score FI. It shows our interac- tion between primal and dual graphs incorporates semantic information of head-tail entities which is crucial to represent the corresponding relation.
Score: En EEe (r : (f N EXEMIM
@hhead
where (+) denotes the inner product, N₂ denotes the number of facts in case n and d denotes the dimension of relation representation. The higher score means the relation representation is more rel- evant and consistent with the head-tail entities and the lower score denotes the relation representation space is more orthogonal to the entity space which is not ideal.
Can we use co-occurrence statistics of relations to replace the attention mechanism in dual graph propagation? In this ablation, we employ co-occurrence of relations (Wu al., 2019) re- place attention mechanism in Eq. 8. To be specific, let H, and T, denote the head-tail entity set re- lation r; respectively, we compute the co-relation weights of and rj as follows:
By Table 8, our approach significantly improves the Transl--score over NSM in cases with different sizes and connectivity of relation graph. It demon- strates our representations of relations and entities are much more corresponding to the semantic re- lationship between them. Incorporating head-tail entities information and the semantic co-relation between relations enhance the construction of re- lation representation, which stimulates the entity representation conversely. Moreover, the semantic consistency of our relation representation bet- ter with a larger size and higher connectivity of relation graph. illustrates the efficiency of our dual graph propagation and dual-primal graph in- teraction module for complex and multi-relational KGs.
Oig (H.UT)) n (HAUT)N
(19)
By table 9, 0.9% score on F1. shows introducing attention mechanism to weigh the co-relation between rela- tions more flexible and efficient than freezing the weights as the co-occurrence statistics. The weights of connection between relations should update as the reasoning process proceeds.
see the performance drops by
6.4 Different Reasoning Steps
For baselines which apply multiple hops reason- ing, the reason step ranges from 2 to 4. We ex- periment on CWQ dataset with hops from 2 to 4 and freeze all other hyperparameters. Table 10 shows we gain stable improvement over the base- line with diverse reasoning steps. It demonstrates the robust efficiency of our iteratively primal graph reasoning, dual graph information propagation, and dual-primal graph interaction framework.
6.3 Ablation Study
In this part, we do 3 ablation studies WebQSP dataset to demonstrate the effectiveness of different components of our approach.
Does dual relation graph information propaga- tion matter? In this ablation, we remove the dual relation graph propagation module, where the se- mantic connection of neighbor relation is not used to enhance the representation of current relation. By Table 9, 0.4% score on Hits@ and 1.5% score on F1. shows our dual graph propagation module provides relevant semantic information from neighborhood relations which share the same head or tail entity to enhance the current relation.
6.5 Case Study
We take an example from WebQSP dataset in figure 3 to show the efficiency of our dual graph infor- mation propagation module and dual-primal graph interaction module. The correct reasoning path contains 2 relations stalm_nttring" and "Perfor- mance_actor" so one key point for the model to an- swer the question is to understand and locate these 2 relations. The baseline NSM fails to address the specific relation information and mistakes relation "Award "omination""" and "Award nomince" for the meaning of acting, which predicts the wrong an- swer "Alan". However, considering that the type of head-tail entity for relation "Award "Award_nomination is film and award, the type of head-tail entity for relation "Award nomination" award and person,
see the performance drops by
Does interaction between dual relation graph and primal entity graph matter? In this abla- tion, we remove the interaction between dual graph and primal graph, where the semantic information head-tail entities is not adopted to update the rep- resentation relation connecting them. By table 9,

---

Fact Number
Relation Number
Quantile
0-0.25
0.25-0.5
0.5-0.75
0.75-1
0-0.25
0.25-0.5
0.5-0.75
0.75-1
Overall
NSM
0.78
1.02
0.94
0.90
0.85
1.02
1.01
0.77
0.91
Ours
3.02
4.10
6.78
8.33
4.72
5.90
5.34
6.27
5.56
Table Evaluating relation and entity representation WehQSP dataset based on metric of TransE-score. We classify the cases into 4 groups by the number of relations and facts respectively compute the average TransE-score and the "Overall" denotes the average score of the whole dataset.
Who plays Rabin Hoad Prince Thieves?
Aurerrmatt
Ouaares olge
Performence choratver
Rubis Hood
Performoncs_sre
Pefermac charader
sunting
Perfermn s2Me
Nationality
Fler_oottyy
Kevit
of
United States
Anari naritaiion
Natioelty
Avani rommiviion
metites
Awarii ponitax
Finn language
Ala
Perss bigrage
Penson, PensonLingunge
Englih
(a) Reasoning graph.
(b) Attention weights hetween relations in dual graph prop- agation.
Figure 3: An example from WebQSP dataset. The question is "Who plays Robin Hood in Prince of Thieves?". The yellow path denotes the reasoning path from topic entity "Prince of Thieves" to intermediate entity "Cast" then to answer "Kevin" and the pink edge denotes that "Cast" satisfies constraint character "Robin Hood".
Model
Hits @
F1
Model
Hits@
-uull Graph Propagation
76.8
71.1
NSM
48.8
44.0
-Interaction
77.1
72.0
Ours(2)
53.9
51.2
-Attention
77.2
71.7
Ours(3)
54.4
49.3
Ours
77.2
72.6
Ours(4) 53.9
48.9
Table
Ablation study on WebQSP dataset.
Table 10: Performance with different reasoning steps n on CWQ dataset. The row "Ours(2)) denotes our model with 2.
their meanings be inferred nominating in- stead of acting by our dual-prrmal graph interaction module. Similarly, the information that entity types head-tail entities for relation "Film_ "Film_starring" are film and cast, entity types of head-tail entities for relation "Performance_actte "atteee__crrrrr are cast and person, helps our model to address the meaning of these 2 relations acting by dual-primal graph interaction. Moreover, our dual graph propagation module as- signs important attention among these relations in the reasoning path like constraint relation "charac- ter". which incorporates neighborhood co-relation to enhance the learning of their representations.
incorporates information from head-tail entities and semantic connection between relations to derive better relation and entity representations. We con- duct experiments on two benchmark datasets in this field and our approach outperforms all the existing methods.
Limitations
We showed that our model is efficient in handling complex questions over KBs with heterogeneous graphs. However, we do not investigate the effect of pretrained knowledge graph embedding which contains more structural and semantic information from a large scale of knowledge graphs. In the future, we will probe the manner of incorporat- ing pretrained KG embedding into our dual-primal graph unified reasoning framework.
7 Conclusion
In this paper, we propose novel and efficient framework of iteratively primal entity graph reason- ing, dual relation graph information propagation, and dual-prima graph interaction for multi-hop KBQA over heterogeneous graphs. Our approach

---

References
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji- Ri-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 553-561.
Mattia Atzeni, Jasmina Bogojeska, and Andreas Loukas. 2021. Sqaler: Scaling question answering by decou- pling multi-hop and logical reasoning. Advances in Neural Information Processing Systems, 34:12587. 12599.
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, and Ji- li-Rong Wen. 2022. Unikgqa: Unified retrieval and reason- ing for solving multi-hop question answering over knowledge graph. arXiv preprint arXiv:2212.0093
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. 2013. Semantic parsing on freebase from question-answer answer pairs. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1533-1544.
Chen Liang. Jonathan Berant, Quoe Le, Kenneth For- bus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23-33
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008 Freebase: a collabo- ratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIG- MOD international conference on Management of data, pages 1247-1250.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Antoine Bordes, Nicolas Usunier, Alberto Gareia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data, Advances in neural information pro- cessing systems. 26.
Costas Mavromatis and George Karypis. 2022. Rearev: Adaptive reasoning for question answer- ing over knowledge graphs. arXiv preprint
2rXiv:2210.13650.
Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. 2021. Program transfer for answering complex questions over knowledge bases. arXiv preprint arXiv:2110.05743.
Yunqi Qiu, Yuanzhuo Wang, Xiaolong Jin, and Kun Zhang. 2020. Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 474-482.
Kyunghyun Cho, Bart Van Merrëenboer, Caglar Gul- cehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078.
Nils Reimers and Tryna Gurevych, 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084.
Amrita Saha, Ghulam Ahmed Ansari, Abhishek Laddha, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2019. Complex program induction for querying knowledge bases in the absence of gold programs. Transactions of the Association for Computational Linguistics, 7:185-200
William W Cohen, Haitian Sun, R Alex Hofer, and Matthew Siegler. 2020. Scalable neural methods for reasoning with a symbolic knowledge base. arXiv preprint 2002: 15.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-ho hopti-hop question answering over knowledge graphs using knowledge hase embeddings. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 4498- 4507.
Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Ques- tion answering over freebase with multi-column con- volutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 260-269.
Priyanka Sen, Armin Oliya, and Amir Saffari. 2021. Expanding end-to-end question answering differ- entiable knowledge graphs with intersection. In Pro- ceedings of the 202/ Conference on Empirical Meth- ods in Natural Language Processing, pages 8805- 8812.
David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyampur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, al. 2010. Building watson: An overview of the deepga project. Al magazine, 31(3)559-79.
Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, and Han- wang Zhang. 2021. Transfernet: An effective and transparent framework for multi-hop question an- swering over relation graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4 4149-4158.
Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. Dialog-to-action: Conversational question answering over large-scale knowledge base. In Advances in Neural Information Processing Systems, pages 2942-2951.

---

Haitian Sun, Tania Bedrax- Bedrax-Weiss, and William Cohen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNILP) pages 2380-2390.
Wenbo Zhao, Tagyoung Chung, Anuj Goyal, and An- geliki Metallinou. 2019. Simple question answering with subgraph ranking and joint-scoring. In Proceed- ings of the 2019 Conference the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 324-334.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Co- hen. 2018. Open domain question answering using carly fusion of knowledge bases and text. arXiv preprint arXiv: 1809.00788
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changcheng Li, and Maosong Sun. 2020. Graph neural networks: review of methods and applica- tions. Al Open, 1:57-81.
Mantong Zhou, Minlie Huang, and Xiaoyan 7hu. 2018. An interpretable reasoning network for multi-relation question answering. In Proceedings of the 27th Inter- national Conference on Computational Linguistics. pages 2010-2022.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-bart base for answering complex questions. arXiv preprint 1XX3::803300644
Petar Veličkovié, Guillem Cucurull, Aranixa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint 1rXiv: 10903.
Example Appendix
This is a section in the appendix.
Xu Wang. Shuai Zhao, Jiale Han, Bo Cheng, Hao Yang, Jianchang Ao, and Zhenzi Li. 2020. Modelling long- distance node relations for kbqa with global dynamic graph. In Proceedings of the 28th International Con- ference on Computational Linguistics. pages 2572- 2582.
Y Wu, X Liu, Y Feng. Z Wang, R Yan, and D Zhao. 2019. Relation-aware entity alignment for hetero- geneous knowledge graphs. In Proceedings of the Sveett-Eiggtth International Joint Conference on Ar- tificial Intelligence. International Joint Conferences on Artificial Intelligence.
Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021. lbg-kbqa: Generation augmented iterative ranking for knowledge base ques- tion answering. arXiv preprint arXiv:2109.088
Wen-tau Yih, Ming-Woy Wei Chang. Xiaodong He, and Jian- feng Gao. 2015, Semantic parsing via staged query graph generation: Question answering with knowl- edge base. Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference Natu- ral Language Processing (Volume Long Pupers), pages 1321-13311
Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang. Yiqun Hu, William Wang. Zhiguo Wang. and Bing Xiang. 2022. Decaf: Joint decoding of answers and logical forms for ques- tion answering over knowledge bases. arXiv preprint arXiv:2210.0006.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexan- der Smola, and Le Song. 2018. Variational reason- ing for question answering with knowledge graph. In Thirty-Second AAAI Conference on Artificial Intelli- gence.


Reviews:
[{"review_id": "nySYFPSFPY", "reviewer": "Reviewer_ngck", "summary": "", "questions": "1) Can you give some motivation for the study of graph reasoning in the context of stronger alternatives? In other words, can you give more justification for the importance of the continued study of this approach? \n\n2) Can you give more context on the selection of benchmarks? Given the paper's emphasis on multi-hop QA and on the specific benefits of the proposed approach in this setting, is there a reason that hard multi-hop benchmarks like HotPotQA, LC-QUAD2.0, and others were not considered?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R1: Regarding the comparison with other models:\n\nThere are two mainstream approaches to solve KGQA: (i) parsing the question to executable KG queries like SPARQL, and (ii) grounding question and KG representations to a common space for reasoning. The former class needs\nground-truth executable queries as stronger supervision than the sole answer entity, which is costly to obtain. So it is improper to compare the reasoning based methods using only answer as supervision with the parsing based methods using stronger supervision. Existing reasoning methods [1,2,4,5] do not compare with parsing methods for fair comparison. Our approach belongs to reasoning based methods and achieves the SOTA performance among the reasoning based methods which do not utilize ground-truth executable queries. The method DecAF [3] belongs to parsing methods using stronger supervision, so we did not compare with it. We will modify the expression and emphasize that we achieve the SOTA of reasoning based methods in KBQA.\n\nRegarding the selection of benchmarks:\n\nWebQSP and CWQ are two widely used benchmarks in multi-hop KBQA field. WebQSP requires up to 2-hop reasoning from knowledge base but CWQ contains more complex multi-hop questions and the questions require up to 4-hops of reasoning over the KG [1,2]. Following existing competitive reasoning methods [1,2,4,5], we choose WebQSP and CWQ as our experimental datasets. As for the two datasets mentioned in your review, HotpotQA is a dataset for multi-hop reading comprehension (MRC) which grounds the answer in several passages instead of the KG. It does not apply to KBQA. The original KB dump used to create the dataset LC-QUAD 2.0 is no longer available online [6] and recent methods do not utilize this dataset in experiments. \n\nRegarding retrieval in KBQA:\n\nRetrieval in KBQA is a stage before the graph reasoning, where a question specific subgraph is constructed from the full KG. We following [1,2] utilize PageRank-Nibble algorithm to extract the subgraph. Retrieval stage is independent from the reasoning stage and is not where most reasoning methods contributed in.\nOur contribution locates in the reasoning stage so we do not compare with other retrieval methods. HopRetriever, MultiHopDPR and AISO are methods for MRC which do not tackle KBQA task. LM based pipelines utilize the ground-truth executable queries as stronger supervision and the ground-truth executable queries are costly to get.\n\nR2: For each ablation we do 5 independent experiments with different random seeds and present the mean value. The variation scores of different seeds are within 0.1%-0.2% and the p-value is less than 0.01. The effectiveness of different modules in our approach are statistically significant. We will add the results related to significance test to our paper.\n\nQ1: The parsing based methods need the ground-truth executable queries as supervision and the performance will drop by a large degree with only answer as supervision. In Table 4 of DecAF [3], the performance drops by 6 and 18 Hits@1 score on WebQSP and CWQ. With answer as the only supervision, our method outperforms DecAF by 3 and 4 Hits@1 score on WebQSP and CWQ. The ground-truth executable query forms bring much more cost and difficulty to annotate in practical. So the reasoning methods are important in real scenes where the ground-truth logical forms are hard to obtain.\n\nQ2: WebQSP and CWQ are two widely used benchmarks in multi-hop KBQA field by both reasoning methods and parsing methods. WebQSP requires up to 2-hop reasoning from knowledge base, and CWQ contains more complex multi-hop questions and the questions require up to 4-hops of reasoning over the KG [1,2]. We following the recent methods [1,2,4,5] to choose the two benchmarks for fair comparison. As for the two datasets mentioned in your review, HotpotQA is a dataset in MRC which grounds the answer in several passages instead of the KG, and it does not apply to KBQA. The original KG dump used to create the dataset LC-QUAD 2.0 is no longer available online [6] and recent methods do not utilize this dataset in experiments. We will explore other KBQA benchmarks in revised version. \n\n[1] Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. He et al. WSDM2021.\n\n[2] ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs. Mavromatis et al. EMNLP2021. \n\n[3] DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases. Yu et al. ICLR2023. \n\n[4] TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph. Shi et al. EMNLP2021.\n\n[5] Sqaler: Scaling question answering by decoupling multi-hop and logical reasoning. Atzeni et al. NIPS2021.\n\n[6] Modern Baselines for SPARQL Semantic Parsing. Banerjee et al. SIGIR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are a few strengths of this paper; in general, it is empirically sound and well written. \n1) the authors do well to illustrate the challenge of multi-hop KBQA using both an illustrative figure and example question from WebQSP. Specifically, they make clear that systems have to levereage connectivity (hops) between relations to answer questions with constraints. This helps the reader understand why deriving some additional strucutural information about the graph might help\n\n2) the authors give a detailed technical explanation of their proposed graph derivation and graph reasoning pipeline. Their proposed system is bespoke and technically complex, so this detail is required to understand what they propose. \n\n3) the authors conduct a clear evaluation on two popular benchmarks (WebQSP and CWQ) and perform a useful set of ablations to reveal the benefits of alternative formulations of their system. Additionally, they present the results of these ablations in a very clear and easy to follow manner.", "reasons_to_reject": "There are two main weaknesses of this paper:\n\n1) the authors make a number of claims that are misleading or not supported by their results. For example,\n1a) They claim state of the art performance on both CWQ and WebQSP but are 5-6% worse than the actual state of the art on both (e.g. Sept 2022's DecAF https://www.semanticscholar.org/reader/0751b6de6c07c52a324450e32ae6581d403603da).\n1b) They claim to focus on multi-hop QA but study WebQSP and CWQ, two benchmarks that are widely known to have a majority of 1 and 2 hop questions. They do not evaluate on any KBQA datasets with explicit multi-hop subsets (e.g. HotpotQA, LC-QUAD 2.0, https://paperswithcode.com/dataset/lc-quad-2-0 and https://aclanthology.org/D18-1259/). \n1c) The authors muddle the distinction between retrieval and graph reasoning approaches; the former is a broad category of very competitive approaches, while the latter is a fairly niche approach that has struggled to perform as well. Curiously, they do not evaluate against alternative retrieval based techniques that are explicitly designed for multi hop KBQA without using graph reasoning, such as HopRetriever (https://arxiv.org/abs/2012.15534), MultiHopDPR (https://openreview.net/pdf?id=EMHoBG0avc1), AISO (https://paperswithcode.com/paper/adaptive-information-seeking-for-open-domain). This is problematic because the field has largely moved away from the authors proposed bespoke graph-reasoning techniques towards direct retrieval and LM based pipelines that perform better. The authors don't address or argue for their chosen class of technique and instead simply omit this important context. It is fine to study an unpopular technique and it is clear that the authors conducted empirically rigorous science. It is a shame that the authors frame the work around a false claim of SOTA and on multi-hop QA, rather than present the work as an improvement in graph reasoning and argue for the continued study of that discipline. I can't give this paper the soundness score the empirical work deserves because the misleading claims the writing makes. \n\n2) The ablations that the authors present seem to indicate that the benefits of the novel parts of their approach are fairly minimal. For example, removing dual-graph propagation only reduces performance by .5% Hit@1 to 1.5 F1; removing other bespoke formulations reduces performance by a similarly small margin (e.g. 0.1% 0.6% and 0.9%). They do not report error bars for their approaches, so it is not possible for us to tell whether this difference is statistically significant. This would be less of a concern if they were using a less complex architecture, in which the typical variance can be inferred from other works. The authors do mitigate this somewhat by providing a case study that additionally supports their claims. Ultimately, the claims that they make about the superiority of dual graph propagation and interactions are weak but defensible.", "questions_to_authors": "1) Can you give some motivation for the study of graph reasoning in the context of stronger alternatives? In other words, can you give more justification for the importance of the continued study of this approach? \n\n2) Can you give more context on the selection of benchmarks? Given the paper's emphasis on multi-hop QA and on the specific benefits of the proposed approach in this setting, is there a reason that hard multi-hop benchmarks like HotPotQA, LC-QUAD2.0, and others were not considered?", "paper_topic_and_main_contributions": "Paper is about KGQA with a graph-based reasoning model. The authors claim to focus on multi-hop KBQA and argue for this setting to assess the claim that prior retrieval techniques do not leverage multi-node path information (\"head-tail entities\") nor information about each predicate neighborhood (\"semantic connection between relations\"). Contributions are:\n\n1) an introduction to the type of graph structure information that presumably would benefit symbolic reasoning for KBQA. This includes an explanation of the graph processing necessary to to derive a \"dual relation graph where each node denotes a relation in the original KG (primal entity graph) and edges are constructed between relations sharing same head or tail entities\n\n2) a 4 module \"graph reasoning\" architecture that uses an LSTM question encoder and pipeline of bespoke networks (MLPs and GAT) to iteratively reasons over original graph, then derived graph, then their interaction. \n\n3) an empirical comparison of their proposed approach with a handful of prior retrieval/graph reasoning approaches.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Dedicated multi-hop QA benchmarks should be cited: \n@inproceedings{yang2018hotpotqa,\n  title={{HotpotQA}: A Dataset for Diverse, Explainable Multi-hop Question Answering},\n  author={Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D.},\n  booktitle={Conference on Empirical Methods in Natural Language Processing ({EMNLP})},\n  year={2018}\n}\n\n@inbook{inbook,\nauthor = {Dubey, Mohnish and Banerjee, Debayan and Abdelkawi, Abdelrahman and Lehmann, Jens},\nyear = {2019},\nmonth = {10},\npages = {69-78},\ntitle = {LC-QuAD 2.0: A Large Dataset for Complex Question Answering over Wikidata and DBpedia},\nisbn = {978-3-030-30795-0},\ndoi = {10.1007/978-3-030-30796-7_5}\n}\n\n\nActual SOTA that they don't beat is not cited: \n@article{Yu2022DecAFJD,\n  title={DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases},\n  author={Donghan Yu and Shenmin Zhang and Patrick Ng and Henghui Zhu and Alexander Hanbo Li and J. Wang and Yiqun Hu and William Wang and Zhiguo Wang and Bing Xiang},\n  journal={ArXiv},\n  year={2022},\n  volume={abs/2210.00063},\n  url={https://api.semanticscholar.org/CorpusID:252683172}\n}\n\nAlternative approaches to this task (namely retrieval for semantic parsing) are not cited:\n@article{Ye2021RNGKBQAGA,\n  title={RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering},\n  author={Xi Ye and Semih Yavuz and Kazuma Hashimoto and Yingbo Zhou and Caiming Xiong},\n  journal={ArXiv},\n  year={2021},\n  volume={abs/2109.08678},\n  url={https://api.semanticscholar.org/CorpusID:237562927}\n}"}}, {"review_id": "p3mGKH2u4Z", "reviewer": "Reviewer_h3Ex", "summary": "", "questions": "A. I am confused about the motivation for employing semantic information of neighborhood relations. As shown in Figure 3, it is not difficult for the existing reasoning models to differentiate between \"Award_nomination --> Award_nominee\" and \"Film_starring --> Performance_actor\" based on the question. Rather, the real challenge lies in recognizing the \"Performance_character -- Robin Hood\" constraint when facing many \"Cast\" CVT nodes. However, it appears that the proposed method cannot alleviate this issue.\nB. It appears that removing any of the sub-modules only results in a slight decline in performance. Therefore, why does combining all of them lead to the ultimate performance improvement?\nC. In essence, you just add two extra sub-module to model the entity type and relation semantic based on the NSM framework. We could potentially achieve good performance by simply adding entity type representation to the existing entity representation, and aggregating neighborhood relation representation with the existing relation representation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R1 : The contributions between our method and NSM are totally different. The novelty of NSM lies in the teacher-student framework where a teacher model is trained beforehand and provides the soft supervision for student model. This doubles the training time and we do not apply this framework for efficiency. We focus on the graph reasoning part which is essential in reasoning methods. We propose to incorporate semantic information of head-tail entities and co-relation among relations to enhance the relation representation for KBQA. To this end, we design the Dual Relation Graph Propagation module as well as Dual-Primal Graph Interaction module, and unify in an iterative manner. Compared with NSM, our method reduces the computation cost and outperforms by 2.9 Hits@1 and 5.2 F1 in WebQSP, as well as 5.6 Hits@1 and 5.3 F1 in CWQ, which demonstrates our efficiency.\n\nR2: SR [1] designs a plug-and-play subgraph retriever and utilizes NSM as graph reasoning. We focus on the improvement on graph reasoning instead of graph retriever, so we did not compare with SR. UniKGQA [2] unifies the retrieval and reasoning in both model architecture and learning for the multi-hop KGQA. Our method outperforms UniKGQA by 0.4 F1 on WebQSP, 3.2 Hits@1 and 0.3 F1 on CWQ. We will add this baseline in revised version.\n\nR3: Thanks for the suggestion. We will revise the table size and formula format in further version.\n\nR4: We conduct the experiments with 5 different random seeds and present the mean value. The p-value is less than 0.01 and our method significantly outperforms the existing SOTA of KBQA reasoning methods. We use SentenceBERT to encode the questions and relations. \nThe number of hidden dimensions d is tuned among {64; 128}. We optimize the model with Adam optimizer, where the learning rate is set tuned among {5e-4; 7e-4; 9e-4} and the batch size is tuned amongst {8; 16; 32}. We tune the number of epochs amongst {30; 50; 70} and apply dropout regularization with probability tuned amongst {0.1; 0.2; 0.3}. We will add the details in Appendix of revised version.\n\nQ1: In Figure 3, it is crucial to understand the relation “Performance_character” to satisfy the constraint. The dual graph propagation module enhances the relation representation by the relevant neighborhood relations like “Performance_actor”. Moreover, the knowledge of entity “Robin Hood” which is a film character promote the relation learning. \n\nQ2: The ultimate improvement is not the simple accumulation of different modules. Considering different modules proceed iteratively, one module could bring some interference to other modules. However from Table 8 we can see different modules have irreplaceable effects in our method.\n\nQ3: The information (knowledge) of entities does not only contains entity type, but also identity, location or so, which are crucial to infer the relation. In Figure 3, the entity type “person” can not infer the precise meaning of relation “Performance_actor” considering many other relations connecting a film and person like “producer”. The knowledge that “Robin” is a film actor helps the model learn the relation. Moreover, Not only the head-tail entities but also co-relation from relevant relations enhance the representation of current relation by our dual graph propagation module.\n\n[1] Zhang, Jing, et al. \"Subgraph retrieval enhanced model for multi-hop knowledge base question answering.\" arXiv preprint arXiv:2202.13296 (2022).\n\n[2] Jiang, Jinhao, et al. \"Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph.\" arXiv preprint arXiv:2212.00959 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose an iterative reasoning method that effectively fuses semantic information of relations and type information of entities.\n2. They thoroughly analyze the method and demonstrate the importance of incorporating information from neighborhood relations and connected entities.\n", "reasons_to_reject": "1. The novelty of their method is restricted as it resembles NSM in general, with the only addition of extra information from relations and entities, through a corresponding sub-module.\n2. Insufficient demonstration of performance gain from the proposed method due to the absence of some important baselines.\n3. The writing style deviates from standard conventions, with non-standard formatting of tables and formulas.\n4. Due to the lack of detailed implementation details, I am concerned about the reproducibility of this paper.", "questions_to_authors": "A. I am confused about the motivation for employing semantic information of neighborhood relations. As shown in Figure 3, it is not difficult for the existing reasoning models to differentiate between \"Award_nomination --> Award_nominee\" and \"Film_starring --> Performance_actor\" based on the question. Rather, the real challenge lies in recognizing the \"Performance_character -- Robin Hood\" constraint when facing many \"Cast\" CVT nodes. However, it appears that the proposed method cannot alleviate this issue.\nB. It appears that removing any of the sub-modules only results in a slight decline in performance. Therefore, why does combining all of them lead to the ultimate performance improvement?\nC. In essence, you just add two extra sub-module to model the entity type and relation semantic based on the NSM framework. We could potentially achieve good performance by simply adding entity type representation to the existing entity representation, and aggregating neighborhood relation representation with the existing relation representation.", "paper_topic_and_main_contributions": "This paper aims to promote question-answering over knowledge graphs, such as Freebase. The motivation is to enhance relation understanding by fusing the semantic information of neighborhood relations and connected entities' types. The experiments conducted exhibit significant performance gains on two public datasets. The main contributions are fully utilizing information from entities and relations and the deep fusion between them.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Zhang, Jing, et al. \"Subgraph retrieval enhanced model for multi-hop knowledge base question answering.\" arXiv preprint arXiv:2202.13296 (2022).\n\nJiang, Jinhao, et al. \"Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph.\" arXiv preprint arXiv:2212.00959 (2022).", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Table 2's size should be modified as it exceeds the page's scope.\n2. When adding \"where xxx\" after formulas (e.g., Eq (3), (4), (6)), ensure to append \", \" to the last formula. And modify the format of the formulae, including Eq (13) and the formulas before Section 5.\n"}}, {"review_id": "ZDFbjTWqTk", "reviewer": "Reviewer_qVC6", "summary": "", "questions": "The word 'Heterogeneous' is included in both the title and keyword list. However, not evident explanation about it is included. Besides, knowledge graph is naturally a heterogeneous graph. So is it necessary to emphasize it evidently in title and keyword list", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work. “heterogeneous” in the title denotes the nodes and relations in primal entity graph and dual relation graph are different. We will give its explanation in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is interesting to consider the co-relation information that exist in many question patterns. The paper proposes the dual relation graph to include this kind of information into its framework, and obtain new sota.\n2. Solid experiments are conducted, which helps explain why this framework works. It includes in detail the difference when the entity/relation graphs has different scales and connectivity, the influence of the dual graph introduction and some ablation study.", "reasons_to_reject": "I saw no evident reasons to reject this paper.", "questions_to_authors": "The word 'Heterogeneous' is included in both the title and keyword list. However, not evident explanation about it is included. Besides, knowledge graph is naturally a heterogeneous graph. So is it necessary to emphasize it evidently in title and keyword list", "paper_topic_and_main_contributions": "This paper aimed to improve the Multi-hop KBQA performance, by incorporating the co-relation information. It proposed the concept of dual relation graph, in which node representing relations while edges representing entities, except for the primal entity graph. By interaction between the primal graph and the dual graph, this paper fuses the representation of both entities and relations. The main framework of this paper includes primal entity graph reasoning, dual graph information propagation, and dual-primal graph interaction. It evaluated the algorithm on two datasets, namely WebQuestionsSP and Complex WebQuestion, and show a new sota performance of KBQA.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section ‘A Example Appendix’ should be removed in final version."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for multi-hop knowledge base question answering. The idea is to use information from related relations and entities to improve reasoning over the KG. The paper provides experimental evidence and analysis supporting their method. However, reviewers also pointed out some issues regarding important baselines & comparisons that were missing.
Decision: Accept-Findings