ReasoningLM: Enabling Structural Subgraph Reasoning  in Pre-trained Language Models for Question Answering over Knowledge Graph
Authors: Jinhao Jiang
Kun Zhou
Xin Zhao
Yaliang Li
Ji-Rong Wen
Venue: EMNLP 2023 Main (2023)
Keywords: Pre-trained Language Model
Knowledge Graph Question Answering

Abstract: Question Answering over Knowledge Graph (KGQA) aims to seek answer entities for the natural language question from a large-scale Knowledge Graph (KG).
To better perform reasoning on KG, recent work typically adopts a pre-trained language model (PLM) to model the question, and a graph neural network (GNN) based module to perform multi-hop reasoning on the KG.
Despite the effectiveness, due to the divergence in model architecture, the PLM and GNN are not closely integrated, limiting the knowledge sharing and fine-grained feature interactions.
To solve it, we aim to simplify the above two-module approach, and develop a more capable PLM that can directly support subgraph reasoning for KGQA, namely ReasoningLM.
In our approach, we propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning, and also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions.
After adaptation, the PLM can be parameter-efficient fine-tuned on downstream tasks.
Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data.
Our codes and data are publicly available at https://github.com/RUCAIBox/ReasoningLM.

ReasoningLM.:
Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph
Jinhao Jiangl, Kun Zhou??, Wayne Xin L Yaliang Lit, Ji-Rong Wen
'Gaoling School of Artificial Intelligence, Renmin University of China.
School of Information, Renmin University of China.
Beijing Key Laboratory of Big Data Management and Analysis Methods. AAlibaba Group.
jjiangjinhao,,rwenn edu. cn, francis braacis_kun_zin lii_kkuuunzzziiie wtthhou@133.comm yaliang. 1 wwwaaaaaaaaaannnnaaaaaaeeeeaeaee com, batmanfly@gmail. com com
Abstract
KG, we can develop KGQA methods by leveraging structural semantics of KG to more accurately infer the answer entities to complex factual questions. Starting from the topic entities mentioned in the question, a typical KGQA approach (Sun et al., 2018) is to perform the multi-hop reasoning along with relations on the KG, until finding a path that can reach the answer entities. To develop this ap- proach, existing methods (Sun et al., 2018; He et al., 2021; Shi et al., 2021) mostly incorporate a text encoder to produce the representation of the given question, and reasoning module perform multi- hop reasoning on the KG using the question rep- resentation. Typically, recent work (Jiang et al., 2022b,a) adopts the pre-trained language mod- els (PLM) (e.g., BERT (Devlin et al., 2019)) and graph neural networks (GNN) (e.g., GAT (Velick- ovic et al., 2017)) to implement the text encoder and reasoning module respectively, which can bet- ter understand the semantic information in the ques- tion and structured knowledge from the KG, im- proving the final performance
Question Answering over Knowledge Graph (KGQA) aims seek answer entities for the natural language question from large- scale Knowledge Graph (KG). To better per- form reasoning KG, recent work typically adopts pre-trained language model (PLM) to model the question, and a graph neural net- work (GNN) based module perform multi- hop reasoning on the KG. Despite the effective- ness, due the divergence in model architec- ture, the PL.M and GNN are not closely inte- grated, limiting the knowledge sharing and fine- grained feature interactions. To solve it, aim to simplify the above two-module approach, and develop a more capable PLM that can di- rectly support subgraph reasoning for KGQA, namely ReasoningLM. In our approach, we propose a subgraph-aware self-attention mech- anism imitate the GNN for performing struc- tured reasoning, and also adopt an adaptation tuning strategy adapt the model parame- ters with 20,000 subgraphs with synthesized questions. After adaptation, the PLM can be parameter--fffcieet fine-tuned on downstream tasks. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, with fewer updated parameters and less training data. Our codes and data are publicly available at https://g tiub........ thub.com/ con/ RUCATBox/Reasoonnngi
Despite the effectiveness, there are two major shortcomings with the aforementioned approach that combines PLM encoder and GNN reasoner. First, due to different model architectures, PLM and GNN are often integrated in a loose way (e.g., relevance score sharing), which largely limits the knowledge sharing and fine-grained interaction be- tween the question and KG (a subgraph spanned by related entities). Second, the GNN based rea- soner performs reasoning mainly based sub- graph structure, which lack rich semantic knowl- edge as that in PLMs, making the reasoning results likely be less effective, especially for complex questions. In addition to the two shortcomings. such a approach also requires a complicated im- plementation in practice, since it involves in two different modules.
1 Introduction
Question answering over knowledge graph (KGQA) (Sun et al., 2018; He et al., 2021) has gar- nered significant attention in recent years, which aims to find answers for the natural language ques- tions based on knowledge graphs (KGs), e.g., Free- base (Bollacker et al., 2008) and Wikidata (Tanon al., 2016). Since massive amount of world knowledge has been formatted into a structured form (e.g., a triple (head, relation, tail)) in the
To address these issues, we aim to simplify the above two-module approach and develop a more
Corresponding author.

---

capable PLM that can directly support structural subgraph reasoning for KGQA. Our approach is inspired by the finding that the Transformer archi- tecture consisting of stacked self atteenttnnion mod- ules can be explained as a fully-connected graph encoder (Dwivedi and Bresson, 2020) in a mathe- matically equivalent way. Therefore. Transformer essentially has the potential to effectively model graph data and further performs graph reasoning, which has been shown in existing work (Ying et al., 2021). While, these attempts based on graph Trans- formers either neglect the modeling in text seman- tics, or cannot capture ane-ggrined semantic inter- action between question and KG subgraph, making them infeasible for KGQA tasks.
graph by leveraging an adapted subgraph-aware self-attention mechanism.
We propose an automatic data construction method for the KGQA task format using LLMs to support the adaptation of PLM to the special input format and attention mechanism.
2 Related Work
Question Answering over Knowledge Graph. Multi-hop KGQA aims to find answer entities that are multiple hops away from the topic enti- ties in a large-scale KG. Existing work (Sun et al., 2018) typically first retrieves a question-relevant subgraph from the KG to reduce the search space and then performs multi-hop reasoning to find the answer entities. Several methods (Sun et al., 2018; He al.. 2021) have been developed to facilitate the answer reasoning over the KG. These methods typically consist of a question encoder to repre- sent the question, and a reason module to perform multi-hop reasoning over the KG using the question representation. Early work (Sun et al., 2018, 2019) uses a simple LSTM to encode the question and a GNN to model the reasoning over KG. However, singular representation of the entire question cre- ates confusion for the GNN regarding the specific relation that should be attended to at each step. To address this concern, subsequent work (He et al., 2021) attempts to decompose the semantics of the question and send the corresponding representation to the GNN module at each step. With the develop- ment of PLMs, recent work (Shi et al., 2021; Jiang et al., 2022b) proposes enhance the question un- derstanding by using PLM the question encoder. They use PLM to compute the semantic similarity between the question and relations at each step and use a simpler GNN to propagate similarity infor- mation and update entity scores over KG.
To this end, in this paper, we propose a subgraph reasoning enhanced PLM, called ReasoningL.M, en- abling both effective question understanding and KG reasoning in unified approach. As the major contribution, we propose a subgraph-aware self- attention mechanism, which can imitate the GNN to model the entities and their relations via attend- ing to neighboring nodes on the KG. Further, such a structural attention mechanism has been integrated into a constrained masking framework, to jointly model question attention, KG to question attention, and KG attention. In this way, we not only perform the knowledge interaction and sharing be- tween the question and KG, but also leverage PLM to perform structural reasoning as GNN. However, since the PLM is originally trained by learning from -enerra-ppurpose natural language text, it is necessary to adapt it to the special input format and attention mechanism. Thus, we propose adapta- tion taning strategy that utilizes 20,000 subgraphs with synthesized questions to adapt the parameters of the PLM. After adaptation, the PLM has been well adapted for subgraph reasoning, hence it can be fine-tuned on different downstream KGQA tasks in parameter-efficient manner, achieving better performance with only a few parameters trained.
PLM for Knowledge Graph Reasoning. Besides KGQA, PLM has also been used for other knowl- edge graph reasoning tasks, such as commonsense reasoning (Yasunaga et al., 2021) or predicting missing facts (Zamini et al., 2022). There are mainly two methods for PLM to leverage the KG. Several studies (Yasunaga et al., 2021) attempt to fuse the representation of the KG into the PLM, which is modeled by the GNN. It can make the PLM aware of the KG to some extent through the modeled representation. However, it's not easy to bridge the gap between the PLM and GNN given
To evaluate the effectiveness of our approach, we conduct extensive experiments on three KGQA datasets. Experimental results demonstrate that our proposed approach
surpass existing state-of- the-art models by a large margin, even with fewer updated parameters and less training data.
Our contributions can be summarized as follows:
We enable the PLM to simultaneously model question understanding, deep interaction between question and subgraph, and reasoning over sub-

---

the different model architecture and initialized pa- rameters, leading sufficient understanding of
Approach
In this section, we present our proposed approach, L.., ReasoningLM, which enables both effective question understanding and KG reasoning in sin- gle PLM for better solving the KGQA task.
KG for PLM. In contrast, a more direct way is to linearize the KG as a sequence and input it to PLMs (Xie et al., 2022; Saxena et al., 2022). In this way, the PLM can directly utilize the KG to perform reasoning. Despite its simplicity, such way neglects the structure of KG, which is an im- portant feature. In contrast, we propose to model the question and KG in single PLM, while captur- ing the structure information with subgraph-awan self-attention mechanism.
4.1 Overview
Existing KGQA methods (Sun et al., 2018) typi- cally adopt PLM to encode the question into latent representation, and GNN to perform the reason- ing over KG guided by the representation. How- ever, due to the architecture divergence, it is hard to closely integrate the PLM and GNN for knowl- edge sharing and feature interaction. Inspired by recent work (Dwivedi and Bresson, 2020) that re- veals the similarity between GNN and Transformer architecture of PLMs, take different perspec- tive to simplify the above two-module approach and develop a more capable PLM that can support structural subgraph reasoning for KGQA.
3 Preliminary
In this section, we present the notations utilized throughout the paper, followed by a formal defini- tion of the KGQA task.
Knowledge Graph (KG). A knowledge graph is commonly composed of collection of triples, ex- pressed as G (e, r, le) e' E E,r R]. where E and R denote the entity set and relation set, re- spectively. A triple (e, r,e') describes the fact that a relation r exists between head entity e and tail entity e'. Furthermore, we introduce entity neigh- borhood to denote both incoming and outgoing triples for an entity c, denoted as No = {(e, (ic,,,c) c simplify
Our basic idea is to implement a GNN within the PLM, to bridge the knowledge gap for question understanding and KG reasoning. To achieve it, we propose the Reasoningl.M, enabling both effective question understanding and KG reasoning in uni- fied approach. Concretely, we first adapt the PLM to implement the functionality of GNN aggre- gate the information from the neighboring entities and relations with subgraph-aware self-attention mechanism (Section 4.2). Then, we adopt adap- tation tuming to optimize the PLM parameters to better adapt it to the special input format and atten- tion mechanism (Section 4.3). Finally, apply the adapted PLM to solve the KGQA task with parameter-efffcient fine-tuning (Section 4.4). The overview of ReasoningLM is shown in Fig- ure 1. In our model, as the PLM can understand the question and perform reasoning in KG in a uni- fied approach, it can freely share and interact with the knowledge from both for improving the KGQA task. Besides, such way also enables the PLM to fully participate in the KG reasoning process (in- stead of GNN solely), which can make full use of its rich knowledge and strong reasoning capacity.
9} U ((e', r, e) E G). In this way, the definition of the neighborhood triples for entity e as Ns r,) g}.
Question Answering over Knowledge Graph (KGQA). Given a natural language question q and a KG G, the task of KGQA aims to find answer en- titie(s), denoted as Ag E &, to the question on the KG. Following previous work (Sun et al., 2018), we assume that the entities mentioned in the ques- tion have already been linked with entities KG, called topic entities, denoted as Ta C E. In this work, we focus on solving the KGQA task where the answer entities are multiple hops away from the topic entities over the KG. Considering the trade-off between efficiency and accuracy, we fol- low existing work (Sun et al., 2018) that solves this task using a retrieval vetrieval-then-rrasooiig framework. In the two-stage framework, given question q and topic entities Ty. the retrieval stage aims to re- trieve a small question-relevant subgraph Gg from the large-scale input KG 9, while the reasoning stage finds answer entities Ag by reasoning over the retrieved subgraph Gy-
4.2 Adapting PLM for Subgraph Reasoning In this section, we discuss how adapt PLM for subgraph reasoning. Next, we introduce the subgraph data serialization and then present the subgraph-award self-attention mechanism to unify question understanding and subgraph reasoning

---

Predicted
0.0
0.7
0.1
0.2
0.0
Entity Scores
D
Linear Layer
Output Hidden States
CLS
SE
En
Fi
T2
O2
Ta
C
Transformer
Transformer Layer with
(x N)
Encoder
Subgraph-aware Self-Attention'
Masked Attention
viatñic
Input Embeddings
Eo
F
F2
Male
goader
Jaxon
Bieha:
sibling
Question
whar is, the, name, of, justin, bieber, brother)
Justin Bieber
rgtsibling
e
Jason "Bieber"
Jusii Richer
stbling
Jasmyn
Rlicher
gandlr
Subgraph
Justin "ieberr
sibling,
Jaxon Bieber
siilinng
Jazmyn Bieber
gender malc, gender, female
Subgraph rh
Famalle
Figure I: illustration of performing answer entity reasoning over a subgraph according to question using ReasoningL.M with proposed subgraph serialization and subgraph-aware self-attention.
within a single PLM, respectively.
port graph reasoning within the PLM, to propa- gate relevance information along with the relation edges on KG. We first initialize the embeddings of the serialized subgraph and the question, and then perform self-attention with constrained masks aggregate their representations.
4.2.1 BFS-based Subgraph Serialization
For KGQA task, the input data typically consists of the natural language question and its relevant subgraph Gg retrieved from the KG. To enable the PL.M to capture the structured information from the subgraph and perform reasoning on it, we propose a breadth-firs: search (BFS) based subgraph serial- ization to convert the subgraph into a sequence. Concretely, starting from a topic entity ep E Tg- we perform the BFS to visit all triples on Gy. It first visits all the triples whose head entity is the topic entity, e.g., (Co, f1, C1), and then moves to the triples whose head entities have been visited before, and forth. Based the order in the BFS process, we concatenate all the visited triples as long sequence. To reduce the node sequence length, we only concatenate the entities or relations for the first time that they are visited, hence the final scrialized subgraph consists of all the visited entities and relations, denoted as:
Embeddings Mapping, Since PLMs do not have the mapping embeddings for the entities and rela- tions within the serialized subgraph, we need to initialize their embeddings before feeding them into the PLM. To embed them into the semantic space of the PLM, first tokenize each entity or relation into subwords' using the tokenizer of the PLM, and then sum their embeddings into a single embedding to represent Finally, we can obtain the embedding matrix of the whole serial- ized subgraph, denoted as Ng, E gixd where d is the embedding dimension and is the length of the input sequence. Next, concatenate it with the token embedding matrix N2 of the question q after tokenization, compose the input token embedding matrix of the PLM:
Sg, {ee, F1, E1, T2; 62, ..y Tim emp
(1)
N [N;; Ng.1
[ng. Teg Negs Nem.
where m is the total number of triples in Sg.- In this way, we can preserve the structure infor- mation of the subgraph within a relatively short sequence (with length as the count of entities and relations in Sg,). which can be varied according to the context length of language models.
Based on it, we also add the position embeddings as Ng = N + E to obtain the input embedding matrix of the PL.M.
Self-Attention with Constrained Masks. After obtaining the input embedding matrix NEs we feed into the multi-layer Transformer encoder
4.2.2 Subgraph-aware Belf-Attention
Based on the serialized subgraph, we leverage the subgraph-aware self-attention mechanism to sup-
PL mostly use Byte- Pair Encoding tokenizer, which may segment an entity or relation name into several subwords.

---

of the PLM, perform reasoning over the sub- graph based on the given question. To enable the fine-grained semantic interaction between the ques- tion and the associated subgraph, we propose a constrained mask mechanism on the self-attention layers for controlling the attention interaction, in- cluding four kinds of attention modes:
matrices of the self-attention layer. In this way. only the self-attention values between invisible kens would be zero (after softmax activation on IINF values), avoiding the PLM to aggregate repre- sentations from them.
4.3 Adaptation Tuning
Full question attention. The representations of each token of question can attend to the other tokens of question (part in Figure 1).
To help the PLM well adapt into the special in- put format and attention mechanism, we adopt the adaptation tuning to adapt the parameters of the PLM. We first collect the tuning dataset based on sampled subgraphs and synthesized questions, and then utilize the answer entity prediction task for training.
Full subgraph + question attention. The rep- resentations of all the entities and relations can attend to the question representations, hence we can perform reasoning on the KG based on the question (part B in Figure 1).
Structural subgraph attention. In the serialized subgraph, an entity can aggregate the information from its one-hop neighboring entities and relations (in a triple), similar to the updated way of node representations in GNN. Further, a relation can aggregate the information from its head and tail entities in the triple, as it establishes link between the two entities (part C in Figure 1).
4.3.1 Tuning Data Construction
To enable the PLM to understand the question and perform reasoning on the KG, we construct an adap- tation dataset in an automatic way to tune its pa- rameters. The dataset consists of 20,000 synthe- sized questions with relevant subgraphs extracted from the KG, and the answer entities. Next, we introduce the process of subgraph extraction and question synthesis.
Except for the above ways, other information flows forbidden in the self-attention layer. In addition to the constraint on the subgraph struc- ture, we also prevent the question attending to the subgraph, which can avoid the question representa- tions to be influenced by the irrelevant information in the subgraph (part D in Figure 1).
Subgraph Extraction. We extract the subgraphs from Wikidata", a general-domain KG with natural language descriptions of the entities and relations. We consider extract of subgraphs centering around popular entities, to better adapt the PLM for understanding commonly-un we use the popular entities in Wikidata5M (Wang et al., 2021) as our seed topic entity set following KQA Pro (Cao et al., 2022), then randomly sample the answer and the subgraph. Starting from the topic entity, perform random walk over the KG, to sample reasoning path with more than 4 hops, whose ending entity is regarded as the an- swer entity. Then, we randomly extract the entities and relations around the topic entity to compose the subgraph, where we guarantee that the entities and relations from the reasoning path are also in- cluded. Such a way is easy to conduct and can automatically extract multiple subgraphs with the answer entities.
knowledge. Thus,
To achieve them, design the constrained self- attention mask M Rixl where the value in the i-th raw and j-th column denotes whether the i-th token can attend to the j-th one (0) or not (-inf), denoted as:
I; E So, and æj E g.
M o
Ii. 2 E Sg, and azj = 1, (2)
-INF
others,
where 2; and xy represent the tokens (i.e., relations, entities or words) in the i-th and j-th positions of the input, and 0ij adjacent (within a KG triple). Then, we utilize the mask matrix to compute the constrained self- attention on the multi-layer Transformer encoder of the PLM follows:
indicates that x; and æj are
Question Synthesis. Based on the sampled rea- soning path and answer entities, we also adopt an automatic way to synthesize the questions. Here, propose two approaches, i.e., rule-based synthe- sis and L.LM-based synthesis. For the rule-
Attn(Q Q, K, V) softmax M)V, (3)
where A E Rix! the original attention matrix, and Q, K, V E Rixd are the input representation
Phtps:/wwww.wkkidta..rg

---

one, we first hand-craft several general templates, and then utilize them to convert the topic entity and the relations on the reasoning path into a natu- ral language question. However, such a way leads to a poor diversity of questions, and also needs human efforts. Recently, as large language mod- els (e.g., ChatGPT) have shown a powerful gener- ation capability (Brown et al., 2020; Zhao et al., 2023) and have been used generate label-free datasets (Chen et al., 2023), we seek help from them to produce more high-quality questions. Con- cretely, we write a prompt to guide ChatGPT, pop- ular LLM, to generate the corresponding question for the answer entity based on the reasoning path and the topic entity. In this way, we cost approxi- mately 15 dollars, and obtain 20,000 questions with diverse formats and fluent expression. The detail prompt we used is shown in Appendix D.
apply to the subgraph retrieval and answer reason- ing subtasks respectively, where we only tune the parameters in the adapters (Houlsby et al., 2019) but freeze other parameters.
Subgraph Retrieval. We follow Zhang al. (2022) fine-tune our model on the subgraph re- trieval subtask, where we optimize our model learn to predict the similarity between the question and relevant relations. During inference, starting from the topic entities, the model iteratively mea- sures the semantic relevance between the question and neighboring relations, and adds proper ones and their corresponding triples into the subgraph, to extract question-relevan subgraph.
Answer Reasoning. Based on the retrieved sub- graph, we also utilize the answer entity prediction task in Section 4.3.2 to fine-tune our Reasoning M,, to learn to accurately find the answer entities of the given question from the subgraph. During infer- ence, we select the highest scoring entity predicted by our approach as the answer entity.
4.3.2 Answer Entity Prediction
Given the synthesized question, extracted sub- graphs and the answer entities, we feed them into our ReasoningL.M, and tume the model parame- ters via the answer entity prediction task. It is a multi-classification task to predict which entity in the subgraph is the answer entity of the question. Concretely, through the multi-layer Transformer encoder with constrained self-attention, obtain the hidden state H E Rixd of the input sequence at the last layer. Then, we add a linear prediction layer with the softmax activation func- tion to transform the hidden state into the answer scores of all entities:
Experiments
5.1 Experimental Setup
can
Adaptation Tuning Corpus. Our adaptation tun- ing corpus is collected from Wikidata (Tanon et al., 2016), a general domain knowledge graph. We download the English Wikidata Dumps (2018/12/31) from the official site, and extract 2,000 entities from Wikidata5M (Wang et al., 2021) as seed topic entities. Finally, we construct 20,000 samples for adaptation tuning and split 1,000 samples the validation set for select- ing the best checkpoint.
softmax H));
(4)
where E R'. Then, we minimize the KL di- vergence between the predicted and ground-truth answer scores as:
Datasets. Following existing work on KGQA (He et al., 2021), we conduct experiments on three pop- ular datasets to evaluate our proposed approach, including WebQuestionsSP (WebQSP) (Yih et al., 2015), Complex WebQuestions 1.1 (CWQ) (Talmor and Berant, 2018), and MetaQA (MQA) (Zhang et al., 2018). Table shows the statistics of the three datasets. We give a detailed description of each dataset in Appendix A
Lat DKi(s,
(5)
where s" is the ground-truth answer scores of all entities, where an entity is if it is a labeled answer entity. Note that we only compute the loss for the entities, the relations and question words not able to be the answer.
4.4 Efficient Fine-tuning
Evaluation Protocol. We follow existing work that treats the reasoning as ranking task for evalu- ation (Sun et al., 2018). For each question, we rank the answer score of all candidate entities and then assess the correctness of the top-1 answer using the
After adaptation tuning, the PLM has been well adapted performing reasoning over the general- domain subgraphs. Therefore, we further perform parameter-efficientl fine-tuning on the PLM, to

---

Table Performance comparison different methods for KGQA (Hits@1 and in percent), We copy the results of LLMs from Jiang et al. (2023) and the results of the other baselines from Jiang al. (2022b). Bold and underline fonts denote the best and the second-best methods, respectively. "FPT" and "PET" denote the full-parameter tuning and parameter-effiienn tuning, respectively. "Rule-SYN" and "LLM-SYN" refer synthesize the questions using rule-based and I. LLM-based strategies, respectively.
Updated Params
WebQSP
CWQ
MQA-IH
MQA-2H
MQA-3H
Models
Hits
F1
Hits
F1
Hits@1
Hits@1
Hits@1
KV-Mem (Miller al., 2016a)
46.7
34.5
18.4
15.7
GraftNet (Sun et al., 2018)
0.5M
66.4
60.4
36,8
32.7
82.5
PullNet (Sun et al., 2019)
68.1
45.9
EmbedKGQA (Saxena al., 2020)
125M
66.6
92.0
40.7
34.6
NSM (He et al., 2021)
3M
68.7
62.8
47.6
42.4
94.8
97.0
91.0
TransferNet (Shi et al., 2021)
111M
71.4
48.6
96.5
97.5
90.1
SR-NSM+F2F (Zhang et al., 2022)
130M
69.5
64.1
49.3
46.3
UniKGQA (Jiang et al., 2022h)
12M
75.1
70.2
50.7
48.0
97.1
98.2
92.6
Davinci-003 (Ouyang et al., 2022)
48.3
52,1
25.3
42.5
ChatGPT
61.2
61.9
31.0
43.2
StrectGPT (Jiang ct al., 2023)
72.6
94.2
93.9
80.2
ReasoningLM (FPT, LLM-SYN)
1M
78.5
71.0
69.0
64.0
96.5
98.3
w FPT, Rule-SYN w PET, LLM-SYN
IM
78.0
70.5
62.8
55.4
96.1
96.9
91.0
IM
76.7
69.1
68.3
62.4
95.7
97.0
90.9
Table 2: Statistics of the experiment datasets.
dation set. After adaptation tuning, we apply the ReasoningL.M to downstream KGQA tasks with parameter-efficient fine-tuning. We add the extra re- trieval and reasoning adapter to the ReasoningL.M for subgraph retrieval and answering reasoning spectively, and only update the reasoning adapter while freezing other parameters.
Type
Task
KG
Train
Dev
Test
WebQSP
Freebase
2,848
250
1,639
CWQ
Freebase
27.639
3,519
3,531
KGQA
MQA-IH
OMDb
161
9,992
9,947
MQA-211
OMDb
210
14,872
14,872
MQA-3H
OMDb
150
14,274
14.274
For the retrieval stage, we follow the pipeline of Zhang et al. (2022) fine-tune our Reason- ingLM and then perform subgraph retrieval. we collect question-relation pairs based on the shortest relation paths between topic entities and answer entities, and then use these pairs to fine-tune the model to compute the similarity between the ques- tion and relations. We directly compute the similar- ity between the question and relations at each hop and freeze other parameters except for the adapter module. we optimize parameters with the AdamW optimizer, where the batch size is 10 and the learn- ing rate 5e-5 for all datasets. Then, we leverage the model to retrieve the subgraph. Specifically, starting from the topic entities, the model itera- tively measures the semantic relevance between the question and neighboring relations, and adds top-k relations and their corresponding triples into the subgraph. We the i as 15 for WebQSP and CWQ, and for MetaQA.
Hits@ metric. Given that a question may have multiple answers, we also adopt the F/ metric.
Baselines. We consider the following three types of baseline methods for performance com- parison: (1) non PLM-based methods: KV-
Mem (Miller et al., 2016b), GraphtNet (Sun et al., 2018), PullNet (Sun et al., 2019), NSM (He et al., 2021); (2) PLM-hased methods: Embed- KGQA (Saxena et al., 2020), TransferNet (Shi et al., 2021), SR+NSM+E2E (Zhang et al., 2022), UniKGQA (Jiang al., 2022b): (3) LLM-based methods: Davinci-003 (Ouyang al., 2022), Chat- GPT, StruetGPT (Jiang al., 2023). We give a de- tailed description of each baseline in Appendix B.
5.2 Implementation Details
In our experiment, base PLM. During adaptation tuning, we optimize parameters with the AdamW optimizer, where the batch size 40 and the learning rate is le-4. We select the best checkpoint of adaptation tuning ac- cording to the evaluation of the constructed vali-
RoBERTa-base our
For the reasoning stage, we fine-tune Reason- ingLM on the retrieved subgraph to perform answer reasoning. Similarly, we only update the adapter model and freeze other parameters. We optimize

---

parameters with the AdamW optimizer, where the batch size is for MetaQA, 60 for WebQSP, and 300 for CWQ while the learning rate is le-4 for all datasets.
Table 3: Ablation study of our the subgraph- self- attention mechanism (SA) and adaptation tuning (AT).
WebQSP
CWQ
Models
Hits@
F1
Hits@l I
5.3 Main Results
ReasoningL.M
78.5
70.1
69.0
64.0
w/o SA w/o AT
68.5
63.2
40.5
38.2
We show the results of all five data in Table 1. First, directly using LLMs (e.g., Davinci-003 and ChatGPT) can achieve performance comparable to part of the supervised learning baselines on the WebQSP dataset. However, LLMs perform not well on the more complex multi-hop datasets, such MQA-2H and MQA-3H. It demonstrates that LLMs are still hard for effectively solving KGQA tasks relying solely the LLM. Despite the incor- poration of external KG can enhance LLMs (i.e., StructGPT), they still have a performance gap com- pared to the strong supervised learning models. Secondly, PLM-based methods (e.g.. Transfer- Net, SR+NSM+E2E, and UnikGQA) can achieve a consistently better performance compared with methods not using PLM (e.g., GraftNet, PullNet, and NSd).And UniKGQA achieves a further per- formance improvement on all datasets, benefiting from its unified architecture to learn the essential capability of question-relation semantic matching for both retrieval and reasoning stages.
67.5
60.4
55.2
43.3
Table 4: Performance of implementing ReasoningL.M with different PLMs on CWQ.
RoBERTa (base)
RoBERTa (large)
DeBERTa (base)
BERT
CWQ
(base)
Hits@
69.0
70.0
68.1
67.4
64.0
65.2
63.1
63.0
rameters with rule-based constructed data, (2) PET, LLM-SYN that updates an added adapter while freezing the model parameters with LLM synthesis data. We can see that although both vari- ants show varying degrees of performance decline, they can still achieve better results compared the existing baselines on WebQSP and CWQ. For MetaQA, the two variants only achieve comparable performance to existing baselines. A possible rea- son is that limited data makes it difficult to adapt the model to specific domain.
Finally, our ReasoningLM is substantially better than all other competitive baselines all datasets with only updating a few parameters (only IM), achieving 4.5% improvement of Hits @ on We- bQSP and 36.1% % improvement of Hits 1 on more difficult CWQ compared the best baseline. Un- like the other baselines, approach develops subgraph reasoning enhanced PLM model the question and subgraph seamlessly. We can utilize the pre-trained knowledge within PLM, while en- abling the reasoning process over the subgraph can directly attend to the question. With further adap- tation tuning, our model can be applied to down- stream tasks with parameter-efficient fine-tuning. These results demonstrate the effectiveness and ef- ficiency of our ReasoningLM model.
5.4 Further Analysis
Ablation Study. Our approach contains two im- portant technical contributions, the first is the subgraph-aware self-attention mechanism (SA), which imitates the GNN reasoning over the graph, and the second the adaptation tuning (AT), which enhances the reasoning capability over KG. Here, we conduct the ablation study to verify their effec- tiveness by removing each one individually. We ex- periment with two variants as: (1) w/o AT removing the adaptation tuning procedure, (2) w/o SA remov- ing the subgraph-aware self-attention mechanism in self-attention mechnism. We show the results of the ablation study in Table All these variants un- derperform the complete ReasoningL.M which in- dicates that the two strategies are both important for developing a subgraph reasoning enhanced PLM for KGQA. Besides, we also analyze combining other reasoning models (e.g.. NSM and UniKGQA) with our retrieval sugraphs in Appendix C.
In our approach, we update the full parame- ters of our model (FPT) during adaptation tun- ing with LLM synthesis data (LLM-SYN). Ac- tually, we can accomplish adaptation tuning smaller cost by updating fewer parameters us- ing cheaper constructed data. Here, we study by proposing two variants of our ReasoningL (1) w FPT, Rule-SYN that updates the full pa-
Variants with Different PLMs. Since our pro- posed adaptation method does not change the orig-

---

model has become familiar with the multi-hop rea- soning over the subgraph after adaptation tuning. can be easily applied to specific downstream KGQA tasks with fewer labeled samples, which is meaningful for sparse data scenarios. To ex- plore it, we compare the final performance changes of our ReasoningLM with two strong baselines UniKGQA and NSM w.r.f the increasing of fine- tuning samples with the same retrieval model. We conduct experiments using CWQ, which is more challenging and has larger training set. The re- sults are presented on the right of Figure Reason- ingLM can obtain consistent performance improve- ments compared with other two baselines under var- ious numbers of tine-tuning samples. It indicates that our Reasoning has a better understanding of the answer reasoning over the KG.
N
CRQQ
10
Reveeett N00
66
1K
X0C
1992
E3K
1.36 2K
Nunber Mddattationn sacing Samples
Nanber of luning Samples
Figure 2: The Hits@1 scores of our ReasoningLMM on WebQSP and CWQ after adaptation tuning with a vari- ous number of samples (Left). And the Hits @ score our ReasoningLM compared with two strong base- lines (i.e., NSM and UniKGQA) on CWQ when fine- tuning with various numbers of samples (Right)
inal model architecture, it can be applied to other different PLMs. To explore the performance with different PLMs, we conduct experiments with other three PLMs (i.e., RoBERTa-large, DeBERTa, and BERT). We show the results in Table 4. It is ob- served that the utilization of DeBERTa-bass and BERT-base also yields performance comparable to RoBERTa-buse. This suggests that our adapta- tion method is agnostic to the PLMs used. At the same time, a larger RoBERTa-large can achieve further performance improvement compared with RoBERTa-base. It indicates the potential of our method to be applied larger PLMs. Limited by computational resources, we would apply our method to larger PLMs in future work.
Conclusion
In this work, proposed subgraph reasoning enhanced PLM to support question understand- ing and KG reasoning in a single PLM, namely ReasoningL.. In our approach, we first adopted BFFS-ased subgraph serialization to enable the PLM to capture the structured information and then proposed subgraph-awan self-attention mecha- nism to support graph reasoning within the PLM based the serialized subgraph. In order to adapt the PLM the special input format and attention mechanism, we further utilized an adaptation tun- ing strategy with a cheap data construction cost of 15 dollars by using ChatGPT in an automatic way. Finally, we applied the ReasoningLMM to solve downstream KGQA tasks with parameter-efficieen fine-tuning. Experimental results have shown that our approach can significantly improve the perfor- mance compared existing strong baselines by updating only 1M parameters.
Adaptation Tuning Efficiency. Although the adap- tation tuning strategy is important to enhance the reasoning capability of ReasoningL.M, too many tuning samples require significant construction and tuning costs. Here, we investigate the downstream performance of ReasoningLM on WebQSP and CWQ w.t.t varying numbers of adaptation tuning samples. As shown in Figure 2, we can see that the ReasoningL.M can reach competitive perfor- mance compared with the best baseline UniKGQA after adaptation tuning with few samples (i.e., 5K). It shows that our approach does not require too much data to complete a successful adaptation tuning. Simultaneously, we can observe that as the number of tuning data increases, our model's per- formance will improve even further and eventually reach stable state. It indicates that we only need a few tuning examples to achieve a trade-off between the tuning costs and downstream performance.
7 Limitations
In our approach, we propose a subgraph reasoning enhanced PLM by adapting existing PLM with- out modifying its original architecture. Therefore, the input is usually limited (e.g.. 512) for most of PLMs, causing our model unable to process the arbitrary size of the retrieved subgraph. We can relieve it by using the relative position embed- ding better retrieval model to obtain the proper size of the subgraph. In addition, although we conduct experiments on multiple KGQA datasets, there is a lack of evaluation on other KG reasoning
Fine-tuning Efficiency. As our ReasoningL.

---

tasks, such as commonsense question answering and knowledge graph completion. They can be transformed into the same input format, which can be potentially solved with our method. From Ta- ble 4, can be seen that increasing the model pa- rameters will lead to further performance improve- ment. However, due to limited computational re- sources, we did not conduct experiments on larger PLMs (e.g,, more than Billion parameters).
2023. Label-free node classification on graphs with large language models (Umms). arXiv preprint arX.v:2310.0666.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguisties.
Acknowledgments
This work was partially supported by Beijing Natu- ral Science Foundation under Grant No. 4222027, National Natural Science Foundation of China un- der Grant No. 62222215, and the Outstanding Inno- vative Talents Cultivation Funded Programs 2022 of Renmin University of China. This work was also supported by Alibaba Group through Alibaba Innovative Research Program. Xin Zhao is the cor- responding author.
Vijay Prakash Dwivedi and Xavier Bresson. 2020. A generalization of transformer networks to graphs. CoRR, abs/2012.09699.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In WSDM '21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021. pages 553-561. ACM.
References
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-efficient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIG- MOD International Conference Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pages 1247-1250. ACM.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Drew A. Hudson and Christopher D. Manning. 2019. Learning by abstraction: The neural state machine. In Advances in Neural Information Processing Sys- tems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5901- 5914.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye. Wayne Xin Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. CORR, obs/2205.
Jinhao Jiang, Kun Zhou, Ji-Rong Wen, and Xin Zhao. 2022a. $great truths are always simple: $ A rather simple knowledge encoder for enhancing the com- monsense reasoning capacity of pre-trained models. In Findings of the Association for Computational Lin- guisties: NAACL 2022, Seattle WA, United States, July 10-15, 2022, pages 1730 1741. Association for Computational Linguisties.
Shulin Cao, Jiaxin Shi, Lianggnnng Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Han- wang Zhang. 2022. KQA pro: A dataset with explicit compositional programs for complex question an- swering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), ACI 2022, Dublin, Ireland, May 22-27, 2022, pages 6101- 6119. Association for Computational Linguistics.
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2022b. Unikgqa: Unified retrieval and reason- ing for solving multi-hop question answering over knowledge graph. CoRR, abs/221 2,00999.
Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, and Jiliang Tang.

---

Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. 2016a. Key-value memory networks for directly read- ing documents. In Proceedings of the 2016 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2016, Austin, Texas, USA, Novem- ber 1-4, 2016, pages 1400-1409. The Association for Computational Linguistics.
early fusion of knowledge bases and text. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 4231-4242. Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLI 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pages 641- 651. Association for Computational Linguistics.
Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. 2016b. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing. EMNLP 2016, Austin, Texus, USA, November 1-4, 2016, pages 1400-1409. The Associ- ation for Computational Linguistics.
Thomas Pellissier Tanon, Denny Vrandecie, Sehastian Schaffert, Thomas Steiner, and Lycia Pintscher. 2016. From freebase wikidata: The great migration. In Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 15, 2016, pages 1419-1428. ACM.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der. Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow instruc- tions with human feedback. In NewrlPS.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford al- paca: An instruction-folllwinn Ilama model. https: //github wttp://totttttteteeeeeeeeeeee watsssu-aaa/saaaaeeaeeee _aapcc.
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sett to-sequence knowledge graph com- pletion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Com-
Petar Velickovie, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio', and Yoshua Ben- gio. 2017. Graph attention networks. ArXiv, 1bs/1710.10003.
putational Linguistics /Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2814- 2828. Association for Computational Linguisties.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. KEPLER: A unified model for knowledge embed- ding and pre-trained language representation. Trans. Assoc. Comput. Linguistics, 9:176 194.
Apoorv Saxena, Aditay Tripathi, and Partha P. Taluk- dar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embed- dings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4498-4507, Association for Computational Linguistics.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong. Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle. Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong. Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 602-65 2316.1 Association for Computational Linguistics.
Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, and Han- wang Zhang. 2021. Transfernet: An effective and transparent framework for multi-hop question an- swering over relation graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 4149-4158, Association for Computa- tional Linguisties.
Haitian Sun, Tamia Medrax-Weiss. and William W. Co- hen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joins Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2380-2390. Association for Computational Linguisties.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut. Percy Liang, and Jure Leskovee. 2021. QA-GNN: reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2021, Online, June 6-11, 2021, pages 535-546. Association for Computational Linguistics.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Co- hen. 2018 Open domain question answering using
Wen-tau Yih, Ming-Was Chang, Xiaodong He, and Jian- feng Gao. 2015. Semantic parsing via staged query

---

graph generation: Question answering with knowl- edge base. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference Natu- ral Language Processing of the Asian Federation of Natural Language Processing, ACI. 2015, July 26-31, 2015, Beijing, China, Volume I: Long Papers, pages 1321-1331. The Association for Computer Linguis- tics.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng. Guolin Ke, Di He. Yanming Shen, and Tic- Yan Liu. 2021. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 64,,, 2021, virtual, pages 28888..
Mohamad Zamini, Hassan Reza, and Minou Rabiei. 2022. A review of knowledge graph completion. Inf., 13(8)399.
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5773-5784. Association for Computational Linguistics.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexan- der J. Smola, and Le Song. 2018. Variational reason- ing for question answering with knowledge graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18) 18), the 30th inno- vative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 6069 6076. AAAI Press.
Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min. Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie. and Ji-Rong Wen. 2023. A survey large language models. CoRR.

---

A Datasets
Table 5: Statistics of the experiment datasets.
Following existing work on KGQA (He et al., 2021), we conduct experiments on three popular datasets to evaluate our proposed approach, includ- ing WebQuestionsSP (WebQSP) (Yih et al., 2015), Complex WebQuestions 1.1 (CWQ) (Talmor and Berant, 2018), and MetaQA (MQA) (Zhang al., 2018).
Type
Task
KG
Train
Dev
Test
WebQSP
Freebase
2,848
250
1,639
CWQ
Freebase
27,639
3,519
3,531
KGQA
MQA-IH
OMDb
161
9,992
9,947
MQA-2H
OMDb
210
14,872
14,872
MQA-3H
OMDb
150
14,274
14,274
MetaQA (Zhang et al., 2018) comprises over 400,000 questions in the movie domain, with an- swer entities located up to three hops away from the topic entities. Based on the number of hops, the dataset is divided into three sub-datasets: MetaQA- lhop, MetaQA -2hpp, and MetaQA-3hop. Existing work has demonstrated that the training data for MetaQA is more than sufficient (He et al., 2021), hence all the comparison methods in our exper- iments can achieve very high performance. We randomly sample just one training case for each question template from the original training set, to form a one-shot training dataset following existing work (He et al., 2021; Jiang et al., 2022b). In this way, the numbers of training samples for MetaQA- lhop, MetaQA-2hop. and MetaQA-3hop are 161, 210, and 150, respectively.
2016b), GraphtNet (Sun et al., 2018), Pull- Net (Sun et al., 2019), NSM (He et al., 2021); (2) PI.M-based methods: EmbedKGQA (Sax- ena et al., 2020), TransferNet (Shi et al., 2021), SR+NSM+E2E (Zhang et al., 2022),
UniKGQA (Jiang et al., 2022b); (3) LLM--ased methods: Davinci-003 (Ouyang et al., 2022), Chat- GPT, StructGPT (Jiang et al., 2023). We give a detailed description of each baseline:
KV-Mem (Miller et al., 2016b) employs a key- value memory table to store KG facts and facilitates multi-hop reasoning through iterative read opera- tions on the memory.
GraftNet (Sun et al., 2018) first utilize a heuristic method to retrieve the question-relevant subgraph and text sentences from the Knowledge Graph (KG) and Wikipedia, respectively. Subse- quently, it employs graph neural network to con- duct multi-hop reasoning on heterogeneous graph constructed from the subgraph and text sentences. PulINet (Sun et al., 2019) trains a graph re- trieval model instead of the heuristic way in Graft- Net for the retrieval task, and then conducts multi- hop reasoning with GraftNet.
WebQuestionSSP (WebQSP) (Yih et al., 2015) consists of 4,737 questions. The answer entities are within a maximum of 2 hops from the topic entity on the Freebase (Bollacker et al., 2008) KG. We adopt the train/valid/test splits from Graft- Net (Sun et al., 2018) for consistency.
Complex WebQuestions 1.1 (CWQ) (Talmor and Berant, 2018) is constructed based on WebQSP, which is more challenging. It complicates WebQSP by extending the question entities or adding con- straints to restrict the answers. The answer entities are within a maximum of 4 hops from the topic entity on the Freebase (Bollacker et al., 2008) KG. Existing work (He et al., 2021) has demonstrated that the training data for MetaQA is more than sufficient. To better reflect the reasoning capability of baselines and our method, we extract one sample for each question template and conduct the one- shot experiment on all three MetaQA sub-datasets following existing work (He et al., 2021). Table 5 shows the statistics of the three datasets.
NSM (He et al., 2021) first conducts retrieval following GraftNet and then adapts the neural state machine (Hudson and Manning, 2019) used in vi- sual reasoning for multi-hop reasoming on the KG. It consists of a question understanding module based on LSTM and graph reasoning module with the adapted neural state machine, which is graph neural network in essence.
EmbedKGQA (Saxena et al., 2020) trans- forms the multi-hop reasoning process of Graft- Net into a link prediction task. This is achieved by comparing pre-trained entity embeddings with question representations derived from a Pre-trained Language Model (PLM).
TransferNet (Shi et al., 2021) first conducts retrieval following GraftNet and then performs the multi-hop reasoning on KG or a text-formed rela- tion graph in a transparent framework. consists of a PLM for question encoding and a graph neural
B Baselines
We consider the following three types of base- line methods for performance comparison: (1) non PLM-based methods: KV-Mem (Miller et al.,

---

network for updating the relevance scores between entities and the question.
C Ablation Study of Retrieval Subgraphs
SR+NSM+E2E (Zhang et al., 2022) first learns a PLM-based relation path retriever to con- duct effective retrieval and then leverages NSM rea- soner to perform multi-hop reasoning. The whole process is optimized in an end-to-end way.
We conduct experiments two strong base- lines (NSM and UniKGQA) with retrieval sub- graphs to explore the effect of the retrieval stage. We show the results in Table We can see that the two baselines achieve consistent performance improvement with our retrieved subgraphs (63.43% Hits @ of UniKGQA w Ours v.s. 50.7% Hits @ of UniKGQA and 61 .9% Hits@) I of NSM Ours V.s. 47.6% Hits @ of NSM). It indicates that our model can achieve better retrieval compared to isting retrieval methods. Although enhanced with our retrieval methods, the performance of the baselines still have a great gap with our Reason- ingLM, This demonstrates the effectiveness our ReasoningL.M to perform multi-hop reasoning over the subgraph.
UniKGQA (Jiang et al., 2022h) is unified model architecture based on PLMs for both re- trieval and reasoning stages. consists of PLM for computing the semantic similarity between the question and relation and a simple graph neural network for propagating the matching information. Davinci-003 (Ouyang et al., 2022) and Chat- GPT are both large language models developed by OpenAL. We can use their provided APIs to access them and solve KGQA tasks.
StructGPT (Jiang et al., 2023) general framework for improving the zero-shot reason- ing ability of LLMs over structured data, such as Knowledge Graph. It use an mnvoking-linearizttin generation procedure that leverages LLMs to read and perform reasoning based on the interface of structured data.
D Prompt for ChatGPT
Inspired by existing work (Taori et al., 2023), we show the prompt of generating questions used by ChatGPT in Table 7.

---

Table Performance of ReasoningLMM and two strong baselines (iee. NSM and UniKGQA) on CWQ based on our retrieval subgraphs represented by "w Ours".
CWQ
ReasoningL.M NSM
NSM w Ours
UniKGQA UniKGQA w Ours
Hits@1
69.0
47.6
61.9 50.1
50.7
63.43
42.4
48,0
$7.65
Here are the guidelines for formulating a question based on the given factual triples, object of the question, and the answer:
1. A factual triple consists of head entity, tail entity, and their relationship, representing a real-world fact. For example, (Gino Finizio, sex or gender, male) indicates that Gino Finizio is male. Your question should pertain to the provided entity based the factual background, and the answer provided should align with the provided answer.
The question should include all of the given information constraints, except for the provided answer, to ensure that all provided information is fully considered in deriving the answer.
Utilize many different entities and relations as possible in the question to promote variety. Questions should generally be one itself.
sentences and require added content aside from the question
Use subordinate clauses to link multiple triples the question, excluding intervening entities when pos- sible. For example, the knowledge "(Elevator Action, platform, Commodore 64), (Commodore 64, Giphy username, commadore)" be conveyed as "What the Giphy username for the platform of Flevator Ac- tion?"
provide a set 4 examples that you can reference: Example 1:
Given factual background: (Faler-Lagrange equation, discoverer or inventor, Leonhard Euler), (Leonhard Euler, student, Mikhail Golovin). Please generate a question about "Euler-Lagrange equation" and the answer to the question should be "Mikhail Golovin". ".
The question is: Who is the student that coined the Euler-Lagrang' equation? Example 2:
Given the factual background: (Spokane, population, 208,916), (Spokane, point time, 2007). Please generate a question about "Spokane" and the answer to the question should be "208,916". The question is: In 2007, what is the population of Spokane? Example 3:
Given the factual background: (Kristen Stewart, place of birth, Los Angeles), (Los Angeles, capital of, Los Angeles County). Please generate a question about "Kristen Stewart" and the answer to the question should be "Los Angeles".
The question is: What is the birth city of Kristen Stewart, which has the county seat of Los Angeles County?
Example 4:
Given the factual background: (Lampedusa Airport, country, Italy), (Italy, capital, Rome), (Italy, start time, 2 June 1946). Please generate question about "Lampedusa Airport" and the answer to the question should be "Rome".
The question is: Starting in 1946, what was the capital of the country to which Lampedusa Airport belonged?
Table 7: Instruction of generating questions in adaptation-tuning strategy.


Reviews:
[{"review_id": "G3yBt4Rsev", "reviewer": "Reviewer_hcTZ", "summary": "", "questions": "1. How will the proposed unified transformer graph encoding method benefit the research community? Can other transformer-based techniques be integrated after this unification? How does this approach differ from previous graph transformers?\n2. What is the motivation behind choosing efficient training strategies? How does the performance of ReasoningLM using efficient training compare to full parameter training strategies and existing efficient training methods?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows.\n\n**1.The motivation behind a unified transformer graph encoding method requires further clarification and justification.**\n\nWe have described the motivation for our approach in detail from line 43 to line 96. Here, we want to give a summarization and further clarification. Existing methods for KGQA utilize a PLM encoder such as BERT to attend to the relevant parts of the question, and a GNN reasoner such as GAT to update entity representations based on information from neighboring relations. However, there are two major shortcomings with this approach. First, **the loose integration of PLM and GNN** limits the fine-grained interaction between the question and KG. Second, **the GNN reasoner lacks rich semantic understanding** of relations within subgraphs, particularly for multi-hop relations. Recent work has shown that the transformer architecture of PLMs has the potential to model graph data effectively. Thus, we propose **incorporating the GNN reasoning process directly into the PLM** self to create a more capable PLM that supports both question understanding and structural subgraph reasoning. By doing so, we can enhance the interaction between questions and subgraphs and the subgraph reasoning with better relation semantics understanding. Leveraging the powerful capabilities of PLM, such a unified KGQA model can be effectively and efficiently deployed for different datasets with parameter-efficient fine-tuning, which is superior in realistic scenarios. Our experiments also demonstrate that our model achieves a significant improvement while updating fewer parameters compared to existing complex KGQA models. \n\n**2.The use of sparse masks for graph encoding has been explored in previous works, and the paper does not sufficiently differentiate its approach from those.**\n\nThanks for your suggestion and we will add this discussion in our paper. Existing sparse attention masks mainly aim to solely model the graph structure by adjusting the transformer architecture. They can not capture fine-grained interactions between the KG and the natural language question simultaneously, making them unsuitable for KGQA task. Besides, as our goal is to build a transferable and powerful KGQA model with efficient training cost, we also devise other new techniques paired with the sparse mask for improving the interaction and understanding between text question and graph structured KG. Concretely, we list the following three differences. \n\nFirstly, we design the special mask to enable the PLM to simultaneously model question understanding, deep interaction between question and subgraph, and reasoning over subgraph by leveraging adapted subgraph-aware self-attention mechanism. \n\nSecondly, since we do not modify the original model architecture, our approach allows the PLM to utilize pre-trained knowledge to better understand textual relations and entities in the subgraph and question when solving KGQA tasks. \n\nThirdly, we propose an adaptation tuning strategy that utilizes a few synthesized data to adapt the PLM to the special input format and attention mechanism for KGQA task. We propose an automatic data construction method for the KGQA task format using LLMs, such as ChatGPT, which has not been explored previously. Our data construction method could inspire the future work about human-free KGQA training data construction with the help of LLMs, since human annotated KGQA datasets are expensive [1].\n\n**3.How will the proposed unified transformer graph encoding method benefit the research community? Can other transformer-based techniques be integrated after this unification? How does this approach differ from previous graph transformers?**\n\nOur core contribution is to provide a new view for modeling the interaction and understanding between text and graph structure, especially for building transferable and powerful KGQA model with efficient training cost. Previous work adopts a mainstream method that utilizes a PLM as text encoder and a GNN as graph reasoner, and tunes all their parameters for learning the KGQA task. In our work, we find that the PLM can also play the role of GNN and only few synthesized examples can directly adapt it to the KGQA task. Thus, we provide a more unified approach that adapts PLM into GNN with few synthesized data and few tuned parameters leveraging the powerful capability of PLM, which is more efficient than existing work and also achieves the SOTA performance. In addition, our proposed method also boost the rethinking about the potential of PLM on KGQA tasks even other tasks that require understanding text information and graph structure information simultaneously, such as commonsense question answering [2] or drug modeling with textual description [3]. Since our approach does not change the model architecture of the PLM, meaning that it can benefit from the evolvement of advanced PLMs, even popular large language models. We leave the investigation about them as future work.\n\n**4.What is the motivation behind choosing efficient training strategies? How does the performance of ReasoningLM using efficient training compare to full parameter training strategies and existing efficient training methods?**\n\nAcross different domains and datasets, the transferability of existing KGQA models is always a challenging problem. Parameter-efficient tunings (e.g., LoRA and Prompt-tuning) have become a hot point for this topic. As only very few task-specific parameters are tuned and most parameters of PLM are shared, it just adds very few cost for the deployment of multiple models for different domains and datasets. In this work, we choose efficient training strategies for such advantage, and our efficient-tuned models have indeed shown its strong performance across different datasets, surpassing SOTA full-parameter tuning models. Our approach also has the following advantages for its good performance. First, we don't modify the architecture of PLM, and thus we can keep its pre-trained knowledge to be utilized and transferred across datasets. Second,  we further propose an adaptation tuning strategy to adapt the original PLM to performing question understanding, interaction between the question and subgraph, and reasoning over the subgraph, controlled by the attention mask. Hence it can be applied to different downstream KGQA tasks with parameter-efficient fine-tuning manner, achieving better performance with only updating a few parameters. \n\nIn addition, since we do not modify the original PLM architecture, existing parameter-efficient fine-tuning methods can be seamlessly used. In our paper, we just select one representative adapter tuning [4] method among existing methods. Here, we present additional ablation experiments that compare the performance of full parameter fine-tuning strategy with existing parameter-efficient fine-tuning strategies, including adapter tuning (our used), LoRA [5], (IA)^3 [6], and UniPELT [7]. Our results show that all parameter-efficient fine-tuning methods are able to achieve better results than existing strong baseline (UniKGQA), and their performance is almost indistinguishable from each other. This indicates that our proposed method is compatible with various parameter-efficient fine-tuning methods, and has a good robustness. Although full parameter fine-tuning provides further performance improvements, it needs a cost of updating more than 100 times the number of parameters required by adapter tuning. Such a way enables researchers to trade off the efficiency and effectiveness when employing our approach in real-world applications.\n\n\\\\begin{array} {|l|c|c|}\n\\\\hline\n& \\\\textrm{WebQSP (Hits@1)} & \\\\textrm{CWQ (Hits@1)} \\\\\\\\\n\\\\hline\n\\\\textrm{UniKGQA} & 75.1 & 50.7 \\\\\\\\\n\\\\hline\n  \\\\textrm{Ours w full parameter} & 80.1 & 71.2 \\\\\\\\\n  \\\\textrm{Ours w adapter tuning}  & 78.5 & 69.0 \\\\\\\\\n  \\\\textrm{Ours w LoRA}  & 77.8 & 67.5 \\\\\\\\\n  \\\\textrm{Ours w (IA\\)3}  & 78.0 & 68.1 \\\\\\\\\n  \\\\textrm{Ours w UniPELT}  & 78.3 & 69.4 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nReference:\n\n[1] Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, Yu Su: Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases. WWW 2021.\n\n[2] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec: QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. NAACL 2021.\n\n[3] Philipp Seidl, Andreu Vall, Sepp Hochreiter, Günter Klambauer: Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language. ICML 2023.\n\n[4] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly: Parameter-Efficient Transfer Learning for NLP. ICML 2019.\n\n[5] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu W ang, Weizhu Chen: LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.\n\n[6] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS 2022.\n\n[7] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, Madian Khabsa: UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning. ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a novel transformer-only graph encoding method that combines question understanding and KG reasoning.\n2. ReasoningLM achieves strong results on multiple KGQA datasets, showcasing its effectiveness.\n3. The paper introduces efficient training strategies, which contribute to the model's performance improvements.", "reasons_to_reject": "1. The motivation behind a unified transformer graph encoding method requires further clarification and justification.\n2. The use of sparse masks for graph encoding has been explored in previous works, and the paper does not sufficiently differentiate its approach from those.\n3. The rationale behind choosing efficient training strategies is not well-explained, and a comparison with full parameter training strategies and existing efficient training methods would be valuable.", "questions_to_authors": "1. How will the proposed unified transformer graph encoding method benefit the research community? Can other transformer-based techniques be integrated after this unification? How does this approach differ from previous graph transformers?\n2. What is the motivation behind choosing efficient training strategies? How does the performance of ReasoningLM using efficient training compare to full parameter training strategies and existing efficient training methods?", "paper_topic_and_main_contributions": "The paper introduces ReasoningLM, a pre-trained language model (PLM) that incorporates subgraph reasoning to enhance question understanding and knowledge graph (KG) reasoning. The proposed approach utilizes a breadth-first search (BFS)-based subgraph serialization technique and introduces a subgraph-aware self-attention mechanism within the PLM. An adaptation tuning strategy using ChatGPT is employed to adapt the PLM to the specialized input format and attention mechanism, reducing data construction costs. ReasoningLM is applied to downstream KGQA tasks using parameter-efficient fine-tuning and demonstrates superior performance compared to existing baselines, achieving significant improvements by updating only a fraction of the model's parameters.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jiQZ5oKnmS", "reviewer": "Reviewer_UNw1", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows.\n\n**1.The method is incremental compared to existing methods and the self-attention mechanism is also widely used in other KBQA studies.**\n\nExisting KBQA studies mainly adopt a combination of PLM as text encoder and GNN as graph reasoner to solve the KGQA task. In the two sub-modules, they indeed leverage the self-attention mechanism to understand the complex question or perform message propagation-and-aggregation over the subgraph. While our goal is to provide a new view for modeling the interaction and understanding between text question and graph structured KG, especially for building transferable and powerful KGQA model with efficient training cost. Concretely, we list the following three differences. Concretely, we have the following three differences. \n\nFirstly, inspired by these graph transformer variants, we enable the PLM to simultaneously model question understanding, deep interaction between question and subgraph, and reasoning over subgraph by leveraging adapted subgraph-aware self-attention mechanism. \n\nSecondly, since we do not modify the original model architecture, our approach allows the PLM to utilize pre-trained knowledge to better understand textual relations and entities in the subgraph and question when solving KGQA tasks. \n\nThirdly, since the PLM is originally trained by learning from general-purpose natural language text, we further propose an adaptation tuning strategy that utilizes a few synthesized data to adapt the PLM to the special input format and attention mechanism. We propose an automatic data construction method for the KGQA task format using LLMs, such as ChatGPT, which has not been explored previously. Our data construction method could inspire the future work about human-free KGQA training data construction with the help of LLMs, since human annotated KGQA datasets are expensive [1].\n\nIn addition to these differences, our proposed method also boost the rethinking about the potential of PLM on KGQA tasks even other tasks that require understanding text information and graph structure information simultaneously, such as commonsense question answering [2] or drug modeling with textual description [3].\n\n**2.More case studies and error case studies should be performed to help see the benefit of the proposed method.**\n\nThanks for your suggestion and we would add more case studies, including error cases after review. Here, we give one case study to see the benefit of our proposed method compared to UniKGQA, a strong baseline which utilizes PLM to model the question and relations and an extra GNN model to perform reasoning over the subgraph. The detailed illustration is as follows, where the predicted reasoning path (\"?x\" represent the answer entities) is extracted from the reasoning process according to the attention weight:\n\n|      | UniKGQA | ReasoningLM | \n| ----------- | ----------- |----------- |\n| question |    who is queen elizabeth 1 father ?  |   who is queen elizabeth 1 father ? |\n| linked topic entity | \"elizabeth i of england\" (m.02rg_) |\"elizabeth i of england\" (m.02rg_) |\n| predicted reasoning path | (m.02rg_, people.person.parents, ?x) | people.person.parents, ?x), (?x, people.person.gender male) |\n| prediction answers | Henry VIII of England (m.03p77), Anne Boleyn (m.09sxn)|Henry VIII of England (m.03p77)|\n|gold answer|Henry VIII of England (m.03p77)|Henry VIII of England (m.03p77)|\n\nAs we can see, starting from the topic entity (i.e., m.02rg_), the model needs to find the \"people.person.parents\" relation to obtain his parents, and then further obtain the farther according to the \"people.person.gender\" relation and \"male\" property. However, the GNN module of UniKGQA can only find the \"people.person.parents\" relation relying on the message propagation-and-aggregation based on the graph structure while fail to attend the further gender property constraint. Therefore, it would return both the mother and father, which is incorrect. In contrast, ReasoningLM can attend the further gender constraint (i.e., (?x, people.person.gender male)) by modeling the question semantics and the connected triples (i.e., (m.02rg_, people.person.parents, ?x), (?x, people.person.gender male) ) in the subgraph within a single PLM. Therefore, our proposed ReasoningLM can better model the interaction between question and subgraph based on the subgraph-aware self-attention mechanism and better understand the question semantics and textual relations and entities in subgraph using the pre-trained knowledge, which is difficult for existing methods that combine a PLM text encoder and a GNN graph reasoner.\n\nReference:\n\n[1] Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, Yu Su: Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases. WWW 2021.\n\n[2] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec: QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. NAACL 2021.\n\n[3] Philipp Seidl, Andreu Vall, Sepp Hochreiter, Günter Klambauer: Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language. ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper is well-written and easy to follow. \n2.The experimental results prove the effectiveness of the proposed method.\n3.The method is simple and straightforward.", "reasons_to_reject": "1.The method is incremental compared to existing methods and the self-attention mechanism is also widely used in other KBQA studies.\n2.More case studies and error case studies should be performed to help see the benefit of the proposed method.", "questions_to_authors": "None", "paper_topic_and_main_contributions": "This paper focus on the knowledge graph question answering problem. The authors argue that existing methods utilize LM to model queries and utilize GNNs to model KGs. However, they are not combined thoroughly. This paper proposes a method to fuse the above methods and proposes ReasoningLM. The experiments on several public datasets demonstrate the effectiveness of this approch.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "merNHE0lLH", "reviewer": "Reviewer_3YnV", "summary": "", "questions": "1. Could the authors provide performance metrics for the ablated system with PET (Partial Rule-SYN) to allow for a fair comparison with PET (LLM-SYN)? This would help determine which of the two approaches is more beneficial when provided the same amount of training data.\n\n2. In Line 323, it would be helpful to elaborate on whether the position embeddings E respect the structural positions of subgraph elements. This clarification would provide insight into the model's understanding of subgraph structure.\n\n3. The rationale behind the choice of using KL divergence loss instead of conventional cross-entropy loss (Eq. 5) is not explicitly discussed. Could the authors elaborate on the advantages of KL divergence in this context and whether it leads to substantial improvements?\n\n4. Addressing the concerns about the BFS-based subgraph serialization is crucial. BFS-based subgraph serialization might incorporate irrelevant triples since not all triples in the k-hop subgraph are useful. The num of triples included can be very large when the k becomes large. How does the proposed approach handle cases where multiple potentially disjoint subgraphs are far apart? Additionally, clarifying the construction of subgraphs in such scenarios would enhance the understanding of the model's capabilities.\n\n5. The paper mentions that the proposed approach requires knowledge of the Golden topic entity. Could the authors clarify whether this is a unique requirement of their approach or if it applies to other baseline methods as well?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows.\n\n**1.Incremental Novelty: The novelty of the proposed approach is somewhat incremental, as similar concepts of structural self-attention have been explored in other graph transformer variants, such as the ETC paper [1]. Additionally, the adaptation tuning strategy appears straightforward. To strengthen the paper's contribution, the authors could discuss more explicitly how their approach differs or improves upon these prior works.**\n\nThanks for your suggestion and we will add this discussion and cite the relevant ETC paper in our paper. Existing graph transformer variants, such as the ETC model, mainly aim to model the structure information or scale up the input length by adjusting the transformer architecture. While our goal is to provide a new view for modeling the interaction and understanding between text question and graph structured KG, especially for building transferable and powerful KGQA model with efficient training cost. Concretely, we list the following three differences.\n\nFirstly, inspired by these graph transformer variants, we enable the PLM to simultaneously model question understanding, deep interaction between question and subgraph, and reasoning over subgraph by leveraging adapted subgraph-aware self-attention mechanism. \n\nSecondly, since we do not modify the original model architecture, our approach allows the PLM to utilize pre-trained knowledge to better understand textual relations and entities in the subgraph and question when solving KGQA tasks. \n\nThirdly, since the PLM is originally trained by learning from general-purpose natural language text, we further propose an adaptation tuning strategy that utilizes a few synthesized data to adapt the PLM to the special input format and attention mechanism. We propose an automatic data construction method for the KGQA task format using LLMs, such as ChatGPT, which has not been explored previously. Our data construction method could inspire the future work about human-free KGQA training data construction with the help of LLMs, since human annotated KGQA datasets are expensive [1].\n\nIn addition to these differences, our proposed method also boost the rethinking about the potential of PLM on KGQA tasks even other tasks that require understanding text information and graph structure information simultaneously, such as commonsense question answering [2] or drug modeling with textual description [3].\n\n**2.Fair Comparison: The comparison with baseline methods raises concerns about fairness due to the availability of additional training data during the adaptation tuning stage (which has proven to be very crucial to the success of the approach). To address this, the authors should consider training the baseline models with the same additional data and conducting a fair comparison to better assess the effectiveness of the proposed approach.**\n\nIn fact, the adaptation tuning dataset is established using the Wikidata KG, which differs from the Freebase KG used in the downstream WebQSP and CWQ datasets, as well as the movie domain KG used in MetaQA dataset. Since most of the baselines model the subgraph reasoning with GNN, they require specific entity embeddings and relation embeddings for each dataset. Despite training the baseline models with the same additional data, it is still difficult for previous methods to utilize the learned knowledge during the adaptation tuning phrase when fine-tuned on the downstream tasks, since new entity embeddings and relation embeddings would be reinitialized for different KGs. Here, we conduct an extra ablation experiment to compare our model with some strong baseline models that are also trained using the same adaptation tuning data. The results indicate that the additional adaptation tuning data does not yield any improvement for the baseline models.\n\\\\begin{array} {|l|c|c|}\n\\\\hline\n& \\\\textrm{WebQSP (Hits@1)} & \\\\textrm{CWQ (Hits@1)} \\\\\\\\\n\\\\hline\n\\\\textrm{NSM} & 68.7 & 47.6 \\\\\\\\\n\\\\hline\n  \\\\textrm{UniKGQA} & 75.1 & 50.7 \\\\\\\\\n\\\\hline\n  \\\\textrm{NSM w adaptation tuning}  & 67.6 & 48.1 \\\\\\\\\n\\\\hline\n  \\\\textrm{UniKGQA w adaptation tuning}  & 74.9 & 51.1 \\\\\\\\\n\\\\hline\n  \\\\textrm{ReasoningLM w adaptation tuning (Ours)}  & 78.5 & 69.0 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n**3.Could the authors provide performance metrics for the ablated system with PET (Partial Rule-SYN) to allow for a fair comparison with PET (LLM-SYN)? This would help determine which of the two approaches is more beneficial when provided the same amount of training data.**\nThanks for your suggestion! We show the performance of PET (Rule-SYN) in the following table. In the table, \"FPT\" and \"PET\" denote adaption tuning with full-parameter tuning and parameter-efficient tuning, respectively. \"Rule-SYN\" and \"LLM-SYN\" refer to synthesized tuning data using rule-based and LLM-based strategies, respectively. Our results demonstrate that high-quality LLM synthesized data is more beneficial than rule-based synthesized data in both full-parameter and parameter-efficient tuning settings, especially for the complex KGQA dataset like CWQ. Besides, using the same synthesized data, full-parameter tuning is obviously better than parameter-efficient tuning.\n\n\\\\begin{array} {|l|c|c|}\n\\\\hline\n& \\\\textrm{WebQSP (Hits@1)} & \\\\textrm{CWQ (Hits@1)} \\\\\\\\\n\\\\hline\n\\\\textrm{FPT + LLM-SYN} & 78.5 & 69.0 \\\\\\\\\n\\\\hline\n\\\\textrm{FPT + Rule-SYN} &  78.0 & 62.8 \\\\\\\\\n\\\\hline\n\\\\textrm{PET + LLM-SYN} &  76.7 & 68.3 \\\\\\\\\n\\\\hline\n\\\\textrm{PET + Rule-SYN} &  75.1 & 61.3 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n**4.In Line 323, it would be helpful to elaborate on whether the position embeddings E respect the structural positions of subgraph elements. This clarification would provide insight into the model's understanding of subgraph structure.**\n\nIn our implementation, we do not modify the original position embedding mechanism and do not utilize the position embeddings to represent the structural position, as our prior experiments have shown that it does not bring performance improvements and even hurt the model performance. The possible reason is that it affects the original natural language understandings of PLM, such new knowledge is hard to be efficiently learned by our few augmented data.  Interestingly, we find that our attention distribution can well match the subgraph structure to some extent.\n\n**5.The rationale behind the choice of using KL divergence loss instead of conventional cross-entropy loss (Eq. 5) is not explicitly discussed. Could the authors elaborate on the advantages of KL divergence in this context and whether it leads to substantial improvements?**\n\nThe model produces a probability distribution over the subgraph's entities to indicate potential answers for the question. Additionally, the label is a probability distribution of each entity in the subgraph where the answer entity's position is assigned 1 and the remaining entities as 0. For multiple answers, we normalize the distribution. To minimize the distance between these two probability distributions, we utilize the KL divergence loss, as commonly used in other baselines. Alternatively, the cross-entropy loss can be used, which has the same effect as the objective function but differs by a constant mathematically. We select the KL divergence loss in this study for convenient comparisons with existing works.\n\n**6.Addressing the concerns about the BFS-based subgraph serialization is crucial. BFS-based subgraph serialization might incorporate irrelevant triples since not all triples in the k-hop subgraph are useful. The num of triples included can be very large when the k becomes large. How does the proposed approach handle cases where multiple potentially disjoint subgraphs are far apart? Additionally, clarifying the construction of subgraphs in such scenarios would enhance the understanding of the model's capabilities.**\n\nIn fact, the scale of the subgraph is relatively small, since we have first performed a subgraph retrieval from the original KG following the mainstream retrieval-then-reasoning paradigm used in existing work [1-4]. The retrieval stage focuses on extracting a relatively small subgraph involving the answer entities. The reasoning stage aims to accurately find the answer entities of the given question by walking along the relations starting from the topic entities. Typically, the number of triples in a subgraph is less than 500. We only retrieve the subgraph starting from one topic entity and avoid cases where multiple disjoint subgraphs correspond to distant topic entities. Concretely, starting from the topic entity, our model, fine-tuned on the subgraph retrieval subtask following existing work [2][5] , iteratively measures the semantic relevance between the question and neighbouring relations, and adds proper ones and their corresponding triples into the subgraph. In this way, a smaller but more question-relevant subgraph would be constructed. \n\n**7.The paper mentions that the proposed approach requires knowledge of the Golden topic entity. Could the authors clarify whether this is a unique requirement of their approach or if it applies to other baseline methods as well?**\n\nFor the adaptation tuning stage, since the tuning data is constructed in an automatic way, we can directly obtain the topic entity and use it when performing adaptation tuning. After adapatation tuning, we do not assume the given golden topic entity when fine-tuned on downstream KGQA tasks. And we just follow existing work [4-8] to perform entity linking in advance with available tools (e.g., Google Knowledge Graph Search API) and models (e.g., ELQ [9]). Our experiment settings are consistent with other baselines, which  makes sure a fair comparison in Table 2.\n\nReference:\n\n[1] Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, Yu Su: Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases. WWW 2021.\n\n[2] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec: QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. NAACL 2021.\n\n[3] Philipp Seidl, Andreu Vall, Sepp Hochreiter, Günter Klambauer: Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language. ICML 2023.\n\n[4] Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William W. Cohen: Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. EMNLP 2018.\n\n[5] Apoorv Saxena, Aditay Tripathi, Partha P. Talukdar: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings. ACL 2020.\n\n[6] Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, Ji-Rong Wen: Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. WSDM 2021.\n\n[7] Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, Hanwang Zhang: TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph. EMNLP 2021.\n\n[8] Jinhao Jiang, Kun Zhou, Xin Zhao, Ji-Rong Wen: UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph. ICLR 2023.\n\n[9] Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, Wen-tau Yih: Efficient One-Pass End-to-End Entity Linking for Questions. EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Well-Motivated Approach:** The proposed approach is well-motivated and addresses a significant challenge in KGQA, namely enhancing subgraph reasoning. Comprehensive experiments have been conducted to verify the effectiveness of the proposed approach.\n\n2. **Broad Applicability:** The paper highlights the agnostic nature of the proposed approach with respect to PLMs, which is demonstrated through experiments in Table 3. This feature increases the versatility of the model and its potential applicability to different settings.", "reasons_to_reject": "1. **Incremental Novelty:** The novelty of the proposed approach is somewhat incremental, as similar concepts of structural self-attention have been explored in other graph transformer variants, such as the ETC paper [1]. Additionally, the adaptation tuning strategy appears straightforward. To strengthen the paper's contribution, the authors could discuss more explicitly how their approach differs or improves upon these prior works.\n\n2. **Fair Comparison:** The comparison with baseline methods raises concerns about fairness due to the availability of additional training data during the adaptation tuning stage (which has proven to be very crucial to the success of the approach). To address this, the authors should consider training the baseline models with the same additional data and conducting a fair comparison to better assess the effectiveness of the proposed approach.\n\n[1] Ainslie, Joshua, et al. \"ETC: Encoding long and structured inputs in transformers.\" arXiv preprint arXiv:2004.08483 (2020).", "questions_to_authors": "1. Could the authors provide performance metrics for the ablated system with PET (Partial Rule-SYN) to allow for a fair comparison with PET (LLM-SYN)? This would help determine which of the two approaches is more beneficial when provided the same amount of training data.\n\n2. In Line 323, it would be helpful to elaborate on whether the position embeddings E respect the structural positions of subgraph elements. This clarification would provide insight into the model's understanding of subgraph structure.\n\n3. The rationale behind the choice of using KL divergence loss instead of conventional cross-entropy loss (Eq. 5) is not explicitly discussed. Could the authors elaborate on the advantages of KL divergence in this context and whether it leads to substantial improvements?\n\n4. Addressing the concerns about the BFS-based subgraph serialization is crucial. BFS-based subgraph serialization might incorporate irrelevant triples since not all triples in the k-hop subgraph are useful. The num of triples included can be very large when the k becomes large. How does the proposed approach handle cases where multiple potentially disjoint subgraphs are far apart? Additionally, clarifying the construction of subgraphs in such scenarios would enhance the understanding of the model's capabilities.\n\n5. The paper mentions that the proposed approach requires knowledge of the Golden topic entity. Could the authors clarify whether this is a unique requirement of their approach or if it applies to other baseline methods as well?", "paper_topic_and_main_contributions": "The paper presents an approach to enhance subgraph reasoning in Knowledge Graph Question Answering (KGQA) by developing a more capable Pre-trained Language Model (PLM) that incorporates subgraph-aware self-attention and an adaptation tuning strategy. The proposed model, named ReasoningLM, outperforms state-of-the-art models in KGQA tasks. The authors conduct extensive experiments, demonstrating the effectiveness of their approach in handling structured reasoning tasks and adapting model parameters with synthesized subgraphs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Ainslie, Joshua, et al. \"ETC: Encoding long and structured inputs in transformers.\" arXiv preprint arXiv:2004.08483 (2020)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new framework that incorporates sub-graph structures into PLMs. In particular, their new framework, ReasoningLM uses subgraph-aware self-attention. Their experimental results show the effectiveness of the proposed method on multiple KBQA. While I agree with the concern about limited novelties noted by all of the reviewers (i.e. sparse masking of transformers has been explored in prior work), incorporating subgraphs for effective masking requires substantial work, and experimental results on KBQA are strong. Therefore, I recommend for acceptance.
Decision: Accept-Main