Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs
Framework for Knowledge Graph Question Answering
Yike Wu∗

Nan Hu∗Sheng Biyike.wu@seu.edu.cn
Southeast University
Nanjing, Jiangsu, China

nanhu@seu.edu.cn
Southeast University
Nanjing, Jiangsu, China

bisheng@seu.edu.cn
Southeast University
Nanjing, Jiangsu, China

Guilin Qi

Jie RenAnhuan Xie

gqi@seu.edu.cn
Southeast University
Nanjing, Jiangsu, Chinarenjie@zhejianglab.com
Zhejiang Lab
Hangzhou, Zhejiang, China

xieanhuan@zhejianglab.com
Zhejiang Lab
Hangzhou, Zhejiang, China

Wei Song†
weisong@zhejianglab.com
Zhejiang Lab
Hangzhou, Zhejiang, China

ABSTRACTKEYWORDS

Despite their competitive performance on knowledge-intensive
tasks, large language models (LLMs) still have limitations in memorizing all world knowledge especially long tail knowledge. In thispaper, we study the KG-augmented language model approach forsolving the knowledge graph question answering (KGQA) taskthat requires rich world knowledge. Existing work has shown that
retrieving KG knowledge to enhance LLMs prompting can significantly improve LLMs performance in KGQA. However, theirapproaches lack a well-formed verbalization of KG knowledge, i.e.,they ignore the gap between KG representations and textual representations. To this end, we propose an answer-sensitive KG-to-Text
approach that can transform KG knowledge into well-textualizedstatements most informative for KGQA. Based on this approach,we propose a KG-to-Text enhanced LLMs framework for solving
the KGQA task. Experiments on several KGQA benchmarks showthat the proposed KG-to-Text augmented LLMs approach outperforms previous KG-augmented LLMs approaches regarding answer
accuracy and usefulness of knowledge statements.1

Knowledge Graph Augmented LLMs, Knowledge Graph Question
Answering, KG-to-Text

CCS CONCEPTS• Computing methodologies → Artificial intelligence; Natural language processing; Natural language generation.
∗ Both authors contributed equally to this research.
† Corresponding author
1 Our code is available at https://github.com/wuyike2000/Retrieve-Rewrite-Answer

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
IJCKG’23, December 08–09, 2023, Tokyo, Japan© 2023 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
https://doi.org/XXXXXXX.XXXXXXXACM Reference Format:
Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, Jie Ren, Anhuan Xie, and Wei Song.
2023. Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Frameworkfor Knowledge Graph Question Answering. In Proceedings of The 12th International Joint Conference on Knowledge Graphs (IJCKG’23). ACM, New
York, NY, USA, 10 pages. https://doi.org/XXXXXXX.XXXXXXX

1INTRODUCTION

Large language models (LLMs) are becoming increasingly popular in natural language processing for their superior competence
in various applications. Although LLMs demonstrate remarkablecapabilities in zero-shot scenarios, their performances on several
knowledge-intensive tasks are insufficiently satisfactory [21]. Thisreveals the enormous parameters in LLMs cannot store all theworld’s knowledge. Several researches indicate that LLMs still suffer from issues like hallucinations and factual inaccuracy whenanswering questions [12, 18, 27]. Specifically, LLMs perform inadequately in the knowledge-intensive task KGQA [9, 35].
To solve this problem, recent work attempts to enhance LLMs
with external knowledge. A line of work [10, 20, 39, 42] involves continual pre-training LLMs on extensive corpora. Nevertheless, thismethod requires a significant amount of textual data, computational
resources, and time investment. Considering knowledge graphs contain a large amount of high-quality structured knowledge [43, 44],prior efforts have sought to explicitly elevate the performance of
LLMs by extracting pertinent knowledge from them. Additionally,
alternative endeavors explore diverse knowledge reservoirs, including web content and documentation [11, 14, 19]. This approach is
employed to address the model’s deficiency in factual knowledge. Inspired by this, other work [1, 33] constructs knowledge-augmented
prompt by prepending question-related factual information to thequestion to enrich the knowledge of LLMs in a more direct fashion.

IJCKG’23, December 08–09, 2023, Tokyo, Japan

Wu and Hu, et al.

Although this approach proves to be successful and cost-effective,
it ignores the importance of knowledge representation.

Figure 2: Comparison between Retrieve-then-Answer framework and our proposed Retrieve-Rewrite-Answer framework.
We use a rewriter module to transform the retrieved subgraphs into textual descriptions.
Figure 1: Two forms of knowledge representation: tripleform text and free-form text. Triple-form text transforms
subgraph to text via simple linear concatenation while freeform text adopts the KG-to-Text method to generate a semantically coherent textual description.
In this paper, we summarize two knowledge representation formats employed in previous work: triple-form text and free-form
text. As shown in Figure 1, triple-form text involves a simple linear
concatenation of triples, transforming them into structured text.
Free-form text converts triples into semantically coherent textual
descriptions based on the KG-to-Text method. Further, we proposea KG-to-Text enhanced framework, Retrieve-Rewrite-Answer tostrengthen the performance of LLMs on KGQA. As shown in Figure 2, compared with previous work that answers questions in a
Retrieve-then-Answer fashion, our framework adopts a Rewriter
module to transform the retrieved triples into textual descriptions,which is more conducive to the understanding of LLMs and ismore beneficial for KGQA. The core of this framework lies in thetask-driven KG-to-Text method. Our designed method is answersensitive and can transform question-related triples into textual
knowledge which is most informative to KGQA. Compared withprevious work that simply adopts an off-the-shelf KG-to-Text model,
we fine-tune open-source LLMs on the KG-to-Text corpus to generate knowledge descriptions beneficial to KGQA. However, themajor challenge is the lack of KG-to-Text annotated data in existing
KGQA benchmarks. Therefore, we propose an automatic corpus
generation method for generating high-quality graph-text pairs
based on the feedback of LLMs.
The contributions of this paper are summarized as follows:• We propose Retrieve-Rewrite-Answer, a KG-to-Text enhanced
LLMs framework for KGQA. In comparison to previous KGaugmented LLMs frameworks, the most significant innovation in our framework lies in the rewrite module. This
module uses fine-tuned LLMs as KG-to-Text models to convert retrieved subgraphs into well-textualized statements
most informative for KGQA.
• To address the issue of scarcity of KG-to-Text corpus annotations, we devise an automatic KG-to-Text corpus generation
method. We extract question-related subgraphs and utilize

ChatGPT as a tool for corpus generation. Based on the feedback from question-answering LLMs, we generate answersensitive knowledge descriptions for the construction of
KG-to-Text labeled data. We fine-tune several open-source
LLMs on the generated corpus and investigate the impact of
textual knowledge generated by different LLMs on KGQA.
• We evaluate our framework on four KGQA benchmarks. Experimental results show that our framework outperforms
previous KG-augmented methods across several LLMs, which
demonstrates its effectiveness. Besides, we investigate theinfluence of different knowledge representation formats on
KGQA and prove that the knowledge produced by our framework is most beneficial.

2

PRELIMINARIES

Knowledge Graph (KG) is a collection of triples (𝑠, 𝑟, 𝑜) composedof subject 𝑠, relation 𝑟 and object 𝑜, denoted by 𝐺 = {(𝑠, 𝑟, 𝑜)|𝑠, 𝑜 ∈𝐸, 𝑟 ∈ 𝑅}, where 𝐸 and 𝑅 denote the entity set and relation set.
KG-to-Text is a natural language generation technique basedon KG. Given a subgraph 𝐺 ′ = {(𝑠, 𝑟, 𝑜)|𝑠, 𝑜 ∈ 𝐸, 𝑟 ∈ 𝑅} fromKG 𝐺, the objective of KG-to-Text is to generate a text sequence
𝑋 = (𝑥 1, 𝑥 2, ..., 𝑥𝑛 ) that is semantically coherent with subgraph 𝐺 ′ .
Knowledge Graph Question Answering (KGQA) is the taskof answering natural language questions based on a set of facts
over KGs. Given a question 𝑞 and a topic entity 𝑒ℎ , the task is togenerate an answer 𝑎 that can correctly respond to the question.

3 RELATED WORK
3.1 KG-Augmented LLM for KGQA
Despite LLMs’ pre-training on massive corpora, they still sufferfrom hallucinations, factual inaccuracy, and outdated knowledge
when it comes to KGQA. Recent efforts have aimed to harness
KGs to enhance the capabilities of LLMs on KGQA [1, 33]. In thesestudies, question-related triples are extracted from KG and transformed into textual format using techniques such as linear verbalization or off-the-shelf KG-to-Text models. The textual representationof triples and the question are transformed into the knowledgeaugmented prompt via predefined templates. The prompt is then fed
into the question-answering LLM to produce more reliable answers.

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

IJCKG’23, December 08–09, 2023, Tokyo, Japan

While these studies demonstrate the effectiveness of this strategy,
they ignore the impact of the knowledge representation format
on the performance of LLMs on KGQA. In this study, we devise
a KG-to-Text corpus generation method to produce high-quality
annotations. Then we utilize LLMs fine-tuned on the corpus totransform question-related subgraphs into knowledge text, which
can further elevate the performance of LLMs on KGQA.

3.2

KG-to-Text

Recent studies in KG-to-Text can be classified into two main approaches: 1) based on graph neural networks (GNNs) [41]: To
preserve the structural information of subgraphs during encoding, researchers focus on designing more complex encoders to
obtain enhanced subgraph representation. Some work has developed graph-structured encoders based on GNNs [6, 22] due to therobust capabilities of GNNs in encoding graph-structured data.
However, GNNs are restricted by local processing nature, as they
iteratively compute node representations based on neighboring
nodes’ features. To overcome this limitation, alternative efforts
have employed Transformer-based architecture [40] in designing
encoders [3, 15, 49]. This approach allows for the establishment ofconnections between any two nodes within the graph, thereby addressing the limitations of GNNs. 2) based on pre-trained language
models (PLMs): With the development of large-scale generative
PLMs such as BART [17], T5 [25] and GPT [24], recent work applies
these models to KG-to-Text, modeling it as an end-to-end generation task [7, 13, 26]. These studies involve modifications to themodel architecture and the introduction of pre-training tasks toenhance the ability to extract structural information. We follow thesecond approach and fine-tune LLMs on the KG-to-Text corpus.

4 METHODS
4.1 Retrieve-Rewrite-Answer
As shown in Figure 3, our proposed Retrieve-Rewrite-Answer framework contains three steps: subgraph retrieval, KG-to-Text, andknowledge text enhanced reasoning.
4.1.1 Retrieve: Subgraph Retrieval. Our retrieve module consistsof three steps: hop prediction, relation path prediction, and triple
sampling, as illustrated in Figure 4.
Hop Prediction. The aim of this step is to predict the hop
number of the question, which is used to predict the relation pathin the next step. We model the hop prediction as a classification
task based on PLMs. Given the question 𝑞, we employ PLMs to
encode the question 𝑞 and obtain the vector representation 𝑞 𝑣 :𝑞 𝑣 = 𝑃𝐿𝑀 (𝑞)(1)

The representation 𝑞 𝑣 is then fed into the linear classification layer
to predict the probability distribution 𝐷ℎ′ of potential hop numbersℎ 1 , ℎ 2 , ..., ℎ𝐻 :𝐷ℎ′ = [𝑑ℎ′ , 𝑑ℎ′ , ..., 𝑑ℎ′ ] = 𝐿𝑖𝑛𝑒𝑎𝑟 (𝑞 𝑣 )
1
2
𝐻

(2)

where 𝑑ℎ′ is the probability of the hop number ℎ𝑐 given the question𝑐representation 𝑞 𝑣 :𝑑ℎ′ = 𝑃 (ℎ𝑐 |𝑞 𝑣 ),𝑐𝑐 = 1, 2, ..., 𝐻(3)

Figure 3: Our proposed framework, Retrieve-Rewrite-Answer
has three steps: subgraph retrieval, KG-to-Text, and knowledge text enhanced reasoning.

The hop number ℎ with the highest probability is selected as thepredicted result.
ℎ = arg max 𝑑ℎ′ ,ℎ𝑐𝑐 = 1, 2, ..., 𝐻(4)𝑐During training, ground truth distribution 𝐷ℎ is represented as a
one-hot vector with a probability of 1 for the true hop number ℎ𝑔𝑜𝑙𝑑
and probabilities of 0 for other hop numbers:
𝐷ℎ = [𝑑ℎ1 , 𝑑ℎ2 , ..., 𝑑ℎ𝐻 ](1, ℎ𝑐 = ℎ𝑔𝑜𝑙𝑑𝑑ℎ𝑐 =𝑐 = 1, 2, ..., 𝐻0, ℎ𝑐 ≠ ℎ𝑔𝑜𝑙𝑑

(5)
(6)

The predicted distribution 𝐷ℎ′ is penalized for being different fromthe ground truth distribution 𝐷ℎ using cross entropy loss 𝐿𝐶𝐸 :𝐿𝐶𝐸 = −𝐷ℎ log 𝐷ℎ′ = −𝐻∑︁𝑑ℎ𝑐 log 𝑑ℎ′𝑐=1

𝐿𝐶𝐸 is utilized to update the model’s parameters.

(7)𝑐

IJCKG’23, December 08–09, 2023, Tokyo, Japan

Wu and Hu, et al.

where 𝑑𝑟′𝑐 is the probability of relation 𝑟𝑐 given the input sequence
representation 𝑄𝑡,𝑣 :𝑑𝑟′𝑐 = 𝑃 (𝑟𝑐 |𝑄𝑡,𝑣 ),𝑐 = 1, 2, ..., 𝑅(14)

We retain top 𝐾 relations with the highest probabilities as the 𝑡-th
hop relations for the (𝑡 − 1) hop relation path 𝑝𝑡 −1,𝑖 .
After ℎ-step prediction, we can obtain 𝐾 ℎ relation paths. The
score of the relation path 𝑝𝑡,𝑖 is the product of the probabilities of
all relations in the path:
𝑆𝑐𝑜𝑟𝑒 (𝑝𝑡,𝑖 ) = 𝑆𝑐𝑜𝑟𝑒 (𝑟𝑖,1 |𝑟𝑖,2 |...|𝑟𝑖,𝑡 ) =𝑡Ö

𝑑𝑟′𝑖,𝑙 , 𝑖 = 1, 2, ..., 𝐾 ℎ (15)𝑙=1Figure 4: Our retrieve module consists of three steps: hop
prediction, relation path prediction, and triple sampling.
Relation Path Prediction. Given the question 𝑞 and the predicted hop number ℎ, we perform ℎ-step prediction, with each stepcorresponding to one hop relation. At step 𝑡, we predict the 𝑡-th
hop relation based on the predicted (𝑡 − 1) hop relation paths andquestion 𝑞 via PLMs as a classification task. Specifically, for eachpredicted relation path, we sample 𝐾 candidate relations for the
next step. In step 1, we encode the question 𝑞 into a vector representation 𝑞 𝑣 . This vector is then passed through a linear classification′ of 𝑅 relations in KG:
layer to compute the distribution 𝐷𝑟,1′
𝐷𝑟,1
= [𝑑𝑟′1 , 𝑑𝑟′2 , ..., 𝑑𝑟′𝑅 ] = 𝐿𝑖𝑛𝑒𝑎𝑟 (𝑞 𝑣 )(8)where 𝑑𝑟′𝑐 is the probability of relation 𝑟𝑐 given the question representation 𝑞 𝑣 :𝑑𝑟′𝑐 = 𝑃 (𝑟𝑐 |𝑞 𝑣 ), 𝑐 = 1, 2, ..., 𝑅(9)We select top 𝐾 relations with the highest probabilities as one hop
relation paths 𝑝 1 .
In the following step 𝑡 (𝑡 > 1), the relation path 𝑝𝑡 −1,𝑖 in the(𝑡 − 1) hop relation paths 𝑝𝑡 −1 can be represented as:
𝑝𝑡 −1,𝑖 = 𝑟𝑖,1 |𝑟𝑖,2 |...|𝑟𝑖,𝑡 −1,

𝑖 = 1, 2, ..., 𝐾 𝑡 −1

(10)

We concatenate the question 𝑞 and the relation path 𝑝𝑡 −1,𝑖 with “|”
as the input sequence 𝑄𝑡 :𝑄𝑡 = 𝑞|𝑟𝑖,1 |𝑟𝑖,2 |...|𝑟𝑖,𝑡 −1(11)

𝑄𝑡 is encoded into vector representation 𝑄𝑡,𝑣 via PLMs and passes
through a linear classification layer to calculate the relation distri′ across 𝑅 relations in KG:
bution 𝐷𝑟,𝑡
𝑄𝑡,𝑣 = 𝑃𝐿𝑀 (𝑄𝑡 )

(12)

′
𝐷𝑟,𝑡
= [𝑑𝑟′1 , 𝑑𝑟′2 , ..., 𝑑𝑟′𝑅 ] = 𝐿𝑖𝑛𝑒𝑎𝑟 (𝑄𝑡,𝑣 )(13)

We employ a training method similar to hop prediction. For question 𝑞 and the ground truth relation path, we concatenate question
𝑞 and (𝑡 − 1) hop relation path as the inputs 𝑄𝑡 −1 . We encode 𝑄𝑡 −1
into vector representation and feed it into linear classification layer′ . The ground truth
to obtain the 𝑡-th hop relation distribution 𝐷𝑟,𝑡
distribution 𝐷𝑟,𝑡 is represented as a one-hot vector with a probability of 1 for the true 𝑡-th hop relation and a probability of 0 for otherrelations. We use cross-entropy loss for parameter optimization.
Triple Sampling. We arrange the predicted relation paths indescending order of scores and sequentially sample reasoning paths
composed of triples from KG until the number of reasoning paths
reaches 𝑀. These reasoning paths are utilized as the relevant knowledge to augment LLMs on KGQA.
4.1.2 Rewrite: KG-to-Text. The core of our rewrite module is totransform structured triples into free-form text based on a KG-toText model. We start by training an open-source LLM based onquestion-related graph-text pairs. Given the graph 𝐺 and corresponding free-form text 𝑦. We verbalize the triples in graph 𝐺 into
triple-form text 𝑥 by concatenating the subject, relation, and object.
Then, we transform triple-form text 𝑥 to graph-to-text transformation prompt 𝑝1 via a template 𝑇 1:“Your task is to transform aknowledge graph to a sentence or multiple sentences. The knowledge
graph is: {triple-form text 𝑥}. The sentence is:”. We take prompt 𝑝1
and free-form text 𝑦 as the input and output respectively and adopt
the teacher forcing strategy in training. Formally, given the prompt
𝑝1, ground truth output sequence 𝑦 = [𝑦1, 𝑦2, ..., 𝑦𝑇 ] and model
vocabulary [𝑣 1, 𝑣 2, ..., 𝑣𝑉 ], the model predicts the probability distri′ of tokens at step 𝑡 based on prompt 𝑝1 and previous
bution 𝐷 𝑣,𝑡(𝑡 − 1) steps correct tokens 𝑦1, 𝑦2, ..., 𝑦𝑡 −1 :′𝐷 𝑣,𝑡
= [𝑑 𝑣′ 1 , 𝑑 𝑣′ 2 , ..., 𝑑 𝑣′𝑉 ](16)

where 𝑑 𝑣′𝑐 is the probability of 𝑣𝑐 given 𝑝1 and 𝑦1, 𝑦2, ..., 𝑦𝑡 −1 :𝑑 𝑣′𝑐 = 𝑃 (𝑣𝑐 |𝑝1, 𝑦1, 𝑦2, ..., 𝑦𝑡 −1 ),𝑐 = 1, 2, ..., 𝑉(17)

Ground truth distribution 𝐷 𝑣,𝑡 is a one-hot vector with a probabilityof 1 for the true next token 𝑦𝑡 and probabilities of 0 for other tokens:
𝐷 𝑣,𝑡 = [𝑑 𝑣1 , 𝑑 𝑣2 , ..., 𝑑 𝑣𝑉 ](1, 𝑣𝑐 = 𝑦𝑡
𝑑 𝑣𝑐 =
𝑐 = 1, 2, ..., 𝑉0, 𝑣𝑐 ≠ 𝑦𝑡

(18)
(19)

The cross entropy loss 𝐿𝐶𝐸 is used to update the parameters:′
𝐽𝑡 = −𝐷 𝑣,𝑡 log 𝐷 𝑣,𝑡
=−

𝑉
∑︁
𝑐=1

′
𝑑 𝑣,𝑐 log 𝑑 𝑣,𝑐(20)

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

IJCKG’23, December 08–09, 2023, Tokyo, Japan

4.2.1 Subgraph Extraction. For KGQA benchmarks that provide
relation paths or reasoning triples (e.g. MetaQA [47], PathQuestion[48]), we obtain subgraphs by querying KG based on annotations.
For benchmarks annotated with SPARQL (e.g. WebQuestionSP [45],ComplexWebQuestion [34]), we modify SPARQL query to retrieveintermediate entities and construct question-related subgraphs.
4.2.2 Text Generation. Given the question 𝑞, we first verbalize therelated subgraph 𝐺 into triple-form text 𝑥 by concatenating the
subject, relation, and object. Then, template 𝑇 1 is adopted to transform triple-form text 𝑥 to graph-to-text transformation prompt 𝑝1.
Finally, we input prompt 𝑝1 into ChatGPT to obtain the corresponding free-form text 𝑦.

Figure 5: Our designed KG-to-Text corpus generation method
has three steps: subgraph extraction, text generation, andquality evaluation.

5 EXPERIMENTS
5.1 Datasets

𝑇

𝐿𝐶𝐸 =

1 ∑︁
𝐽𝑡
𝑇 𝑡 =1(21)

While answering questions, each reasoning path is first linearized into triple-form text and then transformed into a prompt
via template 𝑇 1. The prompt is fed into the fine-tuned LLMs to obtain the corresponding textual description. These descriptions areconsolidated into a paragraph as the question-relevant knowledge
to enhance the performance of the LLMs.
4.1.3 Answer: Knowledge Text Enhanced Reasoning. To integrate
the generated knowledge 𝑦 with the question 𝑞, we devise a template
𝑇 2:“Below are the facts that might be relevant to answer the question:{free-form text 𝑦} Question: {question 𝑞} Answer:”. We map the freeform text 𝑦 and the question 𝑞 to KG-augmented prompt 𝑝2 using
template 𝑇 2. Then we input prompt 𝑝2 into the question-answeringmodel and collect output as the predicted answer 𝑎.

4.2

4.2.3 Quality Evaluation. Due to the absence of annotations, common evaluation metrics including BLEU, METEOR, and ROUGE
cannot be employed. Considering the purpose of the generated text
is to strengthen the performance of LLMs on KGQA, we assess thequality of free-form text 𝑦 based on the feedback from questionanswering models. We map the free-form text 𝑦 and question 𝑞 to
KG-augmented prompt 𝑝2 via template 𝑇 2. Then, we feed prompt
𝑝2 to the question-answering model and get the predicted answer
𝑎. In light of the fact that LLMs typically provide textual passages
as responses instead of single answer entities, we employ hit@1 asa metric to assess the correctness of answer 𝑎. To put it simply, if
answer 𝑎 contains at least one answer entity, the question is considered answered correctly. In this scenario, we collect the triple-form
text 𝑥 and free-form text 𝑦 as a graph-text pair.

Corpus Generation

Existing KGQA benchmarks do not provide graph-text pairs for thequestion-answering task. Therefore, we design a KGQA task-driven
corpus generation method. Considering ChatGPT’s powerful natural language understanding and generation capabilities, inspired by[5, 16, 23], we adopt ChatGPT as the corpus generator. As shownin Figure 5, this process contains three steps: subgraph extraction,text generation, and quality evaluation.

MetaQA [47] is a large-scale multi-hop KGQA benchmark in themovie domain. It provides a knowledge graph including 135k triples,
43k entities, and 9 relations. It contains more than 400k questions
divided into MetaQA 1-hop, MetaQA 2-hop, and MetaQA 3-hop
based on the hop number of the question. Each question is annotated with the head entity, answers, and entity categories involvedin the reasoning path. In this experiment, we choose MetaQA 2-hop
as our benchmark and the“vanilla” version of the questions, totaling
148,724 questions (118,980 train, 14,872 dev, 14,872 test). We collectthe gold relation paths based on the provided entity categories since
there is only one type of relation between two categories of entities.
WebQuestionsSP (WebQSP) [45] is a smaller scale KGQA
benchmark with a larger scale KG. It provides SPARQL queries
for WebQuestions [2] and filters out questions that are not answerable. The remaining 4,737 questions (3,098 train, 1,639 test) areeither 1-hop or 2-hop questions with corresponding topic entities,
inferential chains, and SPARQL queries. Following [31], we prune
the KG to contain only relations mentioned in the questions andthe triples within 2 hops of mentioned entities. The pruned KG
includes 1.8 million entities, 627 relations, and 5.7 million triples.
WebQuestions (WebQ) [2] collects questions from web pages
via Google Suggest API. Following [33], we use a subset of thisbenchmark. Our method requires annotations for relation paths
or SPARQL queries, which are not provided. Therefore, we choose
the questions that are provided with SPARQL queries in WebQSPIJCKG’23, December 08–09, 2023, Tokyo, Japan

Wu and Hu, et al.

Table 1: Experimental results of our proposed framework and baselines on WebQSP/WebQ, where we report hit@1(%). We
implement our KG-to-Text model based on Llama-2-chat (13B). The results of the baselines are copied from the original paper.
The best score is emphasized in bold.
MethodKAPINGSen et al.
Ours

T5 (0.8𝐵)
34.70
69.41

T5 (3𝐵)
25.41
72.11

WebQSP
T5 (11𝐵) T0 (3𝐵)
24.91
52.28
79.36
74.02

T0 (11𝐵)
62.85
68.55

for training and testing. This results in 4,737 questions (3,098 train,
1,639 test). We use the same KG as WebQSP.
ZhejiangQA (ZJQA) is a Chinese KGQA dataset of 20,491 questions provided by Zhejiang Lab. We divide these questions intotrainset, devset, and testset (14,999 train, 2,147 dev, 3,345 test). The
questions are either 1-hop or 2-hop questions primarily in the
robotic domains. Each question is provided with a head entity, answers, and a gold relation path. It also provides a KG with morethan 11k triples, 9k entities, and 39 relations.

5.2

Large Language Models

Our proposed framework has two modules based on LLMs: KG-toText and question-answering. We use Llama-2 (7B, 13B), Flan-T5(3B) for KG-to-Text and Llama-2 (7B, 13B), T5 (0.8B, 3B, 11B), FlanT5 (80M, 3B, 11B), T0 (3B, 11B), ChatGPT for question-answering.
Llama-2 [38] is an updated version of Llama-1 [37], which istrained on a wide range of public online data sources. Among
different variants, we adopt Llama-2-chat (7B, 13B) as our KG-toText and question-answering model. For ZJQA, we employ the
Chinese version of this model, Chinese-Alpaca-2 (7B, 13B)2 .
T5 [25] is an encoder-decoder model pre-trained on multiple
tasks in a text-to-text format. Following [1], we use the LM-adapted
version3 of T5 as the question-answering model on WebQSP.
Flan-T5 [4] is an extension of T5, which is further instruction
tuned on a large-scale collection of automatically generated instructions from existing datasets. We use Flan-T5-XL (3B) as KG-to-Textmodel for MetaQA and Flan-T5-Small (80M), Flan-T5-XL (3B), FlanT5-XXL (11B) as question-answering models for WebQ. We do not
use this model for ZJQA since it does not support Chinese.
T0 [29] is fine-tuned from T5 over a variety of prompts to improve zero-shot generalization performance. We use T0 (3B, 11B) as
the question-answering model to compare our proposed frameworkwith previous work on WebQSP and WebQ.
ChatGPT4 is a large language model built on GPT-3.5 developed
by OpenAI. It is pre-trained on enormous corpus and human annotations and excels in understanding and generating human-like text.
Specifically, we use GPT-3.5 Turbo in this experiment. We cannot
fine-tune ChatGPT since it is not yet open source. Therefore, we
access it via API5 and use it as the question-answering model.

2 https://github.com/ymcui/Chinese-LLaMA-Alpaca-2
3 https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md
4 https://openai.com/blog/chatgpt
5 https://api.openai.com/

Flan-T5 (80𝑀 )
45.52
59.95

5.3

Flan-T5 (3𝐵)
55.58
68.92

WebQ
Flan-T5 (11𝐵)
59.79
73.71

T0 (3𝐵)
53.33
74.02

T0 (11𝐵)
55.64
68.55

Baselines

We compare the proposed framework with previous KG-augmented
LLM methods for KGQA on WebQSP and WebQ:
KAPING [1] verbalizes triples by concatenating subject, relation, and object. During the retrieval process, verbalized triples andquestions are embedded into vector space with an off-the-shelf sentence embedding model. The vector similarity between questionsand verbalized triples is calculated to retrieve triples as augmented
knowledge. The retrieved triples and questions are concatenated in
the form of the prompt and fed into the question-answering model
to obtain answers. We compare with this baseline on WebQSP.
Sen et al. [33] improves the retrieval method of KAPING [1].
This work utilizes Rigel [28] to predict the distribution over relations for each hop and retrieve triples based on the calculated
relation distribution and question entities. The retrieved triples arelinear verbalized and concatenated with the question via prompt
and input into the question-answering model to generate the answer. We compare with this baseline on WebQ.

5.4

Evaluation Metrics

Following the evaluation settings in previous work on generative
KGQA[30, 32, 46], we use hit@1, which measures whether thegenerated answer includes at least one answer entity. Both the predicted answers and the answer entities are converted to lowercase
to mitigate matching issues arising from letter-case differences.

5.5

Implementation Details

WebQSP/WebQ We parse the SPARQL query to extract the gold
relation path for each question. Bert-base-uncased is used as theclassification model for hop prediction and relation path prediction.
We modify the SPARQL query and obtain intermediate entities toconstruct the gold subgraph for each question. All the questions in
train split are used for corpus generation. Over 12k graph-text pairs
are generated and utilized for supervised fine-tuning. In addition,
we exclude 11 test samples without answers.
MetaQA We skip the hop prediction step and use the provided
ground truth hop number. Bert-base-uncased is used as the classification model for relation path prediction. We randomly sample 17kquestions from train split for corpus generation and generate morethan 13k graph-text pairs for KG-to-Text fine-tuning.
ZJQA We use bert-base-chinese for hop prediction and relation
path prediction. We randomly sample 14k questions from train split
for corpus generation and obtain over 13k KG-to-Text annotations.
For WebQSP and WebQ, we set K=5 (i.e. sample 5 relations as thenext possible relations for each predicted relation path) and M=5 (i.e.

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

IJCKG’23, December 08–09, 2023, Tokyo, Japan

Table 2: Experimental results of different knowledge representation formats on MetaQA and ZJQA, where we report hit@1(%).
The below section of the table is the results of our framework implemented by different KG-to-Text models. The number inside
the parentheses behind the LLMs denotes its parameter size. The best score for each question-answering model is emphasized
in bold.
Knowledge Format
No Knowledge
Triple Knowledge
MTL Knowledge
MVP Knowledge
Flan-T5 (3𝐵)
Llama-2-chat (7𝐵)
Llama-2-chat (13𝐵)

Llama-2-chat (7𝐵)
34.31
96.98
86.32
96.13
97.60
97.74
97.77

MetaQA
Llama-2-chat (13𝐵)
34.08
96.78
87.18
98.04
98.78
98.92
99.07

sample up to 5 reasoning paths for each question) in the subgraphretrieval process. For MetaQA and ZJQA, we set K=3 and M=5.
ChatGPT is used as the question-answering model in the corpus
generation for all benchmarks. Our framework is implemented
using Pytorch6 , Transformers7 and Peft libaries8 . We train KG-toText models on 4 NVIDIA Tesla V100 GPUs for 10 epochs adopting
LoRA [8] and run inference over KG-to-Text models and questionanswering models on 1 NVIDIA Tesla V100 GPU.

5.6

Main Results

Table 1 shows the overall results of our proposed framework andbaselines on WebQSP and WebQ. Our KG-to-Text model is implemented based on Llama-2-chat (13B). For the question-answeringmodel, we choose T5 (0.8B, 3B, 11B), T0 (3B, 11B) and Flan-T5 (80M,
3B, 11B), T0 (3B, 11B) as question-answering models on WebQSP
and WebQ respectively. Experimental results demonstrate that ourframework outperforms baselines by a large margin across various LLMs. Notably, it exhibits the most significant advantages on
T5. This suggests that T5, which is pre-trained solely on textual
data, may have limitations in comprehending structured data. This
proves that transforming triple-form text into free-form text canenable LLMs to better understand the provided factual knowledge
and enhance their capabilities on KGQA.

5.7

Ablation Study

We conduct a comparative analysis of different knowledge representation formats and implement our framework utilizing a rangeof LLMs on MetaQA and ZJQA datasets. This ablation study is
performed to investigate the impact of knowledge representation
formats and LLMs on KGQA.
Specifically, we use Llama-2-chat (7B, 13B), Flan-T5-XL (3B) as
the KG-to-Text model and Llama-2-chat (7B, 13B), ChatGPT as thequestion-answering model. We compare knowledge generated by
our framework with no knowledge, triple knowledge, and knowledge generated by the off-the-shelf KG-to-Text model:
No Knowledge Questions are directly fed into LLMs without
additional knowledge. We set this knowledge representation format
6 https://pytorch.org/
7 https://huggingface.co/docs/transformers/main/index
8 https://huggingface.co/docs/peft/index

ChatGPT
60.58
93.19
80.66
92.91
97.71
97.69
97.63

Llama-2-chat (7𝐵)
14.95
88.67
93.09
91.48

ZJQA
Llama-2-chat (13𝐵)
15.75
84.48
93.21
92.11

ChatGPT
15.25
92.02
92.91
92.11

to explore how much improvement KG-augmented methods canbring to LLMs on KGQA.
Triple Knowledge Triple knowledge is the most common strategy used in previous work. We first deduplicate the retrieved triples
to mitigate semantic redundancy and then simply verbalize each
triple by concatenating the subject, relation, and object.
MVP Knowledge To verify the effectiveness of the fine-tuning
process, we choose MVP [36] as the off-the-shelf KG-to-Text model.
MVP is a text-generation LLM that is first pre-trained on 77 datasets
over 11 diverse natural language generation (NLG) tasks in a supervised text-to-text format and then further pre-trained task-specificsoft prompts to enhance the model’s ability on specific tasks. We
use MVP-data-to-text, a variant of MVP which is pre-trained onlabeled data-to-text datasets to perform KG-to-Text in the zero-shot
scenarios. We do not use this knowledge representation format for
ZJQA since MVP does not support Chinese.
MTL Knowledge We select MTL-data-to-text as another off-theshelf KG-to-Text model. This model is a different variant of MVP
and is pre-trained on a mixture of labeled data-to-text datasets.
However, compared with MVP-data-to-text, it lacks training onother NLG tasks and task-specific soft prompts pre-training. We do
not use this model for ZJQA due to its lack of support for Chinese.
Table 2 shows the experimental results of different knowledge
representation formats based on various LLMs as the questionanswering models. Compared with other knowledge representation
formats, knowledge generated by our framework shows further
improvement across multiple LLMs on KGQA.

5.8

Analysis

Impact of Knowledge Representation Formats Table 2 demonstrates that different representations of the same retrieved triples
have a significant influence on KGQA. The above section of Table 2
shows the results of the baseline representation formats. Without incorporating external knowledge, the performance is the worst. This
indicates that LLMs are incapable of storing all factual knowledge
within their vast parameters. Other KG-augmented LLM methods
outperform this baseline by a large margin, demonstrating the effectiveness of incorporating KG knowledge relevant to questions.
Among them, MTL knowledge exhibits the smallest improvement.
This is because MTL is only pre-trained on multiple data-to-text
datasets and its natural language understanding and generationIJCKG’23, December 08–09, 2023, Tokyo, Japan

Wu and Hu, et al.

Table 3: The detailed comparison results over no knowledge and triple knowledge. The below section of the table is the resultsof our framework implemented by different KG-to-Text models. We adopt Llama-2-chat (13B) as the question-answering model.
The best score is emphasized in bold.
Knowledge Format
Triple Knowledge
MTL Knowledge
MVP Knowledge
Flan-T5 (3𝐵)
Llama-2-chat (7𝐵)
Llama-2-chat (13𝐵)

No Knowledge Baseline
Helpful ↑ Harmful ↓
9,428
103
8,201
303
9,583
70
9,672
49
9,690
46
9,703
38

capabilities are not strong enough. Therefore, the transformationfrom subgraphs to text loses semantic information and leads tolimited enhancement. MVP-generated knowledge and triple-form
knowledge result in comparable results. The reason is that MVP
is pre-trained on multiple NLG tasks and further fine-tuned ondata-to-text datasets, possessing stronger capabilities in text comprehension and generation compared to MTL. Nevertheless, it lacks
fine-tuning on domain-specific KG-to-Text corpus, leading to similar results as triplet knowledge. Triple-form knowledge is the mostcommon approach in previous work. However, experimental results
show that LLMs still struggle to extract semantics effectively from
triples. This suggests that LLMs prefer receiving textual knowledge,
as they have been pre-trained on massive corpora and structured
triples are just a part of it. The below section of Table 2 shows
results of our framework. Our framework employs multiple KG-toText models, surpassing all baselines on various question-answeringmodels. This not only demonstrates that our KG-to-Text method cangenerate answer-sensitive textual knowledge but also highlights
the robust applicability of our framework to mainstream LLMs.
Impact of LLMs The below section of Table 2 illustrates different performances of our proposed framework based on different
LLMs. Generally, Llama-2-chat is better at KG-to-Text comparedwith Flan-T5-XL. We speculate this disparity can be attributed todifferences in model parameters. The smaller size of Flan-T5-XL (3B)
compared to Llama-2-chat (7B, 13B) may be responsible for the inferior performance. The two-parameter versions of Llama perform
comparably. We believe this is due to the relative simplicity of KGto-Text compared to other NLG tasks and 7B parameters are enough.
For question-answering models, Llama-2-chat (13B) achieves thebest performance on MetaQA while ChatGPT demonstrates thepoorest performance. It is noteworthy that ChatGPT outperforms
Llama-2-chat (7B, 13B) when no knowledge is provided. This observation indicates that ChatGPT has a greater capacity to retain
knowledge due to its significantly larger parameter size. However,it does not leverage relevant knowledge as effectively as Llama-2chat. For ZJQA, we employ the Chinese version of Llama-2-chat,
Chinese-Alpaca-2. It is further trained based on Llama-2 using asubstantial Chinese corpus and instruction data. Despite having
significantly different model parameters, it achieves performance
on ZJQA that matches or even surpasses ChatGPT. This emphasizes
the significance of continued pre-training. Chinese-Alpaca-2 (13B)
excels in almost all knowledge representation formats but lags behind ChatGPT in triple-form knowledge. Apart from differences

Triple Knowledge Baseline
Helpful ↑Harmful ↓
334
1,761
443
255
453
155
456
137
459
119

in model parameters, we posit this is because ChatGPT has been
trained on a wider range of structured data, enabling it to extract
semantic information from structured data more effectively.
Helpfulness/Harmfulness of Generated Knowledge We further analyze the experimental results of MetaQA to investigate
the positive and negative impact of knowledge generated by different methods on question-answering models. We establish twobaselines: “no knowledge” and “triple knowledge”, and compare
other knowledge formats with these two baselines. We count the
number of questions where the baseline answers incorrectly butother knowledge formats answer correctly (i.e., helpful), as wellas the number of questions where the baseline answers correctly
but other knowledge formats answer incorrectly (i.e., harmful). We
choose Llama-2-chat (13B) as the question-answering model. The
detailed information is presented in Table 3.
The comparison with no knowledge baseline indicates knowledge representation format employed in our framework can better
assist LLMs on KGQA and has fewer adverse effects. Besides, resultson the triple knowledge baseline reveal that our KG-to-Text method
can generate the most informative textual statements for KGQA.
This highlights the necessity of fine-tuning KG-to-Text models.

6

CONCLUSION

In this paper, we propose Retrieve-Rewrite-Answer, a KG-to-Textenhanced LLMs framework for KGQA. Our framework adopts ananswer-sensitive KG-to-Text method to generate textual knowledge which is most informative for KGQA. To address the challenge of missing annotation data of KG-to-Text, we design a KG-toText corpus generation method based on the feedback of questionanswering models. Experimental results demonstrate that our framework outperforms previous KG-augmented methods by a large
margin. However, this framework requires fine-tuning KG-to-Textmodels on graph-to-text corpus and does not incorporate extra data
sources besides KG. In future work, we plan to explore integrating
additional knowledge resources and design an approach capable ofgenerating question-related knowledge in zero-shot scenarios.

7

ACKNOWLEDGMENTS

We thank the reviewers for their valuable comments. This work
is partially supported by the National Nature Science Foundationof China under No. U21A20488 and the Key Research Project ofZhejiang Lab (Grant No. G2021NB0AL03).

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

REFERENCES[1] Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-Augmented
Language Model Prompting for Zero-Shot Knowledge Graph Question Answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE). Association for Computational Linguistics, Toronto,
Canada, 78–106. https://doi.org/10.18653/v1/2023.nlrse-1.7[2] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic
Parsing on Freebase from Question-Answer Pairs. In EMNLP. ACL, 1533–1544.
[3] Deng Cai and Wai Lam. 2020. Graph Transformer for Graph-to-Sequence Learning. In AAAI. AAAI Press, 7464–7471.
[4] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William
Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
2022. Scaling Instruction-Finetuned Language Models. (2022). https://doi.org/
10.48550/arXiv.2210.11416 arXiv:2210.11416[5] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao
Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, et al. 2023. AugGPT: Leveraging ChatGPT for Text Data Augmentation. (2023). https://doi.org/10.48550/
arXiv.2302.13007 arXiv:2302.13007[6] Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely Connected
Graph Convolutional Networks for Graph-to-Sequence Learning. Trans. Assoc.
Comput. Linguistics 7 (2019), 297–312.
[7] Jiuzhou Han and Ehsan Shareghi. 2022. Self-supervised Graph Masking Pretraining for Graph-to-Text Generation. In EMNLP. Association for Computational
Linguistics, 4845–4853.
[8] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large
Language Models. In ICLR. OpenReview.net.
[9] Nan Hu, Yike Wu, Guilin Qi, Dehai Min, Jiaoyan Chen, Jeff Z Pan, and Zafar Ali.
2023. An empirical study of pre-trained language models in simple knowledge
graph question answering. World Wide Web (2023), 1–32.
[10] Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen,
Zirui Wu, and Yansong Feng. 2023. Lawyer LLaMA Technical Report. (2023).
https://doi.org/10.48550/arXiv.2305.15062 arXiv:2305.15062[11] Robert L. Logan IV, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer
Singh. 2019. Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware
Language Modeling. In ACL (1). Association for Computational Linguistics, 5962–
5971.
[12] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii,
Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in
Natural Language Generation. ACM Comput. Surv. 55, 12 (2023), 248:1–248:38.
[13] Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, andMinlie Huang. 2021. JointGT: Graph-Text Joint Representation Learning for Text
Generation from Knowledge Graphs. In ACL/IJCNLP (Findings) (Findings of ACL,
Vol. ACL/IJCNLP 2021). Association for Computational Linguistics, 2526–2538.
[14] Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-Augmented Dialogue Generation. In Proceedings of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers). Association for Computational
Linguistics, Dublin, Ireland, 8460–8478. https://doi.org/10.18653/v1/2022.acllong.579[15] Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh
Hajishirzi. 2019. Text Generation from Knowledge Graphs with Graph Transformers. In NAACL-HLT (1). Association for Computational Linguistics, 2284–2293.
[16] Md Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, andJimmy Huang. 2023. CQSumDP: A ChatGPT-Annotated Resource for QueryFocused Abstractive Summarization Based on Debatepedia. (2023). https://doi.org/10.48550/arXiv.2305.06147 arXiv:2305.06147[17] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman
Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART:
Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. Association for Computational
Linguistics, 7871–7880.
[18] Linhan Li, Huaping Zhang, Chunjin Li, Haowen You, and Wenyao Cui. 2023.
Evaluation on ChatGPT for Chinese Language Understanding. Data Intelligence(09 2023), 1–19. https://doi.org/10.1162/dint_a_00232[19] Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh
Hajishirzi, and Yejin Choi. 2022. Rainier: Reinforced Knowledge Introspector for
Commonsense Question Answering. In EMNLP. Association for Computational
Linguistics, 8938–8958.
[20] Dakuan Lu, Hengkui Wu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng,
Mengkun Han, Yingsi Xin, and Yanghua Xiao. 2023. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model,
Corpus and Benchmark. (2023). https://doi.org/10.48550/arXiv.2302.09432
arXiv:2302.09432[21] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating
Effectiveness of Parametric and Non-Parametric Memories. In ACL (1). Association for Computational Linguistics, 9802–9822.

IJCKG’23, December 08–09, 2023, Tokyo, Japan[22] Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep Graph Convolutional Encoders for Structured Data to Text Generation. In INLG. Association for
Computational Linguistics, 1–9.
[23] Lidiia Ostyakova, Kseniia PetukhovaO, Veronika Smilga, and Dilyara ZharikovaO.
2023. Linguistic Annotation Generation with ChatGPT: a Synthetic Dataset of
Speech Functions for Discourse Annotation of Casual Conversations. In Proceedings of the International Conference “Dialogue, Vol. 2023.
[24] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical
Report.
[25] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang,
Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring theLimits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach.
Learn. Res. 21 (2020), 140:1–140:67.
[26] Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych.
2021. Investigating Pretrained Language Models for Graph-to-Text Generation. In
Proceedings of the 3rd Workshop on Natural Language Processing for Conversational
AI. Association for Computational Linguistics, Online, 211–227. https://doi.org/
10.18653/v1/2021.nlp4convai-1.20[27] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate
Saenko. 2018. Object Hallucination in Image Captioning. In EMNLP. Association
for Computational Linguistics, 4035–4045.
[28] Amir Saffari, Armin Oliya, Priyanka Sen, and Tom Ayoola. 2021. End-to-End
Entity Resolution and Question Answering Using Differentiable Knowledge
Graphs. In EMNLP (1). Association for Computational Linguistics, 4193–4200.
[29] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika,
Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al.
2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In
ICLR. OpenReview.net.
[30] Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-toSequence Knowledge Graph Completion and Question Answering. In ACL (1).
Association for Computational Linguistics, 2814–2828.
[31] Apoorv Saxena, Aditay Tripathi, and Partha P. Talukdar. 2020. Improving Multihop Question Answering over Knowledge Graphs using Knowledge Base Embeddings. In ACL. Association for Computational Linguistics, 4498–4507.
[32] Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering. In COLING.
International Committee on Computational Linguistics, 1604–1619.
[33] Priyanka Sen, Sandeep Mavadia, and Amir Saffari. 2023. Knowledge Graphaugmented Language Models for Complex Question Answering. In Proceedingsof the 1st Workshop on Natural Language Reasoning and Structured Explanations(NLRSE). Association for Computational Linguistics, Toronto, Canada, 1–8. https://doi.org/10.18653/v1/2023.nlrse-1.1[34] Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base forAnswering Complex Questions. In NAACL-HLT. Association for Computational
Linguistics, 641–651.
[35] Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin
Qi. 2023. Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions. (2023). https://doi.org/10.48550/arXiv.2303.07992
arXiv:2303.07992[36] Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2023. MVP: Multi-task
Supervised Pre-training for Natural Language Generation. In ACL (Findings).
Association for Computational Linguistics, 8758–8794.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models.
(2023). https://doi.org/10.48550/arXiv.2302.13971 arXiv:2302.13971[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. (2023).
https://doi.org/10.48550/arXiv.2307.09288 arXiv:2307.09288[39] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin,
Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al.
2023. Towards Generalist Biomedical AI. (2023). https://doi.org/10.48550/arXiv.
2307.14334 arXiv:2307.14334[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All
you Need. In NIPS. 5998–6008.
[41] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, PietroLiò, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR (Poster).
OpenReview.net.
[42] Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, andTing Liu. 2023. HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge.
(2023). https://doi.org/10.48550/arXiv.2304.06975 arXiv:2304.06975[43] Peilu Wang, Hao Jiang, Jingfang Xu, and Qi Zhang. 2019. Knowledge Graph
Construction and Applications for Web Search and Beyond. Data Intelligence 1,
4 (11 2019), 333–349. https://doi.org/10.1162/dint_a_00019

IJCKG’23, December 08–09, 2023, Tokyo, Japan[44] Wenguang Wang, Yonglin Xu, Chunhui Du, Yunwen Chen, Yijie Wang, and Hui
Wen. 2021. Data Set and Evaluation of Automated Construction of Financial
Knowledge Graph. Data Intelligence 3, 3 (09 2021), 418–443. https://doi.org/10.
1162/dint_a_00108[45] Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina
Suh. 2016. The Value of Semantic Parse Labeling for Knowledge Base Question
Answering. In ACL (2). The Association for Computer Linguistics.
[46] Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016.
Neural Generative Question Answering. In IJCAI. IJCAI/AAAI Press, 2972–2978.

Wu and Hu, et al.

[47] Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song.
2018. Variational Reasoning for Question Answering With Knowledge Graph. In
AAAI. AAAI Press, 6069–6076.
[48] Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2018. An Interpretable Reasoning Network for Multi-Relation Question Answering. In COLING. Association
for Computational Linguistics, 2010–2022.
[49] Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou.
2019. Modeling Graph Structure in Transformer for Better AMR-to-Text Generation. In EMNLP/IJCNLP (1). Association for Computational Linguistics, 5458–
5467.