StructGPT: A General Framework for Large Language Model to Reason over Structured Data
Authors: Jinhao Jiang
Kun Zhou
zican Dong
KeMing Ye
Xin Zhao
Ji-Rong Wen
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Structured Data

Abstract: In this paper, we aim to improve the reasoning ability of large language models (LLMs) over structured data in a unified way.  Inspired by the studies on tool augmentation for LLMs, we develop an Iterative Reading-then-Reasoning (IRR) framework to solve question answering tasks based on structured data, called StructGPT. In this framework, we construct the specialized interfaces to collect relevant evidence from structured data (i.e., reading), and let LLMs concentrate on the reasoning task based on the collected information  (i.e., reasoning). Specially, we propose an invoking-linearization-generation procedure to support LLMs in reasoning on the structured data with the help of the interfaces. By iterating this procedure with provided interfaces, our approach can gradually approach the target answers to a given query. Experiments conducted on three types of structured data show that StructGPT greatly improves the performance of LLMs, under the few-shot and zero-shot settings.

StructGPT: A General Framework for Large Language Model to Reason over Structured Data
Jinhao J Kun Zhou? Zican Dong', Keming Ye*, Wayne Xin Zhao and Ji-Rong Wen
GGaoling School of Artificial Intelligence, Renmin University of China.
School of Information, Renmin University of China.
BBeijing Key Laboratory of Big Data Management and Analysis Methods.
UUiiversity of Electronic Science and Technology of China.
www.odrrruueud..com batmanf batmanfly@gmal yyemaii com
Abstract
issues is augment LLMs with external knowl- edge resources, so as to amend the incorrect gen- crations. Among these resources, structured data (e.g., knowledge graphs and databases), has been widely used as the carrier of the required knowl- edge for LLMss Unlike plain text, structured data is organized in standardized format, conforming to some logical data model. For example, knowledge graphs (KGs) are often organized as fact triples that state the relations between head entities and tail entities, and data tables are organized in the form of column-indexed records by rows. How- ever, as structured data has special data formats or schemas that LLMs have not seen during pre- training, they may be not fully grasped or under- stood by LLMs (Wei et al., 2021). A straightfor- ward way to solve this problem is to linearize the structured data into sentence that LLMs can well understand. While, the amount of structured data is often vast, making it infeasible to include all the data records in the input prompt.
In this paper, aim improve the reason- ing ability of large language models (LLMs) over structured data in a unified way. In- spired by the studies tool augmentation for LLMs, we develop an Iterative Reading-ther Reasoning (IRR) framework solve question answering tasks based on structured data, called StructGPP. In this framework, we construct the specialized interfaces to collect relevant ev- idence from structured data (i.e., reading), and let LLMs concentrate on the reasoning task based on the collected information (i.e., rea- soning). Specially, we propose an invoking- linearization generation procedure to support LLMS in reasoning the structured data with the help of the interfaces. By iterating this procedure with provided interfaces, ap- proach can gradually approach the target an- swers to given query. Experiments conducted three types of structured data show that StructGPT greatly improves the performance of LLMs, under the few-shot and zero-shot set- tings. Our codes and data are publicly available at https://gi thub. cow/RUCAIBox/ StructOPT.
Regarding the above challenges, we are inspired by the tool manipulation strategy for augmenting the abilities of LLMs (Schick et al., 2023; Nakano et al., 2021). Our basic idea is to incorporate spe- cialized interfaces (e.g., extracting columns from tables) to manipulate the structured data records. With these interfaces, we can effectively reduce the search space of the data records, and more accu- rately identify the required evidence to fulfill spe- cific tasks. In this way, LLMs can concentrate on reasoning based on the evidence obtained from the interfaces. To implement the interface-augmentte approach, there remain two key problems, namely how design suitable interfaces for specific tasks and how utilize them for reasoning by LLMs, which are the focus of this work.
1 Introduction
Recently, large language models (LLMs) (Brown et al., 2020; Zhao al., 2023) have made remark- able advancements in the NLP field. Existing work (Ouyang et al., 2022a; Zhang et al., 2022) has demonstrated that LLMs (e.g., ChatGPT or GPT- 4 (OpenAI, 2023)) have strong zero-shot capability to solve broad range of tasks using specially de- signed prompts, without task- fine-tuning. Despite the successes, recent work has also re- vealed that L.Ms may generate unfaithful informa- tion in conflict with the factual knowledge (Li et al., 2023), and also fall short of mastering domain- specific or real-time knowledge (Schick et al., 2023; Peng et al., 2023). A direct solution to the above
To design suitable interfaces. we regard multi- ple types of structured data as black-box systems, and design the interfaces to provide accurate, effi- cient data access and filtering for LLMs. For each
Equul contributions. Corresponding author.

---

interface, its implementation is dependent on the characteristic of the structured data, while its func- tionality general to all LLMs, with just a few arguments for specifying the data requirements. Based on these interfaces, we propose an Itera- tive Reading- LLMs to utilize the interfaces to solve the tasks based on structured data, namely StructGPT. This framework considers two major functions to ful- fill different tasks, namely collecting relevant evi- dence (reading) and inferring the answer or plan- ning subsequent steps (reasoning). Specifically, we propose an invoking- -inearizaaiion simearizationngeneatttoon pro- cedure to support LLMs in reading and reasoning on the structured data with the help of the external interfaces. By iterating this procedure with pro- vided interfaces, we gradually approach the target answer to a given question.
cently, with the success of pre-trained language models (PLMs) (e.g. T5 (Raffel et al., 2020), BART (Lewis et al., 2020)), several methods (Raf- fel et al., 2020; Khashabi et al., 2020) have adopted PL.ss as the general encoder or solver for different structured data and tasks. Among them, Unified- SKG (Xie et al., 2022) unifies a number of rea- soning tasks over structured data into a text-to-text format, which concatenates the question and the linearized structured data as input, and then fine- tunes T5 to learn to generate the answer. However, UnifiedSKG also requires to tunc the model param- eters, and is unable to handle large-scale structured data under the limitation of the maximum input length. Instead, our method can utilize the LLM to perform reasoning on structured data without train- ing, and also leverage the interfaces of structured data to better manipulate vast structured data.
(IRR) framework for
To our knowledge, this is the first work that ex- plores how to support LLMs in reasoning on multi- ple types of structured data (including tables, KGs, and DBs) in a unified paradigm. To evaluate the ef- fectiveness of our approach, we conduct extensive experiments on a wide range of tasks (e.g., KG- based question answering (KGQA), Table-based question answering (TableQA), and DB-based Text- to-SQL). Experimental results datasets demon- strate that our approach can effectively enhance the reasoning performance of LLMs on structured data in zero-shot and few-shot settings, even compara- ble with competitive full-data supervised-uuning methods. For example, in KGQA, TableQA, and Text-to-SQL tasks, our approach yields an increase of 11.4% of Hits@1 on WebQSP, 4.2% of accu- racy in TabFact, and 4.7% of execution accuracy in Spider respectively, compared to directly using ChatGPT in the zero-shot setting.
LLMs for Structured Data. Benefitting from the strong few-shot and zero-shot capability, re- cent studies have leveraged LLMs to perform rea- soning over structured data (Chen al., 2023; Li et al., 2023; Cheng et al., 2022; Rajkumar et al., 2022). Existing work can be roughly divided into two types. The first type of method lincarizes the structured data into a sentence (e.g., table rows), and feeds it into the LLMs to generate the answer according to in-context exemplars (Cheng et al., 2022; Chen, 2023). For complex questions or struc- tured data, they first decompose it into multiple simple and short ones and then perform lineariza- tion and generation (Ye al., 2023). Another type of method leverages LLMs to evaluate the plausi- bility of the solution plan based on the knowledge base (Gu et al., 2023), or first generate a solution draft with in-context exemplars and then revise the draft grounding on the knowledge base (Li et al., 2023). However, most of them only focus a specific type of structured data, and are lack of gen- erality across various data and tasks. In StructGPT. we provide unified paradigm that is general various structured data and downstream tasks.
2 Related Work
Reasoning over Structured Data. Structured data (e.g., knowledge graphs, tables, and databases) is an important knowledge carrier for a variety of QA and reasoning tasks. Early work focuses on designing specific model architectures tailored for each type of structured data, such as graph neu- ral networks (Sun et al., 2018), table Transform- ers (Herzig et al., 2020), and tree-structured de- coder (Wang al., 2020). While achieving re- markable performance, these approaches lack gen- crality for various types of structured data and are hard to be transferred across different tasks. Re-
Preliminary
In this section, we introduce the definition of struc- tured data, which mainly consists of three com- monly used types. Then we present the unified problem statement.
Structured Data. Structured data (e.g., data ta- bles and knowledge graphs) refers to the data that

---

is in a standardized format, conforming to some logical data model (Xie et al., 2022: Chen et al., 2009). Due to the formal structure, it easy and efficient to access and query structured data us- ing formal languages (e.g., SQL. and SPARQL. for databases) or specific algorithms (e.g., triples search for knowledge graphs). In this work, we mainly focus three types of structured data, namely knowledge graphs (KG), data tables (Ta- ble), and databases (DB), since they play an impor- tant role as the knowledge source in helping solve complex reasoning tasks, described as follows.
4 Approach
4.1 Overview
In this work, we assume that LLMs have to rely the evidence contained in the structured data to solve the three tasks described in Section 3, An intuitive idea is to conduct a two-stage frame- work as prior studies on retrieval-uugmented ap- proaches (Izacard et al., 2022; Oguz et al., 2022), in which LLMs are employed to first collect suf- ficient evidence relating to the question and then figure out the answer by the LLMs. However, such an approach is not directly applicable to structured data, Although LLMs are capable of solving di- verse tasks in natural language, they have limited capacities in accurately representing and under- standing structured data, especially for their con- tained domain-specific knowledge (Moiseev et al., 2022; Emelin et al., 2022).
Knowledge Graph. A knowledge graph (KG) consists of a number of triples to store the fac- tual knowledge, denoted as 9 = [(c, r, L') c' E E,r RJ. where & and R denote the set of enti- ties and relations, respectively. A triple (e,r,e) represents the fact that there a relation r between the head entity e and the tail entity e'.
Data Table. A data table T (table in short) con- tains multiple columns {- and rows { where each row l denotes a data record formatted by the attributes indexed by columns { and oij denotes the content in the cell corresponding to the position at column and row j.
To address this difficulty, our solution is inspired by the use of specialized tools in solving complex tasks for LLMs (Nakano et al., 2021; Gao et al., 2022b; Schick al., 2023). We noted that struc- tured data is well organized and supports easy ac- cess via formal language or queries (called inter- face for generality). The basic idea of our approach is to disentangle the two processes of reading and reasoning for LLMs: we utilize the interface of structure data to implement accurate, efficient data access and filtering (obtaining the relevant evi- dence), and further utilize the reasoning ability of LLMs to figure out the final plan or result for the question (fulfilling the task). In this way, LLMs can concentrate on the reasoning process in answering the question, without considering the specialized approach to reading the structure data.
Database. A database (DB) typically consists of N data tables, denoted as D = (T1, T2,,... Tx}. Besides the column names, the foreign keys across all tables are also available to link the data from two tables, denoted as [(c] ). where of and cj
(h) denote the i-th and j-th columns in the k-th and h-th tables, respectively.
Problem Statement. This work mainly focuses on using LLMs to solve complex reasoning tasks based on structured data. Formally, it can be de- scribed as a question answering task: given a nat- ural language question q and an accessible struc- tured data S (e.g., a knowledge graph, table, or database), the LLM needs to extract useful evi- dence from S and then generates the expected re- sult to answer the question g based on the extracted evidence. According to the task requirement, the generated result can be either free-form answers in natural language or structured expressions (e.g., SQL statements) to be executed for obtaining the answer from S. Since we consider three types of structured data (Section 4), our tasks can be instan- tiated as follows:
Specially, in our framework, we encapsulate the structure data as a black-box system, and pro- vide specific interfaces for LLMs to access the contained data. Further, we propose an invoking- linearizationngeneration procedure that enables LLMs to read and extract useful evidence from structured data via the corresponding interface. By iterating the above procedure with provided inter- faces, we can gradually obtain the answers by lever- aging the superior reasoning abilities of LLMs.
4.2 Interfaces for Structured Data
Due to the standardized data formats, structured data is often equipped with efficient data manage- ment ways, egg., SQL for the database. In our approach, we aim to provide LLMs with special-
KG based question answering (KGQA) Table based question answering (TableQA)
DB based semantic parsing (Text-to-SQL)

---

Question
Final Answer Executable SQL
KGQA
Who the wile of jeff Probst?
Lisa Ann Russel
TableQA
what the last stadium listod on this chart?
DW
Text-to-SQL
How many visttors below age there?
SELECT count(") FROM visitor HERE age
Extract, Neighbor Ralationst"Aeff Probst")
"spouse, is a" "Team Capacity"
spouse
Extract Column, Nama("Teum")
LLM
Stadium
Extract, Table&Column, Name("mus
"Visitor x ID I...........
Irvoking
Lineniiztionn
Ganeration
visitor
Interfaces
Extract Columa, Name (1)
Estract, Anighbor Relations (e) ENR, 500 ontedd (ssunie a)
cntor: (iean Stdeeterrrrrrrrteeee
TablseColen
200 inpel ETN TrEe
Extract, Columns (e))
XOX ouppet (rod, vidkoc SESNEEs
500 lepel ('Sddans") (iPooderr.. JDW])
Zatract Triptes CD
Extrect Talles Inforration (T7)
woeltpal Pottd Worruppe (ded Is Busslll
Extract SubTable RJ. 03
200-trptt M
DS (Salial
5ou cuppet [Ioote,, (ID.... Aaci KULLI
(0Porsken). JDWI
Structured Data
KG
prodic
Tisa Am
Table
DB
D=sistorll
The ceff
Protot
Sporse
Tham
Staiuum
Capacity
vistt
Mas
Nux Ticke
stala:
Total Sook
Probst Show
comiteee
Name
Level of men
Age
TEE
1iss
Desi
Survivor
Wigan Wantoors
DW
25,138
Visc ID
Nere
New Sort
Open Yer
Figure I: The overview of the proposed iterative reading- maading-then-reasoniig reasoning approach. We design specialized interfaces invoking-linearization linearization generratinnnnn procedure to utilize LLMs for perform-
for reading structured data, and iterate the invoking ing reasoning the interfaces, until deriving the final answer executable SQL..
ized interfaces, helping LLMs to read and utilize the structured data. Next, we present the specially designed interfaces for KG, table, and DB.
LLMs to extract its sub-table containing relevant columns and rows. Thus, we define three functions:
Extract Coluum Naame (T): extracts all the
column names of a table T.
Interfaces for Knowledge Graph. When per- forming complex reasoning on a KG, existing work (Sun et al., 2018) typically starts from a cer- tain entity (about the question topic), and jumps along with the relations until reaching the answer. In this process, LLMs should be aware of the neigh- boring relations of the current entity, and the neigh- boring triples with certain relations to the current entity. Based
Extract Extract_Colummn (T, {c]): extracts the con-
tents of columns from a table T by indices {c}.
Setrratt_SubTade (T, {c), {i}) extracts the sub-table specified by the column indices {c} and row indices (j) from table T.
Interfaces for Database. fied setting when querying the database. LLMs should be aware of all the contained tables and columns (by name) for relevant tables selection. and also acquire the detailed columns and for- cign keys from the selected lables to search for the answer. Thus, we devise two functions as follows:
Considering a simpli-
LLMs select the relevant relations and triples from them find the answer. For this purpose, we devise two functions for as- sisting LLMs to accomplish the above operations. Relatioos
Relations (e): extracts all the neighboring relations of the entity e.
Extract &able& Coltun_aauaeeeee Naaiemnaammeeaa (D): extracts the names of all the tables and their contained columns from the database.
Extract_ Triples (c, (r)): extracts all the triples with the relation in {r} and head entity
Interfaces for Table. Given a data table, LLMS need to know its contained column names, and can access the content by row or column, enabling
SatractTaable fables_Informatinn ((T)): ): extracts the table names, column names, and foreign keys from a set of tables (T}.

---

4.3 Reading and Reasoning with Interfaces Based on the above interfaces, we propose a gen- eral catannglinnaaizatinneeeee -ennrratoonnnneai procedure that can be iterated in multiple turns for utilizing LLMs to perform reading and reasoning on struc- tured data. For each iteration, based on the cur- rently collected data, we first invoke interface to extract relevant evidence from structure data, then linearize into a textual prompt, and finally feed the prompt into the LLM for generation (selecting useful data or predicting the answer).
(i.e., [X]) from linearized extracted information
(i.e, IYD, according the question (i.e., IQD.
The second type of prompt follows the pattern:
"Based on [Y), please generate [Z] for the question
/Ql". It aims to predict the targeted results (i.e., [Z]) for the given question (i.e., (Q/) based on the linearized extracted information (i.e., [YJ). Note that the targeted results can be either the answer string or exccutable formal language (e.g., SQL) that can lead to the final answer.
By iterating the above invoking- invoking-linearizatie -inearizaiinn generation procedure on designed interfaces, LLMs can progressively capture more useful evidence for deriving the final answer.
Invoking an Interface. In this step, we aim to invoke an interface for extracting the relevant infor- mation from the structured data. According to the designed interfaces in Section 4.2, we construct the input based on the currently available data (e.g., en- tity and table), and then invoke the interface to ob- tain more detailed relevant information (e.g., neigh- boring relations and column names), which will be fed into LLMs for collecting useful information or generating the answer.
4.4 Instantiated Downstream Tasks
In the following, we describe the instances of the above general workflow for the tasks described in Section 3, since they deal with very different structure data and vary in the task settings.
KG-based Question Answering (KGQA). This task aims find the answer entities for the question based on the KG. Following existing work (Sun et al., 2018), we denote the mentioned entity the given question q as the topic entity CT, and assume it has been linked to some specific en- tity on the KG through existing linking tools (e.g., Google Knowledge Graph Search API) or mod- cls (e.g., ELQ (Li et al., 2020)). Starting from et, we perform the wanaking-iinariizttinngeee procedure two times using the two interfaces in KG sequentially. First, we invoke the interface Extract_Neighhot Www.eeee__Reeeeee Relation(ET) to extract the can- didate one-hop relations, linearize them to com- pose the input prompt, and then leverage the LLM to select the useful relations (r) according to the question. Then, based on {r). we invoke the Ex- tract_ Pract_Triples (ey, {r)) interface to collect the rel- evant triples for the head entity er and relation in {r], then linearize this information, and finally em- ploy the LLM to select the most relevant triples, whose tail entities will be considered as the final answer. Besides, we can also consider the multi- hop KGQA task (Lan et al., 2021), where after selecting the triples of the current hop, the LLM should assess whether the current information is sufficient to answer the question. Then, LLMs will make according actions based on the assessment, ie., stopping the iterations for producing the an- swer or continuing the iterations on next-hop tail entities from selected triples.
Information Linearization. Given the extracted information, we convert into a textual sentence that can be understood by LLMs. For the infor- mation from KG (i.e., relations and triples), we concatenate them into long sentence marked by specific separation and boundary symbols. For ta- ble and database, we leverage the same way to lin- earize the extracted table names or column names, While for contents in columns and rows, we follow existing work (Pasupat and Liang, 2015) that first converts them into triples, where head entities are the row indices, relations are column names, and tail entities are the content in the cell, e.g., "(row 1, year, 1896)" and "(row I, city, Athens)". Then, for each row, we extract the row indices in the front and omit it in the triples, compose simplified sentence, e.g., "row 1: (year, 1896), (city, Athens)". For multiple rows, we concatenate them into a long sentence via a special separation symbol.
LLM for Generation. After linearization, we design two types of input prompts for LLMs to fulfill different purposes!:
The first type of prompts mostly adopts the fol- lowing pattern: "Here are [Y]. Which [X] are most relevant to answer the question (Ql". It aims to elicit the ability of LLMs to select useful evidence
Note that our used prompts are not always consistent with the two examples, as we have rewritten them to better adapt to the specific datasets and extracted information.

---

Table-based Question Answering (TableQA). For TableQA, we typically need answer the question according to the content in the given ta- ble. We also perform the above procedure by us- ing the three interfaces in turn. Concretely, first, we invoke Extract_ Saattet__oomm___e Baalumn_Nameeaa (T) to extract all column names of a table, linearize them, and leverage LLMs to select the relevant ones {c} according to the question. Then, we invoke Ex- tract_ pract_Columns (T. {c}) extract the contents of all relevant columns, and select the useful row in- dices {j} by LLMs. Subsequently, we further in- voke Extract_Sutie SubTable (T, {c}. {i}) to generate the sub-table for the question. Based on the lin- earized sub-table, the LLM finally generates the answer to the question.
Questions (WTQ) (Pasupat and Liang, 2015), and TabFact (Chen al., 2020). The first two are typ- ical table-based question answering datasets, and the third one is a multiple-choice dataset that con- centrates on table fact verification. WikiSQL re- quires filtering and aggregating information over the table content, and the WTQ demands more advanced reasoning capabillities (e.g., sorting). Tab- Fact needs to judge whether the provided statement agrees with the facts stored in table.
For DB based semantic parsing (Text-to-SQL) SQL), adopt three public datasets, i.e., Spider (Yu al., 2018), Spider-SYN (Gan al., 2021), and Spider- Realistic (Deng et al., 2021). Spider is a typical Sext-tt-SQL dataset covering 20 databases with a set of 1034 evaluation samples. Spider-SYN and Spider-Realistie are two more challenging datasets derived from Spider. Concretely, Spider-SYN man- ually substitutes the synonyms in natural language questions, while Spider- Reallste removes the ques- tions in the evaluation that explicitly mention the required columns' names.
DB-based Semantic Parsing (Text-to-SQL..
This task focuses on generating a SQL query that can be executed to obtain the required information from a database. To achieve this goal, first, we invoke Cetract_Tableds
Column_Name (D) to ob- tain all the table names and their column names in the DB, linearize them, and utilize the LLM select the relevant table names. Then, we in- voke Extract Taclee___ales__e Informatiooonnonnnn ({T)) to obtain all the relevant information (i.e., column names and foreign keys) from these tables. Similarly, by lin- earizing this information and composing the input prompt, the LLM can generate an executable SQL for the given question.
5.2 Evaluation Metrics
For KGQA, we employ Hits@1 1 which assesses whether the top-1 predicted answer correct. In our approach, we focus on generating the most con- fident answer and then checking if the prediction hits any target. As LLMs may generate multiple answers, we also conducted a manual double-check finally (Tan et al., 2023), to judge if wrong answers are included. For TableQA, we adopt two eval- uation metrics, namely demotation accuracy and accuracy. In WTQ and WikiSQL, denotation accu- racy is employed to evaluate whether the predicted answer is the same as the gold answer based on sev-l equivalence. In TabFact, we adopt ac- curacy to assess the correctness of the prediction. For Text-to-SQLE SSQL, we adopt the execution accuracy (EX) to assess whether the execution results of the predicted SQL and the gold SQL are the same.
5 Experiment
We conduct experiments on three complex rea- soning tasks over structured data, iee. KGQA, TableQA, and DB based text-to-SQL.
5.1 Datasets
For KG based QA (KGQA), adopt two benchmark datasets, i.e., WebQuestionsSP (We- bQSP) (Yih et al., 2016) and MetaQA (Zhang et al., 2018) for evaluation. The answer entities in We- bQSP require up to 2-hop reasoning on the Free- base KG. In contrast, MetaQA contains questions in the movie domain, whose answer entities are up to 3 hops away from the topic entities on movie KG (based on OMDb). According to the num- ber of hops, it is split into three sub-datasets i.e., Ieop MetaQA-2hop, and MetaQA-3hop. For Table based QA (TableQA), we adopt Wik-
5.3 Baselines
We compare our method with competitive full- data supervised-tuning baselines tailored to these tasks. Specifically, our method is general iterative reading- monding-then-reaseee thaa-nnnn (IRR) framework that can be used for different L.LMs. And we test our IRR with two different LLMs, i.e., Davinci-003 (text- davinci-003 (Ouyang et al., 2022b)) and Chat-
three widely-used datasets, iSQL (WikiSQL) (Zhong et al., 2017), WikiTable-

---

GPT (ie., gpt-3,5-turbo 2), under zero-shot and few-shot settings 3. Considering the evolution of the closed large language model, e.g., ChatGPT, we have further conducted supplementary experiments on three datasets (i.e., WebQSP, WTQ, and Spider) using the latest August version of ChatGPT. The re- sults are presented in Appendix A. For KGQA, we select KV-Mem (Miller et al., 2016), GragtNet (Sun al., 201 8), EmbedKGQA (Saxena et al., 2020), NSM (He et al., 2021), and UniKGQA (Jiang et al., 2023). For TableQA, we select MAPO (Liang et al., 2018). TAPAS (Herzig et al., 2020; Eisensch- los et al., 2020), UnifiedSKG (T5-3B) (Xie et al., 2022), TAPEX (Liu et al., 2022), and DATER (Ye et al., 2023). For Text-to-SQI, we select RAT- SQL+BERT Large (Wang et al., 2020), TKK- Large (Gao et al., 2022a), T5-3B+PICARD (Raffel al., 2020), RASAT+PICARD (Qi et al., 2022), and BESDSQL-SB+NaISQL. (Li al., 2023).
Table I: Results of different methods for KGQA (Hits@1 in percent). We copy the results in the first block from He et al. (2021) and Jiang et al. (2023). The best results of each block are highlighted in bold.
MQA 1hop
MQA
MQA
Methods
WQSP
2hop
3hop
KV-Mem
46.7
96.2
82.7
48,9
GraftNet
66.4
97.0
94.8
77.7
FmbeddGQQ
66.6
97.5
98.8
94.8
NSM
68.7
97.1
99.9
98.9
UniKGQA
75.1
97.5
99.0
99.1
Davinci-003
48.3
52.1
25.3
42.5
IRR (ours)
71.9
94.4
59.5
70.2
IRR (ours, few-shot)
71.0
97.1
93.5
75.3
ChatGPT
61.2
61.9
31,0
43.2
IRR (ours)
72.6
94.2
93.9
80.2
IRR (ours, fow-shot)
69.6
97.1
97.3
87.0
LLMs can not solely rely on their own knowledge to answer difficult questions, and their augmen- tation with KGs is necessary. In contrast, when incorporating our proposed method to access KG, the performance of Davinci-OP and ChatGPT can be both substantially improved, indicating the ef- fectiveness of our proposed method for supporting LLMs reasoning over KG. By adding few in- context exemplars (i.e., 15 for WQSP and 32 for MQA) to LLMs, we can further improve the model performance. In our approach, we devise interfaces for KG to efficiently read the relevant information, and leverage LLMs to extract useful parts and per- form reasoning. We leverage the IRR procedure on devised interfaces sequentially, which pro- gressively capture more useful detailed evidence for finally obtaining the answer.
Additionally, we incorporate baselines that em- ploy Davinci-003 and ChatGPT directly for achiev- ing the aforementioned tasks in a zero-shot setting. To ensure a fair comparison, we utilize the same instruction prompt to evaluate them, ensuring that the only difference with our method is the usage of structured data. Specifically, in KGQA datasets, we follow existing work (Tan et al., 2023) that uti- lizes LLMs to answer the questions without using KG. In TableQA and Text-to-SQL, we feed the re- quired information of tables with questions into LLMs (Liu et al., 2023c,a), without special treat- ment for the overlength problem.
5.4 Results and Analysis
We show the results on KGQA, TableQA, and Text- to-SQL tasks and analyze them respectively.
Evaluation on TableQA. Table 2 shows the re- sults on three TableQA datasets. First, with the full table the prompt, ChatGPT can also achieve comparable performance on WTQ and TabFact full-data supervised-tune -uuninn methods, but performs not well on more difficult WikiSQL datasets. It also indicates that L.MSs have the capability of under- standing the knowledge within table data to some extent. Second, our proposed method can consis- tently improve the performance of two LLMs a lot in both threc datasets. At the same time, when adding 32 in-context exemplars the LLMs, they can obtain further performance improvements. It indicates the effectiveness of our proposed method in helping LLMs reasoning over Table. Our ap- proach provides a more effective way for LLMs to
Evaluation on KGQA. Table I shows the results on KGQA datasets. First, LLMs can achieve per- formance comparable to the supervised learning model (i.e., 61.2 of ChatGPT v.s. 66.4 of GraftNet and 48.3 of Davinci-003 v.s. 46.7 of KV-Mem) on the WebQSP dataset, in zero-shot setting with- out using KGs. It demonstrates that LLMs indeed grasp a certain amount of knowledge that can help them answer complex questions. However, on more difficult datasets that require multi-hop rea- soning (e.g., MetaQA-2hog and MetaQA-3hop). the two LLMs perform not well. It indicates that
hhups://llatormu www.plafwooo..oooiii..ommmo www.oopppeeooonnnceeeeeee com///ocs/moollldggt..5 IIn our experiment, we use the June version of Davinci-003 and ChatGPT.

---

Table Results of different methods for TableQA (de- notation accuracy for WTQ and WikiSQL, accuracy for TabFact). We copy the results of TAPAS on TabFact from Eisenschlos al. (2020), and others in the first block from their original papers. The best results of each block are highlighted in bold.
Table 3: Performance comparison different methods for Text-to-SQL. (execution accuracy in percent). We copy the results of RAT-SQL+BERT targe and TKK- Large from Deng et al. (2021) and Gao et (2022a). respectively. And we copy the results of the other three methods in the first block from Liu et al. (2023b). The best results of each block are highlighted in bold.
Methods
WTQ
WikiSQL
TabFact
Spider- SYN
Spider- Realistic
MAPO
43.8
72.6
Methods
Spider
TAPAS
48.8
83.6
81.0
UnifiedSKG (T5-3B)
49.3
86.0
83.7
RAT-SQL + BERT
72.3
62.1
TAPEX
57.5
89.5
84.2
TKK-Large
73.2
60.5
64.4
DATER
65.9
93.0
T5-3B PICARD
79.3
69.8
71.4
RASAT + PICARD
80.5
70.7
71.9
Davinci-003
34.8
49.1
80.7
RESDSQL NatQQI.
84.1
76.9
81.9
+IRR (ours)
39.2
51.8
76.5
IRR (ours, few-shot)
57.0
64.6
87.3
Davinci-003
68.8
60.1
63.2
ChatGPT
43.3
51.6
82.9
+ IRR (ours)
69.5
60.3
IRR (ours)
48.4
54.4
87.1
+ IRR (ours, few-shot)
72.7
63.2
70.7
IRR (ours, few-shot)
52.2
65.6
87.6
CharGPT
70.1
58.6
63.4
IRR (ours)
74.8
62.0
+ IRR (ours, few-shot)
77.8
64.0
72.0
iteratively access and utilize the relevant informa- tion from the table, which reduces the influence of irrelevant and redundant information.
then linearize them and compose the input prompt. Here, we utilize the instruction (i.e., provide only one relevant relation that's present in the candidate) to elicit the L.LM to generate the most relevant rela- tion, i.e., education. Based on the selected relation, we further invoke the Extract_Tripe Triples function to extract the triples with the relation to the topic en- tity. After linearization, another instruction (i.e., you just need to provide only one answer entity), is adopted for guiding the LLM to generate the final answer, i.e., Monroe County High School. Besides, we show the representative examples of TableQA and Text-to-SQL in Appendix B.
Evaluation on Text-to-SQL. Table 3 shows the results on DB-based datasets. First, with all the information from DB (table names, column names, and foreign keys) the prompt, the LLMs have the capability of directly generating a suitable SQL query of the question, performing well on three datasets. Whereas, the performance of LLMs is not better than competitive full-data supervised- tuning methods, showing the difficulty of this task. As our proposed method can extract relevant ta- bles and columns, it also alleviates the influence of irrelevant information for LLMs to generate the SQL query. Simultaneously, with the assistance of 32 in-context exemplars, L.MSs exhibit enhanced comprehension of the mapping between natural language questions and their corresponding SQL queries. The consistent performance improvements over the three datasets whenever in zero-shot or few-shot settings also indicate the effectiveness of our proposed method.
Error Analysis. To systemically analyze the short- comings of our approach, first select three datasets (i.e., WebQSP, WTQ, and Spider) with different types of structured data, and randomly sample 100 error cases from each dataset. Then, we manually examine these failures and classify them into five categories:
Selection Error: the relevant information has not been selected by the LLM.
Case Study, We show an example of KGQA in Figure 2, to help understand the working process of our method. Given the question, the interfaces of the structured data are sequentially invoked to iteratively extract more useful and detailed infor- mation. In each iteration, we first invoke the Ex- tract mact_Neighhor Reighhbo_Rellttoon function to extract the neighboring relations (e.g., birthplace, residence, and education) of the topic entity "Harper Lee".
Reasoning Error: given the extracted relevant information, the LLM fails to generate the ground- truth answer or SQL.
Generation Format Error: the generated an- swer is in an abnormal format that fails to be iden- tified by result parser.
Relations
Hallucination: the generated results are incon- sistent with the extracted information.
Other Errors: other uncategorizable errors.

---

Invoker Extract, Neighbor Relations (Harper
Q Question: what highschool did harper lee go to?
college
Oxford
candidate relations: education... ... birthplace.
lype
CVT 0
Institution
question ... Provide only relevant relation that's present in the candidates...
Return [education ... birthplacee
residmoo
edlacation
I Lineariz:: "education birthplace"
Horpe Leo
bisthplace
education
CVT
Instibation
Inooke
high school
Monroe
County
The relevaa ant relation: education.
Extract Triples (Haper Lee, [education])
G emeerate
The triples (Harper Lee, education,
Return
college
Dxiluni
CVT 0) Based these triples give
[[Torper education, CVT .)...
type
LVT
institution
the final answer entity
edducation
You just need to provide only one answer entity. If you think...
Linestize:
restdence
Tarme Loo
Bitthplace
"(Harpr Lee, education, CVT CVT_0):...
odicaation
Institution
CVT
high school
Monroe County
Answer: Monroe County High School
Generate
Figure 2: Case study of our method on WebQSP.
WebQSP (KGQA)
WikiSQL (TableQA)
Spider (Text-to-SQL)
13%
8%
4% 4%
Selection Emror
21%
3%
28%
8%
Reasoning Error
8%
2%
34%
Generating Format Error
Hallucination
74%
30%
63%
Other Errors
Figure 3: Proportions different error types three datasets over different types of structured data.
We show the statistics in Figure 3. First, for the three datasets, the distributions of occurring errors different. In WikiSQL, the frequencies of generation format, selection, and reasoning er- rors are relatively uniform. Whereas. in WebQSP, the selection error is the major error type (74%), since the KGQA task requires selecting the most relevant one from thousands of relations, which is not easy work. In Spider, reasoning error occurs more (62%), since the Text-to-SQL task requires LLMs to generate SQL that can be executed to obtain the answer, which is also hard for LLMs.
Conclusion
In this work, we proposed a general framework for improving the zero-shot reasoning ability of LLMs over structured data, namely StructGPT. In our ap- proach, we first constructed the specialized inter- faces that support accurate and efficient data ac- cess, and then proposed invoking-linearizalin generation procedure that leverages LLMs to read and perform reasoning based on the interface. By iterating the above procedure using the interfaces sequentially, LLMs can progressively capture more useful and detailed evidence and finally generate the answer. To verify the effectiveness of our approach, we implemented our approach on KG based QA, table based QA and DB based semantic parsing tasks. Experimental results on 8 datasets show that our approach can boost the zero-shot per- formance of LLMs by large margin, and achieve comparable performance as full-data supervised- tuning methods. We also provide detailed error analysis to point out the weakness of our approach, for enlighting other researchers in related areas.
According to the error distributions, it is promis- ing to refine the major error cases to specifically im- prove the performance on each dataset. Concretely, we can devise more high-quality prompts that elicit LLMS to carefully make decisions when selecting and reasoning KGQA and Text-to-SQU tasks, rc- spectively. Besides, we also consider adding more interfaces and iteration turns for decomposing the hard probelm into multiple simple ones, to simplify the complex reasoning task for better performance. We will try the above solutions in our future work.

---

7 Limitations
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020.
Although StructGPT demonstrates remarkable per- formance across tasks over structured data, there are some limitations of our method. First, the two LLMs used in our model, í.e., ChatGPT and Davinci-003, have a strong capability of following instructions. Hence, more experiments are required evaluate our method with in-context learning on other LLMs that perform poorly at instruction following. Similarly, we only evaluate question answering tasks based on structured data. Future work should include wider evaluation scenarios to evaluate the universality of our method, e.g., data-to-text and formal- laammall-nngug-tt-ttttt (Xic et al., 2022). Finally, since it is difficult to control the an- swer format during the generation process of LLMs in different datasets, there are several format errors in generated texts as shown in Section 5. Therefore, the performance of our method can be further im- proved by meticulously designing the prompt and answer parsing for different datasets.
Yi Chen, Wei Wang. Ziyang Liu, and Xuemin Lin. 2009, Keyword search on structured and aani-structured data. In Proceedings of the ACM SIGMOD Interna- tional Conference on Management of Data, SIGMOD 2009, Providence, Rhode Island, USA, June 29 July 2. 2009, pages 1005-1010. ACM.
Zhikai Chen, Haitao Mao. Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. 2023. Explor- ing the potential of large language models (Ilms) in learning on graphs. CoRR, abs/2307.053
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2022. Binding language models in symbolic languages. CoRR.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure- Structure-rouunded pretraining for text-to-sql. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1337-1350.
Acknowledgments
This work was partially supported by National Nat- ural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation un- der Grant No. 4222027 and L233008. And this work is also partially supported by the Outstanding Innovative Talents Cultivation Funded Programs 2022 of Renmin University of China. Xin Zhao is the corresponding author.
Julian Martin Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with intermediate pre-training. In Findings of the Associ- ation for Computational Linguistics; EMNLP 2020, Online Event, 16-20 November 2020, pages 281-296.
References
Denis Emelin, Daniele Bonadiman. Sawsan Alqahtani. Yi Zhang, and Saab Mansour. 2022. Injecting do- main knowledge in language models for task-oriented dialogue systems. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pages 11962-11999999 11974. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12. 2020, virtual.
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Peng- sheng Huang. 2021. Towards robustness of text-to- sql models against synonym substitution. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing, ACIAICCLLPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2505-2515.
Wenhu Chen. 2023. Large language models are few(1)- shot table reasoners. In Findings of the Associa- tion for Computational Linguistics: EACL 2023, Dubrovnik, Croutia, May 2-6, 2023, pages 1090- I 1100, Association for Computational Linguistics.
Chang Gao, Bowen Li, Wenxuan Zhang, Wai Lam, Bin- hua Li, Fei Huang. Luo Si, and Yongbin Li. 2022a. Towards generalizable and robust text-to-sql parsing. arXiv preprint arXiv:2210. 12674.

---

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022b. PAL: program-aided language models. CoRR.
for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguis- rics, ACL 2020, Online, July 5-10, 2020, pages 7871- 7880.
Yu Gu, Xiang Deng, and Yu Su. 2023. Don't 't gener- atc, discriminate: A proposal for grounding language models to real-world environments. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties /Volue 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. pages 4928 4949. Association for Computational Linguis- tics.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to to-end entity linking for questions. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6433-6441. Associa- tion for Computational Linguistics.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In WSDM '21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021, pages 553-561. ACM.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Decoupling the skeleton parsing and schema linking for text-to-sql. arXiv preprint arXiv:230..05655.
Chen Liang. Mohammad Norouzi, Jonathan Berant, Quoc V. Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and se- mantic parsing. In Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 10015-10027.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisen- schlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 4333.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023a. A comprehensive evaluation of chatgpt's 's zero- shot text-to-sql capability. CoRR, abs/2303. 133547.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. CoRR, ahs/2208.03299.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023b. A comprehensive evaluation of chatgpt's zero-shot text-to-sql capability. CoRR, abs/2303. 13547.
Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji- Ji-Rong Wen. 2023. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowl- edge graph. In The Eleventh International Confer- ence on Learning Representattons, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and
Lou. 2022. TAPEX: table pre-training via learning a neural SQL executor. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
Daniel Khashabi. Sewon Min. Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord. Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format bound- aries with single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, Findings of ACL
Qian Liu, Fan Zhou, Zhenghao Jiang. Longxu Dou, and Min Lin. 2023c. From zero to hero: Examining the power of symbolic tasks in instruction tuning. CoRR. abs/2304.07995.
Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly read- ing documents. In Proceedings of the 2016 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2016, Austin, Texas, USA, Novem- ber 1-4, 2016, pages 1400-1409.
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. A sur- vey on complex knowledge base question answering: Methods, challenges and solutions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 4483 4491. jjcai.org.
Fedor Moiseev, Zhe Dong. Enrique Alfonseca, and Mar- tin Jaggi. 2022. SKILL: structured knowledge infu- sion for large language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1581- 1588. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequeme pre-training

---

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser- assisted question-answering with human feedback. CoRR.
Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 2185)55485-5511
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabilities of large language models. CoRR.
Barlas Oguz. Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Sejr Schlichtkrull, Sonal Gupta. Yashar Mehdad, and Scott Yih. 2022. Unik-qa: Unified representations of structured and unstructured knowledge for open- domain question answering. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1535-1546. Association for Computational Linguis- tics.
Apoorv Saxena, Aditay Tripathi, and Partha Taluk- dar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embed- dings. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1498-4507.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. CoRR.
OpenAL. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Co- hen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 4231-4244
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der. Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022a. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evaluation of chatgpt as a question answering system for answering complex questions. CoRR, abs/2303.07992
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022b. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. Rat-sql: Relation-aware schema encoding and linking for text- to-sql parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578.
Panupong Pasupat and Percy Liang. 2015. Compo- sitional semantic parsing on semi-structurrd tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1470- 1480.
Xiaokai Wei, Shen Wang, Dejiao Zhang, Parminder Bhatia, and Andrew O. Arnold. 2021. Knowledge enhanced pretrained language models: A compresh- ensive survey. CoRR, abs/2 10.08455.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong,
Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengeheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni. Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer and Tao Yu. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 602-631.
Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng. Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. CoRR, abs/2302. 12813.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Chenghu Zhou, Xinbing Wang. Quanshi Zhang, and Zhouhan Lin. 2022. Rasat: Integrating relational structures into pretrained seq2seq model for text-to- sql. arXiv preprint ar arXiv:2055.6693
Yunhu Ye, Binyuan Hui, Min Yang. Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language mod- els are versatile decomposers: Decompose evidence and questions for bable-based reasoning. CoRR.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou,

---

Wen-tau Yih, Matthew Richardson, Christopher Meek. Ming- Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Anmual Meet- ing of the Association for Computational Linguisties. ACI. 2016, August 7-12, 2016, Berlin, Germany, Vol- ume 2: Short Papers. The Association for Computer Linguistics.
Tao Yu, Rui Zhang. Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October November 2018, pages 3911-3921.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01066
Yuyu Zhang, Hamjun Dai, Zornitsa Kozareva, Alexan- der J. Smola, and Le Song. 2018. Variational reason- ing for question answering with knowledge graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th inno- vative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI- 18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 6069-6076.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. 2023. survey of large language models. CoRR.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR.

---

Table 4: Results of different version of ChatGPT for WebQSP, WTQ, and Spider.
ExxracttColuumne and Extract_SubTablee SubTable functions and proper instructions, we elicit the LLM select the useful row indices (i.e., item 8) and finally generate the answer (i.e,, 19th).
Methods
WebQSP
WTQ
Spider
ChatGPT (June)
61.2
43.3
70.1
For database, we also first invoke the Ex- tract tract_TTalleee Table&Column, Name to extract all the ta- ble names and column names, linearize them and utilize the instruction (i.e., which tables do you need to complete the SQLite SQL query?) prompt the LLM. Then, based on the selected bles (i.e., Dogs and Breeds), we further invoke the Extract_Tables Tablic__nnfoomattinn Information function and prompt the LLM via an instruction (i.e., complete sqlite SQL query only with no explanation) to generate the SQL for the question, which can be executed to obtain the final answer.
ChatGPT (June) IRR
72.6
48.4
74.8
ChatGPT (August)
62.1
41.1
75.2
ChatGPT (August) IRR
75.3
50.4
77.1
A Experiment With Latest Version of LLM
We have noted that the ChatGPT is continuously evolving. Furthermore, we have conducted sup- plementary experiments on three datasets using the latest August version of LLM. The results are presented in the Table 4. It is noteworthy that Chat- GPT indeed continuously evolves, as evidenced by its distinct performance compared that of the June version. Although the evolved ChatGPT un- derperforms compared to the June version on the WTQ dataset, our approach can consistently fur- ther enhances the ChatGPT performance with the evolved version on all three tasks. It indicates the robustness of our proposed method.
B
Case Study
Here, we select one representative example for each type of structured data and present the case study in Figure 4. For KG, we first invoke the Ex- tract_ LeeNNNeeiooooeeattitee Relations function extract the neighboring relations (e.g., birthplace, residence, and education) of the topic entity "Harper Lee". then linearize them and compose the input prompt. the prompt, we utilize the instruction (i.e., pro- vide only one relevant relation that's present in the candidate) to elicit the LLM to generate the most relevant relation, i.e., education. Based on the selected relation, tract_ tract_Triples function extract the triples with the relation the topic entity. After linearization, another instruction (i.e., you just need to provide only one answer entity), adopted for guiding the LLM to generate the final answer, ie., Monroe County High School.
further invoke the Ex-
For table, we first invoke the Ex- tract_ Coot_ColumnnNMe Name function to extract the column names from the table for linearization, and then design the prompt (i.e., which columns are most relevant to answering the question?) for the LLM to select the useful columns, i.e., District and Incumbent. Then, by using the

---

Invoke Extract Neighbor Relations (Harper
uestioon: what highschool did haper lee go to?
college
(xxoord
The candidate relations: eduration hirthplace.
type
CVT institution
The question
Return [education.... birdppacee]
residencco
edincation
Provide only one relevant relation that's present: in the candidates...
Linearie "oducation birtaplaod"
Harpe Lee
birthplace
education
type
CVT
institution
high
Monrve County
Imoker
school
The relevant relation: education.
Extract Triples (Harer (educatioo])
Generate
The triples are: (Harper Lee, eduration,
Retur:
collego
Oxford
CVT 0) Based on these triples give
[Harper Lee, education, CVT 0)
type
CVV
institution
the final answer entity.
edducation
You just need to provide only one answer entity.
Lineurizer
rusidenco
Harpe Lee)
you think...
"[Harpe Lee, education, CVT_0), __.)...""
educationn
birthplace
type
CVT
institution
Mouroe County
G Cenerate
A nswer: Monroe County High School
school
(a) Case of WebQSP (KGQA)
Invoke: Extract Columa Name (table)
Qussiion what district was incumbent charies hawkins?
District
Incumbent
2007
To answer ... first look at the available columns
Result
table: "District", "Incumbent"; Which columns are most relovant answering the question?....
Reshure [Didtrict, 2007 Result]
Marty Williams
John
1st
Lincarize "District, .... 2007 Result"
charles haw hawkins
Robert
19st
Inooke
...
eneerate
Columns: District, Incumbent.
Extract Columns (table (District....)
2007 Result
District
Incumbent
To answer Below list rows .... row (District, 1st); (Incumbent, Marty
Rehurn [(row I, (District, 19th)...)....
Marty Williams
John
1st
Williams) ... (2007 Result ...]
Miller
Lincaria: "row 1: (Disttict, Isl); ..."
which rows should be considered?...
charles haw kins
Robert
Inooke
19st
Hurt
Rows: 19
Generate
Extract, SubTable fuall, [Distriet..... (19))
The table contains:
District
Incum
2007
Result
1: (District, 19th): (Incumbent, Charles Hawkins),
Reture (rro 1, (District, 19th),
Marty Williams
John
1st
Using this information, In what district was the Linearize: "(row 1, (District, 19th), ...)" incumbent Charles Hawkins?...
Miller
charles bawkins
Robert
Answer: 19th
19st
Hurt
G Generate
(b) Case of WTQ (TableQA)
Insole: Extract, Columm Name (database)
Q uestioon:
What is the name of the breed with the most dogs?
Table
Column names
### Here are the SqliteQQL tables
Broods(+ broed code, brood name):
Retumn [Dogs ... Breeds]
Dogs
dog
breed code
##* W hat is the name of the breed with the most dogs? W hich tables do you need complete the SQLite SQL query?
Linearize "Dogs Breeds"
Breads
breed cade
... breed name
Dogs.breed code + Breed breed code
Invoke: Extract Tables Information (DDgss Breeds))
Dogs Breeds
Generate
#*# Complete sqlite SQL query only and
Table name
Column names
Breeds(" breed, code, breed name); #** What is name the breed with most dogs? SELECT
Return [Dogsídog id, ...)..]
Dogs
dog
breed code
Linsutiz "Dogsídog, y..."
Bresds breed
breed name
G Generate Breeds.breet name FROM Dogs JOIN
Dogs.breed code +> Breed breed code
(c) Case of Spider (Text-to- to -QLL)))
Figure 4: Case study of our method on KGQA, TableQA and Text-to-SQL. task.


Reviews:
[{"review_id": "wO9SicXV2T", "reviewer": "Reviewer_b2MA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows. \n\n**1.The experiments can be more convincing. As far as I understand, the current method needs to assume the entities are extracted and given so that the intermedia functions/interfaces can be utilized. This is a strong assumption: 1) entity extraction and linking is not trivial, 2) it breaks the end-to-end property of semantic parsers. Discussions (or even experiments) should be included.**\n\nThank you for your valuable feedback. Here, we want to first outline the task formulations for three types of tasks as the following table.\n\\\\begin{array} {|c|c|c|c|}\n\\\\hline\n\\\\textrm{Task} & \\\\textrm{input} & \\\\textrm{Structured~Data} & \\\\textrm{Output} \\\\\\\\\n\\\\hline\n\\\\textrm{KGQA} & \\\\textrm{question} &  \\\\textrm{KG} &  \\\\textrm{answer entities}  \\\\\\\\\n\\\\hline\n\\\\textrm{TableQA} & \\\\textrm{question} &  \\\\textrm{Table} &  \\\\textrm{answers}  \\\\\\\\\n\\\\hline\n\\\\textrm{Text-to-SQL} & \\\\textrm{question} &  \\\\textrm{DB} &  \\\\textrm{SQL}  \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n(1) Only for  KGQA task, entity linking is essential, which indeed is a **standard and necessary preprocessing** operation to support subsequent reasoning process [1-4]. Notably, to ensure the practical application, existing work [1-4] adopts available tools (e.g., Google Knowledge Graph Search API) and models (e.g., ELQ [5])  for performing entity linking instead of using golden annotated ones. We just follow such preprocessing paradigm and focus more on the afterward reasoning process. Actually, it is very expensive to implement entity linking through LLM, as the KG generally consists of millions of entities. As a comparison, the traditional entity linking tools and models work more efficiently and can even achieve better performance. In this way, our experimental settings are **consistent with existing KGQA work**, which ensures fair comparisons with baselines presented in Table 1. In the future work, we would investigate the potential of utilizing LLMs to implement entity linking.\n\n(2) For TableQA and Text-to-SQL tasks, our task formulation also aligns with previous supervised fine-tuned models, where the input consists of the question with table information (for TableQA) or DB information (for Text-to-SQL), and the output is the answers (for TableQA) or SQL (for Text-to-SQL). Differently, our approach focuses on the zero-shot settings without any fine-tuning on these tasks, hence we can not rely on supervised tuning (used in previous work) to adapt into the special structured data format. And our experiments in Table 2 and Table 3 (ChatGPT v.s. ChatGPT + IRR) have shown that simply feeding the complete structured data information (e.g., table or DB) into the LLM can not achieve a good performance. Therefore, we propose a unified framework that separately fulfills all the sub-tasks within the end-to-end reasoning process. Concretely, LLMs first select partially required information from the original structured data (e.g., the sub-table content for TableQA or the DB meta-information for Text-to-SQL), which is then utilized to generate target outputs. Such a way can reduce the input length, mitigate the effect of noisy information, and facilitate zero-shot generalization using our designed prompts for improved performance.\n\n**2.The idea of having an intermedia functions/interfaces is similar to designing an intermedia query language (e.g., \"Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation\").**\n\nThanks for your recommendation, and we will add this reference. In this work, we aim to propose a unified and general framework that supports LLMs for reasoning over various structured data. In this way, our approach is not binded to downstream specific tasks, and can solve them only needs the corresponding interfaces. Instead, due to the diverse formats of structured data, it is challenging to employ a proper custom query language to unify them. \n\nConcretely, we propose to design the intermediate interfaces tailored to each type of structured data and establish a unified interaction approach between LLMs and these interfaces. This design ensures the scalability and flexibility of our approach, as we just require to design new interfaces specially for new structured data and tasks in our framework. Your recommended work has provided us with new insights, prompting us to consider transforming questions into an intermediate representation that bridges natural language and the final Query Language. We consider to try this idea in the future, and we also welcome further discussion on developing a new programming language for enhancing interaction between LLM and structured data.\n\nReference:\n\n[1] Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William W. Cohen: Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. EMNLP 2018.\n\n[2] Apoorv Saxena, Aditay Tripathi, Partha P. Talukdar: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings. ACL 2020.\n\n[3] Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, Ji-Rong Wen: Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. WSDM 2021.\n\n[4] Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, Hanwang Zhang: TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph. EMNLP 2021.\n\n[5] Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, Wen-tau Yih: Efficient One-Pass End-to-End Entity Linking for Questions. EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* a simple and universal method for helping LLMs handle structured data\n* the experiments and error analysis are informative and thorough", "reasons_to_reject": "* The experiments can be more convincing. As far as I understand, the current method needs to assume the entities are extracted and given so that the intermedia functions/interfaces can be utilized. This is a strong assumption: 1) entity extraction and linking is not trivial, 2) it breaks the end-to-end property of semantic parsers.  Discussions (or even experiments) should be included.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper developed an iterative reading-then-reasoning framework for tasks that involve querying structured data (e.g., tables, knowledge graph or relational databases). The key idea is to decompose the mapping from input NL to output answer/SQL into a series of reading and reasoning steps. The reading step is based on some specialized APIs that make it easier to retrieve relevant columns/relations of the entities mentioned in NL. Then during the reasoning step, models can select and use necessary columns/relations to synthesize a SQL or directly generate an answer. The design of intermedia data-accessing functions are shown to be very useful to augment LLM for handling structured data such as tables, knowledge graphs and relational database.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The idea of having an intermedia functions/interfaces is similar to designing an intermedia query language (e.g., \"Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation\")."}}, {"review_id": "7R1eYtoBOB", "reviewer": "Reviewer_1ymw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows.\n\n**1.The actual content of the paper is rather limited, and there are instances of redundant descriptions. For example, the two paragraphs from line 229 to line 272 redundantly describe the motivation and work approach already covered in the introduction. Additionally, I suggest that since the author has mentioned the lack of domain knowledge in LLMs for specific tasks, it would be helpful to present a small-scale validation experiment using any of the datasets mentioned in this paper.**\n\nThanks for your valuable suggestion. In the approach section, we provide a comprehensive overview that aims to ensure this part is self-contained and easily comprehensible. We will revise this section by providing a more concise overview and adding the detailed prompt designs and interaction ways for better illustrating our method after the review period.\n\nIn fact, in our experiment, we have reported the performance of LLMs for specific tasks. For the example of KGQA task, the WebQSP and MetaQA datasets need world knowledge and movie domain knowledge, respectively. We directly utilize LLMs to answer questions without accessing any external knowledge source and the results are presented in Table 1. As we can see, LLMs perform poorly on these datasets. In contrast, their performance significantly improves when leveraging the domain knowledge from KG with our proposed method. This highlights the lack of domain-specific knowledge in LLMs for specific tasks and demonstrates how our approach effectively mitigates this issue by enabling access to external structured data. We will also add this description to illustrate the lack of domain knowledge in LLMs for specific tasks in our paper following your advice.\n\n**2.Since ChatGPT has been used as an LLM, it is recommended to mention the date of its retrieval, as different versions of the model may yield different results.**\n\nThanks for your suggestion. We will note that the June version of ChatGPT was utilized in the experiment section. Furthermore, we have conducted supplementary experiments on three datasets using the latest August version of LLM. The results are presented in the following table. It is noteworthy that ChatGPT indeed continuously evolves, as evidenced by its distinct performance compared to that of the June version. Although the evolved ChatGPT underperforms compared to the June version on the WTQ dataset, our approach can consistently further enhances the ChatGPT performance with the evolved version on all three tasks. It indicates the robustness of our proposed method.\n\n\\\\begin{array} {|l|c|c|c|}\n\\\\hline\n\\\\textrm{Model} & \\\\textrm{WebQSP} & \\\\textrm{WTQ} & \\\\textrm{Spider} \\\\\\\\\n\\\\hline\n\\\\textrm{ChatGPT (June)} & 61.2 & 43.3 & 70.1 \\\\\\\\\n\\\\hline\n\\\\textrm{ChatGPT (August)} & 62.1 & 41.1 & 75.2 \\\\\\\\\n\\\\hline\n\\\\textrm{+ IRR (June)} & 72.6 & 48.4 & 74.8 \\\\\\\\\n\\\\hline\n\\\\textrm{+ IRR (August)} & 75.3 & 50.4 & 77.1 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n**3.There are already existing open-source toolkits that integrate a multitude of tool calls (e.g., langchain), and the author should provide some descriptions or examples to illustrate the characteristics and advantages of the proposed method compared to these existing tools.**\n\nThanks for your suggestion and we will add this discussion in our next version paper following your advice. Existing open-source toolkit such as langchain is a Python development framework that allows developers to expand the capabilities of LLMs, and build end-to-end language model applications. Under this framework, LLMs can access external tools, such as search engines, math calculators, and Python interpreters, to easily expand their capabilities. Simultaneously, it enables LLMs to access external data resources using index and retrieval tools. However, the framework has two major limitations that make it difficult to process structured data. \n\nFirstly, the **data format of structured data** is different from normal document format, which makes it hard for LLMs to understand it using traditional dense retrieval tools which ignores the structure information. \n\nSecondly, the framework's **direct retrieval then utilization** approach has limited ability to support LLMs in solving complex user queries that requires multi-hop reasoning over structured data.\n\nIn contrast, our proposed iterative reading-then-reasoning approach allows LLMs to **iteratively access external structured data** and **select the required data** to perform reasoning using designed interfaces of structured data and invoking-linearization-generation procedures. Besides, our approach is scalability and flexibility, as we just require to design new interfaces specifically for new structured data and tasks in our framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The article has a logical and well-structured writing style.\n- Considering the outstanding performance of LLMs in various NLP tasks, the motivation behind the paper is reasonable. Investigating how to enhance the effectiveness of LLMs in domain-specific tasks that require domain knowledge is a meaningful research topic.\n- The article provides rich experimental results and case analyses.", "reasons_to_reject": "As mentioned above, the motivation of this paper is reasonable. However, from a research perspective, this paper represents a relatively lightweight work. For the community, the significance of this paper lies more in validating the ability of ChatGPT to integrate specific knowledge. My concerns regarding this paper are as follows:\n\n- The actual content of the paper is rather limited, and there are instances of redundant descriptions. For example, the two paragraphs from line 229 to line 272 redundantly describe the motivation and work approach already covered in the introduction. Additionally, I suggest that since the author has mentioned the lack of domain knowledge in LLMs for specific tasks, it would be helpful to present a small-scale validation experiment using any of the datasets mentioned in this paper.\n- Since ChatGPT has been used as an LLM, it is recommended to mention the date of its retrieval, as different versions of the model may yield different results. \n- There are already existing open-source toolkits that integrate a multitude of tool calls (e.g., langchain), and the author should provide some descriptions or examples to illustrate the characteristics and advantages of the proposed method compared to these existing tools.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a framework called Iterative Reading-then-Reasoning (IRR) that enhances the inference capability of LLM in specific tasks by incorporating structured knowledge obtained through the pre-designed external interfaces. The structured knowledge includes three aspects - Knowledge, Table, and Database. The authors design interfaces and implement the entire framework for KGQA, TableQA, and Text-to-SQL tasks. They conducted experiments on multiple typical datasets for these tasks and the results demonstrate that the framework outperforms ChatGPT zero-shot significantly.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WMCznmtNc6", "reviewer": "Reviewer_KDgm", "summary": "", "questions": "Why does the LLM extract spouse, Stadium or visitor, as indicated in figure 1? \n- how do you extract for example the name Harper Lee from the question? Is a named entity extraction tool used? Or is this another call to an LLM?(figure 2)", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows.\n\n**1.unclear description of approach - Figure 1 is quite confusing with multiple arrows pointing to an LLM box - It's unclear how the prompts interact with the interface. Is this an agent-based approach?**\n\nThe core process of our approach is the iterative invoking-linearization-generation procedure illustrated in Figure 1. Concretely, at each iteration, we first invoke the interface to extract relevant information from structured data. Secondly, we convert the extracted information into a textual sentence by linearizing it. Next, we construct the LLMs' input by combining the linearized extracted information **(arrow from \"Linearization\" box to \"LLM\" box)** and the question **(arrow from \"Question\" box to \"LLM\" box)**. Finally, the LLMs select useful evidence for next iteration of reasoning **(arrow from \"LLM\" box to selected information box)** or generating the target outputs **(arrow from \"LLM\" box to \"Final Answer or Executable SQL\" box)**. In Figure 1, we present the data flow of the three types of tasks in parallel, distinguished by different colors. The solid arrows indicate the data flow, while the dashed arrows represent multiple iterations. Here, we further present an example of KGQA task to explicate the aforementioned process along with the prompts utilized in each step.\n\n```\nInvoking: \nExtract_Neighbor_Relations (Harper Lee)\n\nReturn: \n[fictional_universe.fictional_character_creator.fictional_characters_created, en, ... ,award.award_nominee.award_nominations, award.award_winner.awards_won, base.biblioness.kjauthors]\n------------------------------------------\nLinearization: \nfictional_universe.fictional_character_creator.fictional_characters_created, en, ... ,award.award_nominee.award_nominations, award.award_winner.awards_won, base.biblioness.kjauthors.\n------------------------------------------\nGeneration:  \nThe candidate relations: fictional_universe.fictional_character_creator.fictional_characters_created, en, ... ,award.award_nominee.award_nominations, award.award_winner.awards_won, base.biblioness.kjauthors.\nThe question is \"what highschool did harper lee go to?\" and you'll start with \"harper lee\". To answer this question, typically you would need to identify some relations that correspond to the meaning of the question. Therefore, select one relation from the candidate relations above that can be used to answer the question. Provide only one relevant relation that's present in the candidates, and begin your response with \"The relevant relation: \n\nOutput:\nThe relevant relation: people.person.education\n------------------------------------------\nInvoking: \nExtract_Triples (Harper Lee, [people.person.education])\n\nReturn: \n[[Harper Lee, people.person.education, CVT_0], [CVT_0, education.education.end_date, 1944], [CVT_0, education.education.institution, Monroe County High School]...]\n------------------------------------------\nLinearization: \n(Harper Lee, people.person.education, CVT_0), (CVT_0, education.education.end_date, 1944), (CVT_0, education.education.institution, Monroe County High School).\n------------------------------------------\nGeneration:  \nThe triples are: (Harper Lee, people.person.education, CVT_0), (CVT_0, education.education.end_date, 1944), (CVT_0, education.education.institution, Monroe County High School).\nBased on these triples, if you believe you have gathered sufficient information to answer \"what highschool did harper lee go to?\", give me the final answer entity and start your response with \"The final answers:\". You just need to provide only one answer entity. If you think you still do not have enough information to answer the question, respond \"Need further information\".\n\nOutput:\nThe final answer: Harper Lee went to Monroe County High School.\n```\n\n\nAs you can see, such a process indeed enable the LLMs as an agent to interact with various structured data under a general framework, which is not be easily implemented by traditional agent-based approaches.\n\n**2.performance is below or just on pair with prior art work**\n\nIt should  be noted that prior SOTA methods are almost small models  that are specially fine-tuned by  the training set in the experiments. In addition, a model trained on some dataset is difficult to generalize or work well on another different evaluation dataset. As  a comparison, our approach are zero-shot or few-shot learning, and a fixed LLM solve all  types of tasks. Although LLMs have shown surprising performance recently, they still difficultly handle the structured data without any fine-tuning, whose performance has a significant gap compared to existing supervised fine-tuned models. Our approach helps LLM greatly reduce this performance gap in three typically structured data, and even surpass partial supervised fine-tuned models, which demonstrates the superiority and generality of our approach. Performance improvement is just to illustrate the effectiveness of our approach. Most importantly, our work provide an new perspective how to apply the LLMs to the structured data, which is meaningful to deploy LLM in realistic scenarios.  \n\n**3.Reproducibility with chatGPT is difficult due to non-determinism of the system and it is unclear with model version was used**\n\nThe difficult reproducibility problem of ChatGPT mainly coms from the decoding randomness and the continuously evolved model version. Both of them are objective factors that can not  be controlled by our method and we have try our best to mitigate their effects.\n\nFirst, during our experiment, we set all the hyper-parameters relevant to decoding randomness (e.g., temperature) as the value of no randomness (e.g., 0) according  to the official guidelines of OpenAI. And we have open sourced all of the used prompt, hyper-parameter config, and the final ChatGPT output to ensure the reproducibility of our results. And the June version of ChatGPT was utilized in the experiments, and our results can be fully reproduced under this version. We will supplement all these details in the revised paper.\n\nSecond, our approach is a general LLM-agnostic framework, whose effectiveness would not be influenced by the different model versions. We conduct the supplementary experiments on three datasets using the latest August version of LLM. The results are presented in the following table. It is noteworthy that ChatGPT indeed continuously evolves, as evidenced by its distinct performance compared to that of the June version. Although the evolved ChatGPT underperforms compared to the June version on the WTQ dataset, our approach can consistently further enhances the ChatGPT performance with the evolved version on all three tasks. It indicates the robustness of our proposed method. \n\n\\\\begin{array} {|l|c|c|c|}\n\\\\hline\n\\\\textrm{Model} & \\\\textrm{WebQSP} & \\\\textrm{WTQ} & \\\\textrm{Spider} \\\\\\\\\n\\\\hline\n\\\\textrm{ChatGPT (June)} & 61.2 & 43.3 & 70.1 \\\\\\\\\n\\\\hline\n\\\\textrm{ChatGPT (August)} & 62.1 & 41.1 & 75.2 \\\\\\\\\n\\\\hline\n\\\\textrm{+ IRR (June)} & 72.6 & 48.4 & 74.8 \\\\\\\\\n\\\\hline\n\\\\textrm{+ IRR (August)} & 75.3 & 50.4 & 77.1 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n\n**4.Why does the LLM extract spouse, Stadium or visitor, as indicated in figure 1? how do you extract for example the name Harper Lee from the question? Is a named entity extraction tool used? Or is this another call to an LLM?(figure 2)**\n\nIn Figure 1, the output from the LLMs indicates that they have selected the required information from the linearized input data, which was extracted by invoking the interfaces of structured data. In detail, the \"spouse\" is the selected relevant **relation** after invoking the *Extract_Neighbor_Relations()* interface of KG, the \"Stadium\" is the selected relevant **column** name after invoking the *Extract_Column_Name()* interface of table, and the \"visitor\" is the selected relevant **table** after invoking *Extract_Table&Column_Name()* interface of DB. You can refer to the above answer to Q1 for the detail description of our approach process.\n\nAs for the extracted \"Harper Lee\" from the question, it can be extracted via entity linking. As described in Section 4.4, for KGQA dataset, we just follow existing work [1-4] to perform entity linking in advance with existing linking tools (e.g., Google Knowledge Graph Search API) and models (e.g., ELQ [5])  for performing entity linking instead of using golden annotated ones. Please note, we don't assume directly given the golden entity linking results to sure that our approach can be applied to realistic scenarios. Our experiment settings are consistent with existing works on KGQA tasks, which  makes sure a fair comparison with baselines in Table 1.\n\nReference:\n\n[1] Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William W. Cohen: Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. EMNLP 2018.\n\n[2] Apoorv Saxena, Aditay Tripathi, Partha P. Talukdar: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings. ACL 2020.\n\n[3] Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, Ji-Rong Wen: Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. WSDM 2021.\n\n[4] Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, Hanwang Zhang: TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph. EMNLP 2021.\n\n[5] Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, Wen-tau Yih: Efficient One-Pass End-to-End Entity Linking for Questions. EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- novel combination of KG, table and text-to-SQL approaches utilizing LLMs", "reasons_to_reject": "unclear description of approach\n\t- Figure 1 is quite confusing with multiple arrows pointing to an LLM box\n\t- It's unlcear how the prompts interact with the interface. Is this an agent-based approach?\n- performance is below or just en par with prior art work\n- Reproducibility with chatGPT is difficult due to non-determinism of the system and it is unclear with model version was used [addressed by rebuttal]", "questions_to_authors": "Why does the LLM extract spouse, Stadium or visitor, as indicated in figure 1? \n- how do you extract for example the name Harper Lee from the question? Is a named entity extraction tool used? Or is this another call to an LLM?(figure 2)", "paper_topic_and_main_contributions": "The paper proposes an Iterative Reading-then-Reasoning (IRR) framework for LLMs utilizing interfaces to structured data. The framework considers two major functions to fulfill two different tasks, namely collecting relevant evidence (i.e. *reading*) and inferring the answer or planning subsequent steps (i.e., *reasoning*). The paper proposes an invoking-linearization-generation procedure to support LLMs in reading and reasoning on structured data with the help of external interfaces. By iteratively applying this procedure with different interfaces, the target answer can be gradually approached to a given question.\n\nThe authors conduct experiments on a wide range of tasks (e.g., KG-based question answering, Table-based question answering, and DB-based Text-to-SQL) and demonstrates that the IRR framework can effectively enhance the reasoning performance of LLMs on structured data in zero-shot and few-shot settings but not compared to prior art systems that use  supervised-tuning methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a mechanism for iterative reading then reasoning on large language models over structured data sources. A number of data source formats are considered, including SQL, tables and KG data. The method exploits the table / data structure as an interface  or type hint for information extraction and uses this information to return information from a linearized version of the data. 

There is reasonable consistency with reviews, stating that the methodology is sound and has reasonable excitement. The method is dependent on the underlying LM. However, the methods in the paper describe an approach that should be agnostic to the underlying LM provider. It would be advantageous to also provide further experiments with open source LLMs as part of the final version. The disadvantage of only relying on OpenAI's implementation is (as demonstrated by the author's rebuttal), the final answer accuracy can be heavily dependent on the version of the model. 

The reviewers identified several reasons to reject and spotted some useful areas that were missing in the paper, including the representation of the task as generating an intermediate query language. I would further suggest linking this to work in multi-step QA: for example the BREAK task: https://arxiv.org/abs/2001.11770 - regarding the point made by the reviewer about open source toolkits such as langchain, i believe that a module released by the authors of by this paper would be of significant value for the community and i would encourage the authors to release a their code.
Decision: Accept-Main