ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought
Authors: Hanchong Zhang
Ruisheng Cao
Lu Chen
Hongshen Xu
Kai Yu
Venue: EMNLP 2023 Findings (2023)
Keywords: text-to-SQL
large language models
in-context learning
chain of thought

Abstract: Recently Large Language Models (LLMs) have been proven to have strong abilities in various domains and tasks. We study the problem of prompt designing in the text-to-SQL task and attempt to improve the LLMs' reasoning ability when generating SQL queries. Besides the trivial few-shot in-context learning setting, we design our chain-of-thought (CoT) prompt with a similar method to schema linking. We provide a method named ACT-SQL to automatically generate auto-CoT exemplars and thus the whole process doesn't need manual labeling. Our approach is cost-saving since we only use the LLMs' API call once when generating one SQL query. Furthermore, we extend our in-context learning method to the multi-turn text-to-SQL task. The experiment results show that the LLMs' performance can benefit from our ACT-SQL approach. Our approach achieves SOTA performance on the Spider dev set among existing in-context learning approaches.

ACT-SQL:
In-Context Learning for Text-to-SQL with Automatically-Genrratee Chain-of-Thought
Hanchong Zhang, Ruisheng Cao, Lu Chen; Hongshen Xu and Kai Yu
X-LANCE Lab, Department of Computer Science and Engineering
MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China (zhanghanchong, @eenlusz)@sj tu. edu. cn
Abstract
overlap. Prior research endeavors have focused on training specialized text-to-SQL. models and op- timizing their structural components to enhance overall performance. Notably, these efforts have yielded impressive model performances across vari- ous datasets. Nevertheless, the construction of such models necessitates substantial number of high- quality training examples and entails significant time investments for finetuning. Moreover, these models often possess intricate structures, rendering their deployment challenging.
Recently Large Language Models (LLMs) have been proven to have strong abilities in vari- ous domains
tasks. We study the prob- lem of prompt designing in the text-to-SQL task and attempt to improve the LLMs' rea- soning ability when generating SQL queries. Besides the trivial few-shot in-context learning setting, we design our chain-of of- thought (CoT) prompt with similar method to schema link- ing. We provide a method named ACT-SQL automatically generate auto-CoT exemplars and thus the whole process doesn't 'tts need man- ual labeling. Our approach is cost-saving since we only use the LLMs' API call once when generating one SQL query. Furthermore, we extend our in-context learning method to the multi-turn text-to-SQL task. The experiment results show that the LLMs' performance can benefit from our ACT-SQL approach. Our ap- proach achieves SOTA performance on the Spi- der dev set among existing in-context learning approaches.
Recent research has provided empirical evidence cstablishing the substantial capabilities of Large Language Models (LLMs), such as GPT-3 (Brown et al.. 2020) and ChatGPT (Ouyang et al., 2022), across wide spectrum of domains and tasks. As the scale of LLMs continues to expand, scholarly investigations have revealed the presence of emer- gent abilities (Wei et al., 2022) exclusive to larger LLMs and absent in their smaller counterparts. Therefore, the latest studies employ LLMs the context of the text-to-SQL task, utilizing the in- context learning method (Brown et al., 2020). Ow- ing to the impressive performance demonstrated by LLMs in zero-shot or few-shot prompting scenar- ios, the need for extensive finetuning using an abun- dance of training examples has been rendered un- necessary. Consequently, the integration of LLMs in the text-to-SQL process yields notable time and cost savings.
1 Introduction
The text-to-SQL task (Zhong et al., 2017; Xu et al., 2017) aims translate the natural language ques- tion into the corresponding SQL query with the given database schema. It the key technique to establish the natural language interface on re- lational databases, which can help common users access data from relational databases in more convenient way.
Nonetheless, contemporary in-context learning approaches for text-to-SQL encounter certain chal- lenges. For instance, Rajkumar et al. (2022), in comparison to SOTA finetuned models, employ a simplistic prompt designing approach that yields relatively subpar performance. Similarly, Pour- reza and Rafiei (2023) employs a convoluted work- flow to generate the final SQL query, resulting in achieving SOTA performance on the test set of the Spider dataset. However, this approach proves time- time-consuming and resource- iesuurceinttessvve, as it ne-
Recent studies in text-to-SQL. research have pri- marily centered on the development of semantic parsers within the framework of cross-domain anal- ysis. In cross-domain text-to-SQL datasets such as Spider (Yu et al., 2018). SParC (Yu et al., 2019b), and CoSQL (Yu et al., 201 9a), the databases em- ployed in the train set, dev set, and test set do not
"The corresponding author is Lu Chen.
!Our code is publicly available at https: http::///ithul http://.tth.hh///////// X- X-LANCE/ter MANNCE/text25qIGGPP

---

cessitates multiple API calls to LLMs during the query generation process. Morcover, the recent advancements in-context learning methods for text-to-SQL have yet to be extended to multi-turn datasets, such as SParC, CoSQL, and DIR (Li et al., 2023b).
mance under the text-to-SQL task with differ- ent prompting styles and few-shot exemplar selection strategies.
2. We propose our approach named ACT-SQL generate auto-CoT.. The ACT-SQL achieves the SOTA performance on the Spider dev set among existing in-context learning methods. Furthermore, our automatic method cost- saving and time-saving and does not need ex- tra LLMs' API calls.
Despite the proficiency of LLMs as zero-shot and few-shot learners, the mere superficial prompt design fails to fully activate their capabilities. To address this limitation, Wei et al. (2023) proposes a novel prompting technique called Mhain-of-thought (CoT). Through the CoT method, the prompt text encompasses a comprehensive thinking process that guides LLMs towards accurate deduction of answers. Notably, the CoT method mirrors the sequential nature of human reasoning, wherein in- termediate answers obtained before arriving at a final conclusion. Given the intricate nature of the text-to-SQL. task, the CoT method proves highly suitable, as generating the SQL query entails com- plex reasoning processes. However, existing CoT methodologies necessitate extensive time invest- ments in the selection of canonical examples and manual labeling. The text-to-SQL task lacks au- tomated approach for generating CoT sequences.
We extend our method onto the multi-turn text- to-SQL task and achieve comparable perfor- mances with finetuned models on the SParC and CoSQL datasets.
Related Work
Text-to-SQL. models Over the past several years, text-to-SQL researches mainly focus on building well-designed deep neural networks (Chen et al., 202 lb; Cao et al., 2023). RATSQL model (Wang et al., 2020) and LGESQL model (Cao et al., 2021) are AST-based approaches, where AST is the ab- breviation of the abstract syntax tree. They encode the input and decode the AST of the SQL query with predefined grammar. AST-based approaches perform well but are generally complex to deploy. PICARD (Scholak et al., 2021) is a sequence- to-sequence model. SQL is a formal language that follows strict grammar rules. Directly finetuning pretrained language models (PLMs) on text-to-SQL. datasets would make PLMs likely to generate in- valid SQL queries. The PICARD model rejects invalid tokens at each decoding step and constrains the generated results into certain output space. Although these specialized models have achieved excellent performances, there still exist some inevitable disadvantages. In order to train a text-to-SQL model, abundant high-quality training examples are needed. Constructing and labeling a large-scale text-to-SQL dataset is always not easy and would consume a lot of resources and time. Training and finetuning the model is also a hard project which costs many computing resources.
In this paper, we propose our in-context learn- ing method for the text-to-SQL task with the automatically--eneraal COT. First, under the zero- shot setting, we study the influence on LLMs' performance caused by the input format of the database schema. Second, under the few-shot set- ting, we provide hybrid strategy to select exem- plars and study the influence L.Mss' perfor- mance caused by the number of exemplars. Our ex- periment results show that the strategy is effective. Third, we present our approach named ACT-SQL to generate auto-Coi C for the dataset training exam- ple consisting of the database schema, the question, and the corresponding SQL query. The experi- ment results show that the generated auto-CoT can indeed improve the LLMs' performance. The ACT- SQL achieves the SOTA performance on the Spider dev set among existing in-context learning meth- ods. In addition, the ACT-SQL does not need to use extra LLMs' API calls, which means that our work- flow is relatively fast and cheap. Finally, we apply our approach in multi-tum mulli--urrn text-to-SQL datasets in- cluding SParC and CoSQL and achieve comparable accuracy scores with finetuned models. Our main contributions be summarized as follows:
In-context learning for text-to-SQI. Since LLMs have shown amazing ability across various domains and have been applied in many academic and industrial fields, the latest researches begin to activate the L.MS'' ability for the text-to-SQL. task. Rajkumar et al. (2022) uses the trivial zero- shot and sho learning setting and performs an
1. We explore the influence on LLMs' perfor-

---

empirical evaluation of text-to-SQL capabilities of LLMs including GPT-3 (Brown al., 2020) and Codex (Chen et al., 2021a), They perform the zero- shot prompt learning on Spider (Yu et al., 2018), a large-scale human-labeled cross-domain text-to- SQL dataset. Their work is relatively simple and the performance falls behind finetuned models.
late the relatively simple natural language question into the correct SQL query, though no exemplar is provided. Under the zero-shot setting, the LLMs merely take the database schema and the question as the input. Thus the input format the database schema would mainly influence the LLMs' perfor- mance. Generally, we use five different database schema styles, which are shown in Appendix C.I:
Nan et al. (2023) mainly concentrates on the strategy of exemplars selection. Their work achieves good performance on several cross- domain datasets including Spider, Spider-Syn (Gan et al., 2021a), Spider-DK (Gan et al., 2021b) and Spider-Reen Realistic (Deng et al., 2021). However, their work requires an extra preliminary predictor to eval- uate the SQL's difficulty level and needs to use LLMs' API call many times due to the majority vote method.
1. Table e (Column) lists each table followed by its columns in each line. This style follows the official document provided by OpenAl'.
2. Table (Column) (PF) adds primary keys and foreign keys at the end of Table (Coluun)
Create describes all tables and columns with the "create table" statement in the SQL grammar. "NoPF" represents that no information on primary keys and foreign keys is added. Compared with Table (Column), this input format contains information col- umn types (e.g. number and text) and is more similar real SQL statements.
DIN-SQL (Pourreza and Rafiei, 2023) provides relatively complex approach. DIN-SQL consists of a complex workflow that decomposes the problem into several simpler sub-problems. With the LLM GPT-4, DIN-SQL has surpassed previous finetuned models and has achieved the best score on the Spi- der dataset. But DIN-SQL's workflow is obviously slow and expensive since it uses LLMs' API call many times generate one SQL..
4. Create(EoC) adds primary keys and foreign keys based on Create (NoPF). "EoC" repre- sents that they are added the end of the corresponding column.
3 Methodology
With the in-context learning method, the SQL gen- eration process can be formulated as
5. Create(EoT) adds primary keys and foreign keys based on (NoPF). "EoT" repre- sents that they are added the end of the table.
S LLM(IT, D, Q,
I represents the instruction. D represents the database schema. Q represents the question. E (D1,Q1, P.).... (Dn, Qn. P.)] is the list of ex- emplars where P; is the answer prompt which con- tains the correct SQL for the i-th exemplar. Thus the performance of LLMS mainly influenced by the database prompt style, the exemplar selection strategy, and the exemplar prompt design.
Furthermore, database contents are concerned. Specifically e example rows are appended to each table. Appendix C.2 shows instances where c = 3.
3.2 Exemplar Selection
Given a few exemplars, LLMs can benefit and ac- quire tips from them and thus generate SQL queries with a more standard format and higher accuracy. Exemplar selection is an important work under the few-shot setting, which would influence the LLMs' performance lot.
In this section, we first describe the prompt styles of the database schema. Then we state our strat- egy of exemplar selection for the few-shot learning setting. Furthermore, introduce our ACT-SQL approach, i.c. the automatically generated CoT method for constructing effective answer prompts. Finally, we extend our approach to the multi-turn text-to-SQL task.
We select exemplars using a hybrid strategy. Specifically, we first of all select n. examples from the training dataset at random. These dataset exam- ples are named static exemplars. They would be used in the context of every test case. As for each specific test case, we select n. extra examples from
3.1 Database Prompt Style
Previous works have shown that given the database schema, strong LLMs (e.g. GPT models) can trans-
htttps://plafforrts opwmai.ppaa.....mm wtwpii..omeeeeee exappes///////////m/ee/. default-sqL baaalgtraannn translate

---

the training dataset. These dataset examples are named dynamic exemplars since they are selected according to some features of the current test case. Consequently, there are total ns + na exemplars for each test case.
We design our chain-of-thought prompt with a similar method schema linking. Figure 1 shows an instance of the manually labeled CoT for the example from the train set of the Spider dataset (Yu et al., 2018). As suggested in Kojima et al. (2023), the CoT prompt starts with "Let's think step by step". For each slice of the question sentence that may contain some information about the schema item, add them into the CoT prompting text in the format shown in Figure 1. Furthermore, the values mentioned in the question and the SQL query are also a concern. The final SQL query is appended at the end of the CoT prompt.
In order to get dynamic exemplars that are more relevant to the current test case, we compare the nat- ural language question of the current test case with all questions in the training dataset. We calculate the similarity scores with the suitable pretrained model and then select the top-Na training dataset examples. We believe that dynamic exemplars with more relevant questions would provide more effec- tive information the LLMs.
Auto-CoT Although CoT prompts can be manu- ally labeled, it costs a lot of time find sufficient canonical and effective training dataset examples for CoT labeling. In addition, manually labeled CoT exemplars are fixed, which means that they are all static exemplars and dynamic exemplars are deficient. In order to deal with problems in the manual labeling process, introduce au- tomatic method generate auto-CoT prompts for every example in the training dataset.
3.3 Chain-of-Thought Prompt Design
Under the few-shot learning setting, it has been proven that the LLMs' performance can benefit lot from the chain-of-thouuht (CoT) (Wei et al., 2023) method. In the text-to-SQL task, only the database schema, the question, and the correspond- ing SQL query are provided in the prompt under the trivial few-shot learning setting. However, with the CoT method, the thought process of how to write the correct SQL query is added to the prompt. These prompting texts can help the LLMs think step by step when generating the complete SQL query and thus can activate the logical reasoning ability of the LLMs.
Given the question & = (2,.92.... ................................ ...)))))))) and the SQL query 8, the q, represents the i-th to- ken in the question sentence. We define di = (g, I++1,-... ,gi as slice of the original question. We first enumerate each column [tab]- [col] appear- ing in the SQL query, where [tab] represents the table name and [col] represents the column name. For each column, we use the suitable pretrained model to compute the similarity scores between the current column and all the question sentence slices. The most relevant slice is
In previous works, some grammar-based text-to- SQL models utilize the graph encoding technique to jointly encode both the database schema and the question. Schema linking (Bogin et al., 2019; Wang et al., 2020; Cao et al., 2021) is a commonly used algorithm for building the input graph. If the question tokens exactly or partially match some schema item (i.e. table and column) names, then they are linked with the specific graph edge. It is obvious that the schema linking method can help the text-to-SQU -QL models fetch the most relevant ta- bles and columns among a great number of schema items based on the question.
arg max Sim([tab)-[col) (c.)). Tr
where Sim is the similarity function. We link the column and its most relevant slice and add them to the auto-CoT prompt the same format as the manual labeled CoT prompt. Note that during this process, we ignore the column appearing in the GROUP BY clause of the SQL query, since the GROUP BY column is commonly not mentioned directly in the question.
Monualty Labeled CoT
product
Queation: Aid name all ciorets with city wes greater than 10 posslition larger than 100020.
SGL SELLO Danad, natte HOM datret WEEE: caly_eess to cate city, Popsualion 100020
Secondly, enumerate each table [tab] appear- ing in the SQL query, where [lab] represents the table name. In this process, we eliminate tables that have occurred in the columns, since those tables have been added into the auto CoT prompt. The left tables only appear in the FROM clause and
Coll:
According by stes
ciorkee columns plente: Desntot nome) used.
According 'eity areat, columni jdsttic jolunrsjdsttet City, Areal may used.
According sosp-tator: co uniaa pauteet City. Aspoatori may used Vatuee nczen2) may used
final anower
SELBST Disstol FROM debrk WHERE City_ City Populotlin 100200
Figure 1: Manually labeled CoT for the dataset example.

---

indicate some extra information. For each table. we also compute all the similarity scores and find out the most relevant question slice, i.e.,
4 Experiments
4.1 Experiment Setup
Models We mainly use the GPT-3.5-turbe model to evaluate our proposed approach. The GPT-3.5- turbo model is a low-cost LLM and is very large to have the emergent ability (Wei al., 2022) for handling the text-to-SQL task. In addition, we use the GPT-4 model to evaluate our auto-CoT method on the Spider dataset (Yu et al., 2018), since the GPT-4 model has a stronger reasoning ability but is much more expensive. We use the PLM text2vec- base-chinese to compute the similar- ity scores when selecting dynamic exemplars and generating auto-CoT prompts.
arg max Sim([tab), qs.)). Qi.
We link the table and its most relevant slice and add them the auto-CoT.
Finally, we enumerate the values in the SQL query and then add them to the auto-CoT. Figure 2 shows an instance of the auto-generated CoT from the train set of the Spider dataset.
Auto-CoT
DB ID: twitter"
Questton: What the parttion id of the user named "trnn Man":
SOL: SELECT partttonid FROM user profiis WHERE name = Tron Man'' Coll
Let's think
Hyperparameters The temperature in LLMs' API set to 0, i.c. the greedy decoding strat- egy is applied. The text-to-SQL tasks require the model to generate SQL queries with strict gram- mar rules. The LLMs are likely to generate invalid SQL queries or to write SQL queries that are not relevant to the given questions if the temperature is too high. The number of max tokens is set to 150 for the trivial in-context learning setting and 750 when using the CoT method.
Accorcing fuser named", columns lveer protles.namee)
used,
Accarcing "partitio the usert,
[user_ profiles partffonic) used.
(ron Man] may
final answer ip ECT paritionid FRCM user profiee WHERE name tron Man!
Figure 2: Auto-Col for the dataset example.
3.4 Extension for Multi-turn Text-to-SQL
The prompts described in the previous sections are designed for the single-turn text-to-SQL task. However, questions in the multi-turn text-to-SQL. task are context-dependent and thus those prompts cannot be directly used. Moreover, the auto-Col ToT method is also disabled under the multi-turn set- ting, since the auto-CoT method finds information about schema linking based on the question slices. Under the multi-turn setting, this information may distribute into several context-dependent sentences.
Datasets We mainly evaluate our proposed ap- proach on Spider, large-scale human-labeled cross-domain text-to-SQL dataset across 200 databases covering 138 domains. The Spider dataset contains 8,659 examples in the train set and 1,034 examples in the dev set. It also provides the evaluation script which divides SQL queries into four categories (i.c. easy, medium, hard, and extra) according to the difficulty level. The test set of Spider is not publicly available. We conduct the experiments on the dev set.
In order to deal with the challenge of the multi- turn text-to-SQL task, we use LLMs to convert the multi-turn text-to-SQL task into the single-turn text-to-SQL task. Concretely, with the help of the LLMs, we can rewrite the question sentences and remove the context dependency among them. Thus each rewritten question and its corresponding SQL query turn into new independent dataset exam- ple. We then directly apply the previous in-context learning method in the converted multi-turn text-to- SQL task.
In addition, we also conduct the in-context learning experiments
Spider-Syn (Gan et al., 2021a), Spider-DK (Gan et al., 2021b) and Spider- Realistic (Deng al., 2021). Based on Spider, Spider-Syn replaces some schema-related tokens in the question with synonyms, which would make models unable to discover useful schema items with the simple string mattning method. Spider- DK defines five types of domain knowledge and modifies some examples by adding domain knowl- edge that reflects real-world question paraphrases. Sp evaluates the models' generalization ability across domains when domain knowledge does not frequently appear in the train set. Spider- Realistic removes explicit mentions of column
The quality of the rewritten questions would in- fluence the L.LMs' performance a lot. It is neces- sary to manually label some rewriting exemplars in order to fix the format and improve the quality of the LLMS' outputs. For each multi-turn text- to-SQL dataset, we select 10 examples from the train set at random and manually label the rewritten results.

---

names to evaluate the model's ability to capture text-table alignment.
As discussed in Section 3.1, the LLMs' perfor- mance is mainly influenced by the database prompt style and the rows of database contents under the zero-shot learning setting. We first conduct experi- ments for studying the influence on LLMs' perfor- mance caused by the rows of database contents. We fix the LLM the G model and the database style as Table (Column))m)) and only change the rows of database contents for each table in the prompt. Figure shows the result on the Spider dev set. The LLM achieves the lowest score when no database content is provided. This indicates that database contents can provide useful tips for LLMs, especially when the testing case is sensitive to values in SQL where Table shows two cases. In the first case, the 3 records from the database contain exactly one cell value "France" instead of "French" for the column "singer: SinizesiCiizeesshiip.. Thus the LLM successfully predicts the correct value when these records are added to the prompt. the second case, the database contents point out that "Aberdeen" is the city name so that the LLM can predict the correct SQL structure.
As for multi-turn ext-to-SSL datascts, we con- duct our experiments on SParC (Yu et al., 2019b) and CoSQL (Yu et al., 2019a). SParC totally con- sists of 4,298 coherent question sequences includ- ing 12k+ individual questions and the correspond- ing SQL queries. CoSQL totally contains 10k+ annotated SQL queries. Each dialogue in CoSQL simulates a real-world scenario where the common user is exploring the database and the expert is retrieving answers with SQL..
We use three commonly used evaluation metrics of the text-to-SQL. task: exact match accuracy (EM), execution accuracy (EX), and test-suite accuracy (TS). The EM metric requires each component of the predicted SQL to be equivalent to the corresponding component of the gold SQL. Values in the SQL query are not concerned with the EM metric. The EX metric requires the execution result of the predicted SQL to be correct. Since there may exist different SQL queries that represent the same semantic, the EX metric is commonly more precise than the EM met- ric. The TS metric also evaluates the execution result but requires the result to be correct under multiple database instances per database schema². For multi-turn text-to-SQL. datasets, we evalu- ate our approach with question match accuracy (QM) and interaction match accuracy (IM). The QM score is if the predicted SQL query for the single question is correct. The IM score is if all the predicted SQL queries in the interaction are correct.
Evaluation metrics
What are the names of the singers who are not French citizens?
Question:
DB content O: SELECT Name FROM singer WHERE Citizenship != "French'
DB content 3: SELECT Name FROM singer
WHERE Citizenship != "France"
Questions: Give the flight numbers of flights leav- ing from Aberdeen.
DB content SELECT FlightNo FROM flights WHERE SourceAirport 'Aberdeen'
DB content SELECT FlighuNo FROM flights WHERE Source IN (SELECT AirportCode FROM airports WHERE City 'Aberdeen')
4.2 Zero-shot Results
Table I: Case study for different rows of DB contents.
The LLM gets the best score when the rows of database contents
3. Too much database content in the prompt would not improve the LLMs' performance. Therefore, we always set the rows of database contents to 3 for the subsequent experi- ments.
Table 3 shows the different performances of the GPT-3.5-turbo model when using different database styles. In general, Table (Column))))) and Table (Column) (PF) achieve higher scores than the other three database styles with the zero- shot learning setting because these two database styles follow the OpenAl's official document and
Figure 3: Zero-shot performances of GPT-3.5-turbo using DB style with different rows of DB contents Spider dev set.
htttps://ithhl http:///ggiuuuu.comtattttttt com/taoyds/ best-suite te- see-sql-eaa

---

Finetuned Model
EM
EX
TS
T5-3B+PICARD (Scholak et al., 2021)
75.5
79.3
71.9
RASAT+PICARD (Qi al., 2022)
75.3
80.5
N-best List Rerankers + PICARD (Zeng et al., 2022)
76.4
80.6
Graphix-3BBPIICARD (Li et al., 2023c)
77.1
81.0
RESDSQL-3B+NaISQL (Li et al., 2023a)
80.5
84.1
LLM
Approach
API per SQL
EM
EX
TS
Codex Davinci
Rajkumar et al. (2022)
67.0
55.1
Codex Davinci
Chang and Foster-Lussier (2023)
76.8
Codex Davinci
DIN-SQL (Pourreza and Rafici, 2023)
4
57.2
69.9
GPT-4
DIN-SQL (Pourreza and Rafiei, 2023)
4
60.1
82.8
74.2
GPT-3.5-turbo
ACT-SQL (Ours)
1
62.7
80.4
71.4
GPT-4
ACT-SQL (Ours)
61.7
82.9
74.5
Table 2: Performances of ACT-SQL and other previous works on Spider dev set.
DB Style
EM
EX
TS
The experiment results prove that our ACT- SQL approach is effective. When the GPT-3.5- turbo model uses the ACT-SQL approach with the Create(EoT) database style, it achieves the best EM score of 62.7% and the best TS score of 71.4%. The best database style changes because LLMs can learn from exemplars. Table 13 shows the case study for the ACT-SQL method. With the trivial few-shot learning setting, there is redundant col- umn "TV "TVV_VVVVaeelleee Channel. figameellight_dddffiiiooooooooo wew.idiiionnnTTT appear- ing in the SELECT clause. When the ACT-SQL method is applied, the entire output generated by the LLM contains the complete thinking process which successfully does the schema linking, After clarifying all the tables and columns that may be used SQL, the LLM eventually writes the correct SQL query without any redundant schema item.
Table(Column)
45.3
78.3
69.4
Table (Column)) (PF)
45.4
79.0
69.1
Create (NoPF)
45.3
77.0
66.1
Croate(EoC)
44.8
79.2
67.7
Create(EoT)
44.8
78.3
67.9
Table Zero-shot performances of GPT-3.5-turbo with different DB styles Spider dev set.
may be more similar to LLMs' pretrained data. Create (EoC) and Create (EoT) performs better than Create (NoPF) in EX and TS metrics. This indicates that primary keys and foreign keys in the prompt can offer LLMs effective information.
4.3 Few-shot Results
Table shows all the few-shot experiment results on the Spider dev set, where different database styles and different numbers of static and dynamic exemplars are used. Compared with the zero-shot results, it is obvious that all the EM scores increase lot. This is because SQL queries from the same dataset usually share similar grammar and structure and thus the exemplars from the Spider train set lead LLMs to output a similar SQL query.
Since the GPT-4 model is expensive, we use the GPT-4 model to evaluate our ACT-SQL ap- proach only with the Create database style and n, Rd 2. Table 2 shows the performances of our ACT-SQL and other previous works using in-context learning with LLMs. The ACT-SQL approach uses the LLMs' API call only once for generating one SQL query and achieves the highest EM, EX, and TS scores among existing in-context learning approaches. ACT-SQL's performance is also comparable to finetuned models. Actually, finetuned models would get higher scores on the dev set than the test set, since these models selected by the dev set performance. Instead, in- context learning methods would not suffer the per- formance gap between the dev set and the test set. Table shows some previous works' performances on Spider dev set and test set. For finetumed ap-
Under the trivial few-shot learning setting, the TS scores also get improved by 1%-3% except for the Table(Column) database style. Table (Column) longer performs better than Table (Column) (PF), since LLMs' accuracy for predicting hard and extra hard SQL queries get increased with the few-shot exemplars and thus pri- mary keys and foreign keys in the prompt become more important.

---

proaches mentioned in the table, the performances drop from the dev the set. On the con- trary, for in-context learning approaches mentioned in the table, the performances increase from the dev set to the test set. After all, finetuned models are selected by the dev set performance, which would lead to the overfitting on the dev set and the per- formance dropping the test set. For in-context learning approaches, the dev set and the test set equal the model. Performances between the dev set and the test set are only affected by the dataset feature.
Approach
EM
EX
TS
Graphix-B-PPICARD
72.4
RESDSSQL-3B+NatSQL
77.4
81.9
Few-shot (Ours)
52.4
76.4
62.0
ACT-SQL (Ours)
53.5
75.8
61.2
Table 7: Performances of GPT-3,5-turbo and other pre-
vious works on Spider-Realiste dev set.
QM
IM
Appraach
EX
T8
KM
GAZP+BERT (Zhong et 2020)
48.9
47.8
RASAT+FICARD (Qi et al., 2023)
67.7
73.3
49.1
54.0
Fow-shot (Ouns)
48.4
64.0
558
24.6
398
31.5
ACT-SQ)I(OOrs))
51.0
63.8
5699
24.4
38.9
29.6
Finetuned Approach
Dev
Test
Table 8: Performances of GPT-3.5-turbo and other pre-
Graphix-3B-PICARI RESDSSL-3BB-NaSSOL
81.0
77.6
vious works on SParC dev set.
84.1
79.9
In-context Learning
Dev
Test
C3 (Dong al., 2023)
81.8
82.3
4.4
Multi-turn Datasets Results
DIN-SQI. (Pourreza and Rafiei, 2023)
82.8
85.3
Table 8 and Table 9 show the GPT-3.5-turbo's per- formances on two multi-turn text-to-SQL datasets, i.e. SParC and CoSQL. The database style is set to Create(CoT) and nid nt are set The ACT-SQL approach is not that effective when applied to multi-turn datasets. We believe that our two-phase method causes bad performance. In the first phase, we use LLMs to rewrite questions in the interaction and convert the multi-turn dataset into the single-turn dataset. Sometimes the rewrit- ten result's quality is bad, which influences the schema- linking process. Table 10 shows rewrit- instances from the SParC dev set. In the first instance, the LLM correctly rewrites all sentences without missing any key information. However, in the second instance, the LLM does not remove the context dependency for the second sentence. This also leads to the error in the third sentence, where the keyword "airline" in the first sentence is miss- ing. In general, comparable finetuned models (GAZP + BERT) though there is still a big room for improvement. Improving LLMs' performance on this difficult task a challenging future work. We just complete the initial exploration.
Table 4: Performances of different previous approaches on Spider dev set and test set.
before.
Table 5, Table 6 and Table 7 shows the GPT- 3.5-turbo's performances on Spider-Syn, Spider- DK. and Spider-Realiste dev set. We use the CEoT database style and set n, nd 2. The experiment results show that our approach still comparable to finetuned models on Spider-Syn and Spider-Realistee datasets. On the Spider-DK dataset, our approach's ' EX score surpasses fine- tuned models. This is due to the wide range of domain knowledge stored in LLMs.
Approach 66.9
EM
EX
TS
Graphix-3B+PICARD
RESDSQLL3B++NatQQ1
69.1
76.9
Few-shot (Ours)
47.2
63.7
54.5
ACT-SQL (Ours)
51.5
67.9
59.3
in-context learning method is
Table 5: Performances of GPT-3.5-turbo and other pre- vious works on Spider-Syn dev set.
Approach
EM
EX
TS
GrapHiX-3B+PICARD
51.2
RESDSQL-3B+NasSQI.
53.3
66.0
QM
IM
Approach
Few-shot (Ours)
50.7
68.8
61.7
EM
EX
TS
EM
EX
TS
ACT-SQL (Ours)
49.5
68.2
62.4
GAZP+BERT
42.0
38.8
RASAT+PCCARD
58.8
67.0
27.0
39.6
Few-shot (Ours)
45.4
63.9
55.5
14.7
29.4
22.9
Table 6: Performances of GPT-3.5-ture 3.5ttubb and other pre- vious works on Spider-DK dev set.
ACT-SQL (Ours)
46.0
63.7
55.2
13.3
30.7
21.5
Table 9: Performances of GPT-3.5-turbo and other pre-
vious works on CoSQL dev set.

---

Original questions:
these datasets can be conducted in future work.
I. What are all the flights that leave from Ab- erdeen?
Acknowledgements
2. Of those, which land in Ashley?
We thank all the anonymous reviewers for their thoughtful comments. This work has been supported by the China NSFC Project (No.62106142 and No.6212010600 Municipal Science and Technology Major Project (20IIHHZZZ00122) and Startup Fund for Young- man Research at SJTU (SFYR at SJTU).
How many are there?
Rewritten questions:
1. What are all the flights that depart from Ab- erdeen? v
Shanghai
2. What are the flights that depart from Aberdeen and land in Ashley? V
3. How many flights depart from Aherdeen and
land in Ashley?
Original questions:
What are all the airlines?
References
2. Of these, which Jetblue Airways?
Ben Bogin, Jonathan Berant, and Matt Gardner. 2019. Representing schema structure with graph neural net- works for text-to-SQL parsing. In Proceedings the 57th Annual Meeting of the Association for Computa- rional Linguistics, pages 4560-4565, Florence, Italy, Association for Computational Linguistics.
3. What the country corresponding it? Rewritten questions:
1. What are all the airlines? /
2. Which one of them is Jetblue Airways? x
3. What is the country corresponding to Jetblue Airways? x
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Table 10: Case study for rewritten questions from SParC
dev set.
5 Conclusion
LLMs have shown a strong ability in various do- mains with the in-context learning method. The lat- est studies have attempted to use LLMs to solve the text-to-SQL task. However, previous prompting ap- proaches either perform worse than finetuned mod- els or need to use LLMs' API call many times. We design the CoT prompt which can be automatically generated and propose our ACT-SQL approach. The ACT-SQL approach uses LLMs' API call only once generate one SQL query. The experiment results prove that our approach achieves state-of- the-art performance on the Spider dev set among existing in-context learning approaches. Further- more, we extend our approach to multi-turn text-(o- SQL. datasets.
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. LGESQL: Line graph enhanced text-to-SQI. model with mixed local and non-local relations. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 2541-2555, Online. Association for Computational Linguistics.
Ruisheng Cao. Lu Chen, Jieyu Li, Hanchong Zhang, Hongshen Xu, Wangyou Zhang, and Kai Yu. 2023. A heterogeneous graph abstract syntax tree frame- work for text-to-sql. IEEE Transactions on Patlern Analysis and Machine Intelligence, 45(11):13796 13813.
Limitations
Shuaichen Chang and Eric Fosler- 2023. How to prompt Ilms for text-to-sql: :: A study in zero-shot, single-domain, and cross-domain settings.
There are some limitations in our work. First of all, we use hybrid strategy for the exemplar selec- tion. The numbers of static and dynamic exemplars are hyperparameters and still need manually deter- mined. In addition, it is relatively simple strategy that still needs improvement. Furthermore, our approach achieves relatively poor scores on some robustness variants of the Spider dataset and some multi-turn text-to-SQL. datasets. Exploration of
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards. Yuri Burda. Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishknn, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter,

---

Philiippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert. Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021a. Evaluat- ing large language models trained on code.
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin. Chenhao Ma, Nan Huo. Fei Huang. Wenyu Du, Luo Si, and Yongbin Li. 2023c. Graphix-t5: Mixing pre- trained transformers with graph-aware layers for text- to-sql parsing.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev, 2023, Enhancing few-shot text-to- sql capabillitess of large language models: A study on prompt design strategies.
Long Quyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Zhi Chen, Lu Chen, Yanbin Zhao, Ruisheng Cao. Zi- han Xu, Su Zhu, and Kai Yu. 2021b. ShadowGNN: Graph projection neural network for text-to-SQL parser. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 5567-5577, Online. Association for Computational Linguistics.
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text- to-sql with self-correction.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov. Huan Sun, and Matthew Richardson. 2021. Structure- grounded pretraining for text-to-SQL. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1337-1350, Online. As- sociation for Computational Linguistics.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan. Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022. RASAT: Integrating relational structures into pretrained Seq2Seq model for text-to-SQL. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 22159229,, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Xuemei Dong, Chao Zhang, Yuhang Ge. Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023, C3: Zero-shot text-to-sql with chatgpt.
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Peng- sheng Huang. la Towards robustness of text- to-SQL models against synonym substitution. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 2505- 2515, Online. Association for Computational Lin- guistics.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabilities of large language models.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau, 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana. Domini- can Republic. Association for Computational Lin- guistics.
Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. Exploring underexplored limitations of cross- domain text-to-SQL. generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8926-8931, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578, Online. Association for Computational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.
Jieyu Li, Zhi Chen, Lu Chen, Zichen Zhu, Hanqi Li, Ruisheng Cao, and Kai Yu. 2023b. Dir: A large-scale dialogue rewrite dataset for cross-domain conversa- tional text-to-sql. Applied Sciences, 13(4).
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Bdd Chi, Quoe Le, and

---

Denny Zhou. 2023. ofain-of-thought prompting elic- its reasoning in large language models.
Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sql- net: Generating structured queries from natural lan- guage without reinforcement learning. arXiv preprint arXiv:1711.04433
Tao Yu, Rui Zhang. Heyang Er, Suyi Li, Eric Xue, Bo Pang. Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang. Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li. Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin- cent Zhang, Caiming Xiong, Richard Socher. Walter Lasecki, and Dragomir Radev. 2019a. CoSQL: A conversational text-to-SQL. challenge towards cross- domain natural language interfaces databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNNPP) pages 1962- 1979, Hong Kong, China. Association for Computa- tional Linguistics.
Tao Yu, Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingn- ing Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeles labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 391 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li. Heyang Er, Irene Li, Bo Pang. Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vin- cent Zhang. Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. SParC: Cross-domain se- mantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 451 1-4223, Florence, Italy. Association for Computational Linguistics.
Lu Zeng, Sree Hari Krishnan Parthasarathi, and Dilek Hakkani-Thu 2022. N-best hypotheses reranking for text-to-sql systems.
Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer: 2020. Grounded adaptation for shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6869- 6882, Online. Association for Computational Lin- guistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00100

---

A
Detailed Experiment Results
Few-shot
ACT-SQL
DB Style
Rd
EM
EX
TS
EM
EX
TS
0
4
58.0
77.5
68.8
58.3
78.9
69.2
Table (Column))
2
2
59.7
77.9
68.7
60.7
78.7
69.4
4
0
53.1
78.6
68.6
55.9
78.1
69.4
0
4
59.3
79.9
70.6
60.8
79.8
70.2
Table (Column) (PF)
2
2
60.1
79.1
69.3
62.3
79.1
71.0
4
0
54.8
80.0
70.3
57.0
77.9
69.6
0
4
58.8
78.2
68.8
58.8
79.8
68.7
Create (NoPF)
2
2
60.0
79.2
68.9
60.6
79.1
69.1
4
0
56.3
79.4
69.8
57.4
80.7
70.7
0
4
58.5
79.5
69.0
58.6
78.7
68.7
Create (EoC)
2
2
60.1
79,4
68.7
61.9
79.0
71.1
4
0
55.1
79.5
68.9
58.1
80.8
71.0
0
4
59.4
79.3
69.5
60.3
79.1
69.8
Create (EoT)
2
2
61.4
79.4
69.0
62.7
80.4
71.4
4
0
56.6
80.1
69.6
55.5
78.0
68.5
Table 11: Few-shot performances GPT-3.5-turbo with different DB styles and different numbers of static and dynamic exemplars Spider dev set.
LLM GPT-3.5-turbo
Approach ACT-SQL (Ours)
Easy
Medium
Hard
Extra
All
89.1
77.8
62.6
36.7
71.4
GPT-4
ACT-SQL (Ours)
91.1
79.4
67.2
44.0
74.5
Table 12: Detailed performances ACT-SQL on Spider dev set.
B
Auto-CoT by LLM
Question: Find the package choice and series name of the TV channel that has high definition TV.
Few-shot: SELECT _ame,, Hight_definition_TT Package_ Spttog FROM TV_Channel WHERE
Hight_defiittio_]1 'yes'
ACT-SQL: SELECT package_option, series name FROM TV Channel WHERE Hight definitienninnnnnn1 TV "yes"
Entire Output:
Let's think step by step.
According "TV channel that high definition TV", columns Chaanel..iightddefleee. TV] may be used.
According "package choice and series name". columns [TV__ Channel.Package. Pacptaaaaaaapptiiinn and [TV Channel.serie__nee _nmiees___me______ may be used.
So the final answer is:
SELECT package option, series name FROM TV Channel WHERE Hight definition TV "yes"
Table 13: Case study for ACT-SQL. The entire output is totally generated by the LLM with the ACT-SQL approach.
C Prompts
this section, we list detailed prompts used in our experiments.

---

C.1 Database Styles
C.1.1 Table(Column)
# stadium(Stadium ID, Location, Name, Capacity, Highest, Lowest, Average) # singer(Singer_Il) _D, Name, Country, Song_Nama Longgrelease_vee _eer,, Age, Is ls_male) # concert(concert ID, concert _ame, Theme, Stadium_ID, Year)
#
in jnn_conneet(conneett___ ID, Singer_ID)
C.1.2 Table(Column)(FF)
# stadium(Stadium ID, Location, Name, Capacity, Highest, Lowest, Average)
# singer(Singer ID, Name, Country, Song_ Name, Song_release year, Age, Is male)
# concert(concert ID, concert _amee Theme, Stadium_ ID, Year)
# srnger_jnn_coccrr(ee
ID, Singer_
# primary keys = cttrr_ii_cconeercconerrtt// # foreign keys Croncer.SStaiumm ID stadium. SwwddttttdimDDD.. singer in concert. concert.Singer. ID
MaadiumStadium_Ni ID,
singer.Singer ID,
concert.concert__ ID,
singer.Singer ID, singer in_ concert.concert ID concert.concert_ID)
C.1.3 Create(NoPF) create table stadium ( Stadium ID number, Location text,
Name text,
Capacity number,
Highest number,
Lowest number,
Average number
create table singer
Singer_ID ID number,
Name text,
Country text,
Song_ Name text,
Song release release_year year text,
Age number,
"s_mlee others
create table concert
concert ID number, Lancert_Namce text, Theme text, Stadium_ ID text,
Year text
create table singer_in in_ concert concert_ID number, Singer_ ID text
C.1.4 Create(EoC)
create table stadium (
Stadium ID number primary key, Location text,

---

Name text,
Capacity number,
Highest number,
Lowest number,
Average number
create table singer
Singer ID number primary key,
Name text,
Country text,
Song_Name text,
long_rrecase___eee year text, Age number, Is Is_male others
create table concert
concert ID number primary key,
concert Name text,
Theme text,
Stadium ID text references stadium(Stadium__u) Year text
create table singer_ la__jinccnnoa concert
concert ID number primary key references concert(concert_ID) Singer ID text references singer(Singer_ ID)
C.1.5
Create(EoT)
create table stadium ( Stadium ID number, Location text,
Name text,
Capacity number,
Highest number,
Lowest number,
Average number, primary key (Stadium_In _D)
create table singer
Singer ID number,
Name text, Country text,
Song_Name text,
Song_release__err year text, Age number,
Is male others. primary key (Singer_IID ID)
create table concert
concert ID number, concert_ text, Theme text,

---

Stadium ID text,
Year text,
primary key (concert_ID)
foreign key (Stadium_ ID) references stadium Stadium__I
create table singer_ ct__jincconne concert
concert_ ID number.
Singer_ID ID text,
primary key (concert_ ID),
foreign key (Singer_ID) [D) references singer(Singr__D) _D)),
foreign key (concert_ID) references concer((concert_I)
C.2 Database Contents
We only the Table(Column) and the (reate(EoT] database styles in the following prompt examples. The other three database styles are similar. The rows of database contents is set in the following prompt examples.
C.2.1 Table(Column)
# stadium(Stadium ID, Location, Name, Capacity, Highest, Lowest, Average)
/
3 example rows from table stadium:
Stadium ID
Location
Name
Capacity
Highest
Lowest
Average
Raith Rovers
Stark's Park
10104
4812
1294
2106
2
Ayr United
Somerset Park
11998
2363
1057
1477
3 East Fife
Bayview Stadium
2000
1980
533
864
**/
# singer(Singer ID, Name, Country, Song Name, Song_relcase,.0 Age, Is male)
/*
3 example rows from table singer:
Singer_ID
Name
Country
Song_Nane Name
Song_ release_ year
Age
Is Is_male
Joe Sharp
Netherlands
You 1992
52
2 Timbaland
United States
Dangerous
2008
32
3 Justin Brown
France
Hey Oh
2013
29
T
**/
# concert(concertt ID, concert_ Name, Theme, Stadium_ ID, Year)
/9
3 example rows from table concert:
concert ID
concert Name
Theme
Stadium ID
Year
Auditions
Free choice
2014
2 Super bootcamp
Free choice 2
2
2014
3 Home Visits
Bleeding Love
2015
**/
# singer_in li__iinnnoooa in_concert(coocenttt ID, Singer_ID) /
3 example rows from table singer in_ concert::
concert ID
Singer_ID
#8/

---

C.2.2 T
create table stadium (
Stadium ID number, Location text,
Name text.
Capacity number,
Highest number,
Lowest number,
Average number, primary key (Stadium_Ji) ID)
f
3 example from table stadium:
Stadium ID
Location
Name
Capacity 10104
Highest
Lowest
Average
Raith Rovers
Stark's Park
4812
1294
2106
2
Ayr United
Somerset Park
11998
2363
1057
1477
3
East Fife
Bayview Stadium
2000
1980
533
864
#*/
create table singer
Singer_ID number,
Name text,
Country text,
Song_ _ame text,
Song_release _ear text, Age number,
Is_male others,
primary key (Singer_ID)
f
3 example from table singer:
Singer_ID
Name
Country
Song Name
Song release_ year
Age
Is male
Joe Sharp
Netherlands
You
1992
2
Timbaland
United States
Dangerous
2008
32
3
Justin Brown
France
Hey Oh
2013
29
T
#*/
create table concert
concert_ ID number,
concert Name text,
Theme text, Stadium ID text,
Year text,
primary key (concert_ID),
foreign key (Stadium_ID) references (aadiuu(Stadium,.
/*
3 example rows from table concert:
concert_ID
concert_Nan Name
Theme
Stadium_ID
Year
Auditions
Free choice
2014
2
Super bootcamp
Free choice 2
2
2014
3
Home Visits
Bleeding Love
2015
**/

---

create table singer in concert concert ID number, Singer ID text,
primary key (concert_ID))
foreign key (Singer_ID) references singer(Singer_ID) foreign key (concert_ID) ID) references concer(cconeert ID)
f
3 example rows from table singer_in concert: iencerrr__concrt::
concert ID
Singer_ID ID
#*/
C.3 In-context Learning Prompts
We only use the Create(EoT) database styles in the following prompt examples. The other four database styles are similar. The rows of database contents is set the following prompt examples. Under the few-shot setting, the first two shots are static exemplars and the last two shots are dynamic exemplars.
C.3.1 Zero-shot
role: system
content:
Given the database schema, you need translate the question into the SQL query.
role: user
content:
Database schern::
create table stadium (
Stadium ID number,
Location text,
Name text,
Capacity number,
Highest number,
Lowest number,
Average number, primary key (Stadium_ID _D)
/9
3 example rows from table stadium:
Stadium ID
Location
Name
Capacity 10104
Highest
Lowest
Average
Raith Rovers
Stark's Park
4812
1294
2106
2
Ayr United
Somerset Park
11998
2363
1057
1477
3
East Fife
Bayview Stadium
2000
1980
533
864
**/
create table singer
Singer_ ID number,
Name text,
Country text,
Song_Name text,
text,
Age number,

---

Is s__male others.
primary key (Singer [Dn
3 example rows from table singer:
Singer
ID
Name
Country
Song_Name
Song_ relcase_yceec _eaar
Age
Is ssmaae
Joe Sharp
Netherlands
You
1992
52
F
2
Timbaland
United States
Dangerous
2008
32
T
3
Justin Brown
France
Hey Oh
2013
29
T
#*/
create table concert
concert ID number,
concert_Namee text,
Theme text,
Stadium ID text,
Year text,
primary key (concert_ ID),
foreign key (Stadium ID) references Kadium(Stadium ID)
3 example rows from table concert:
concert_ID
concert Concert_Name Free choice Free choice 2
Theme
Stadium _D
Year
Auditions
2014
2
Super bootcamp
2
2014
3
Home Visits
Bleeding Love
2
2015
**/
create table singer_in trnger_in_concertt
concert
concert ID number,
Singer_ ID lext,
primary key (concert_ID) ID),
foreign key (Singer (D) references singer(Singer_Ju)
foreign key (concert_ID) references concert(concert_ ID)
/
3 example rows from table singer_in concert: incoc_ir_cconert::
concert ID
Singer ID
#*/
Question: How many singers do we have?
C.3.2 Few-shot
role: system
content:
Given the database schema, you need to translate the question into the SQL query.
role: user
content:
Datahase schema:
create table ACCOUNTS custid number,

---

name text,
primary key (custid)
3 example rows from table ACCOUNTS:
custid
name
Вrown
2
Wang
3
O' O'mahony
#*/
create table SAVINGS
custid number, balance number, primary key (custid), foreign key (custid) references ACCOUNTS(custid)
y
3 example rows from table SAVINGS:
custid
balance
200000.0
2
999999999.0
3
230000.0
#*/
create table CHECKING custid number, balance number, primary key (custid), foreign key (custid) references ACCoUNTS(custid)
3 example rows from table CHECKING:
custid
balance
10000.0
2
2000.0
3
3000.0
#*/
Question: Find the name and savings balance the top accounts with the highest saving balance sorted by savings balance in descending order.
role: assistant
content:
SELECT TI Lname, T2lbalance FROM accounts AS TI JOIN savings AS T2 ON TLccssidd T22custid ORDER BY balaace DESC LIMIT 3
role: user
content:
Database schema: create table flight fino number,
origin text, destination text,

---

distance number, departure date time. arrival_date date time, price number, aid number, primary key (fino), foreign key (aid) references aircraft(aid)
/*
3 example rows from table flight:
fino
origin
destination
distance
departure date
arrival date
price
aid
99
Los Angeles
Washington D.C.
2308
04/12/2005 09:30
04/12/2005 09:40
235.98
13
Los Angeles
Chicago
1749
04/12/2005 08:45
04/12/2005 08:45
220.98
3 Los Angeles
346
Dallas
1251
04/12/2005 11:50
04/12/2005 07:05
182
#*/
create table aircraft ( aid number, name text, distance number, primary key (aid)
/*
3 example rows from table aircraft:
aid
name
distance
Bocing 747-400
8430
2
Boeing 737-800
3383
3
Airbus A340-300
7120
*a/
create table employee eid number, name text, salary number, primary key (eid)
/*
3 example rows from table employee:
eid
name
salary
242518965
James Smith
120433
141582651
Mary Johnson
178345
11564812 **/
John Williams
153972
create table certificate
eid number, aid number, primary key (cid), foreign key (aid) references aircraft(aid), foreign key (eid) references employee(eid)

---

3 example rows from table certificate:
eid
aid
11564812
2
11564812
10
90873519
6
#*/
Question: Which destination has least number of flights?
role: assistant
content: SELECT destination FROM Flight GROUP BY destination ORDER BY count +) LIMIT
role: user
content:
Database schema: create table Activity actid number, activity _ame text, primary key (actid)
y
3 example rows from table Activity:
actid
activity_nand __me
770
Mountain Climbing
771
Canoeing
772
Kayaking
#*/
create table Participates_in
stuid number,
actid number,
foreign key (actid) references Activity(actid), foreign key (stuid) references Student(StulD)
y
3 example rows from table Participates_in in:
stuid
actid
1001
770
1001
771
1001
777
#*/
create table Ceculty_Partccipatter
FaeID number,
actid number, foreign key (actid) references Activity(actidd, foreign key (FacID) references FaceltyFFaeID)
/
3 example rows from table Faculty_Participatee Participattstss... in: FacID
actid

---

1082
784
1082
785
1082
790
##/
create table Student
StulD number,
LName text,
Fname text,
Age number,
Sex text,
Major number,
Advisor number,
city_code text, primary key (StulD)
/
3 example rows from table Student:
StulD
LName
Fname
Age
Sex
Major
Advisor
city_code
1001
Smith
Linda
18
600
1121
BAL
1002
Kim
Tracy
600
7712
HKG
1003
Jones
Shiela
600
7792
WAS
**/
create table Faculty
FacID number,
Lname text,
Fname text,
Rank text,
Sex text,
Phone number,
Room text,
Building text,
primary key (FacID)
/
3 example rows from table Faculty:
FacID
Lname
Fname
Rank Instructor
Sex
Phone
Room
Building
1082
Giuliano
Mark
2424
224
NEB
1121
Goodrich
Michael
Professor
M
3593
219
NEB
1148
Masson
Gerald
Professor
M
3402
224B
NEB
#*/
Question: How many female Professors do we have?
role: assistant
content:
SELECT count(*) FROM Faculty WHERE Sex 'F' AND Rank "Professor"
role: user
content:
Database schema:
create table region (
Region_ ID number,

---

Region_ name Regi
text,
Date text,
Label text,
Format text,
Catalogue text,
primary key (Region_Il)
f
3 example from table region:
Region ID
Region name
Date
Label
Format
Catalogue
United Kingdom
July 2002
Parlophone
CD
540 3622
2
United Kingdom
July 2002
Parlophone
LP
539 9821
3
Japan
July 2002
Toshiba-EMI
CD
TOCP-66045
#*/
create table party
Party_ID number,
Minister text,
-ofkcof text,
office text,
Region_ ID number, Party_ Party_name name text, primary key (Party_IDD)
foreign key (Region_In) ID) references region(Region ID)
f
3 example rows from table party:
Party_ __
Minister
Took_ office
Region
Party name
Dr. Kwame Nkrumah (MP)
1957
1958
Convention Peoples Party
2
Kojo Botsio (MP)
1958
1959
Progress Party
3
Ebenezer Ako-Adjes (MP)
1959
1960
3
3
#*/
create table member
Member_ID number,
Member_Name text,
Party ID text,
In office text, primary key (Member_ID)
foreign key (Party ID) references party(Party_Iu) ID)
/
3 example rows from table member:
Member ID
Member Name
Party_ ID
In office
Hon Tony Abbott
3
1994-present
2
Hon Dick Adams
2
1993-2013
3
Anthony Albanese
2
1996-present
**/
create table party_events
Event ID number,
Event Name text,
Party_ ID number,
Member in charge_ ID number,

---

primary key (Bvent_ID))
foreign key (Member in charge_ ID) references member(Member,__. ID).
foreign key (Party_ID) references party(Party_ID)
/
3 example rows from table party_events:
Event_ID
Event Name
Party_ID
Member_in inlliincchha charge_
Annaual Meeting
1
4
2
Conference
12
3
Annaual Meeting
2
2
**/
Question:
How many parties do we have?
role: assistant
content:
SELECT count(DISTINCT party_name) FROM party
role: user
content:
Database schema:
create table stadium (
Stadium _D number,
Location text,
Name text,
Capacity number,
Highest number,
Lowest number.
Average number, primary key (Stadium ID)
3 example rows from table stadium:
Stadium_ID _I
Location
Name
Capacity 10104
Highest
Lowest
Average
Raith Rovers
Stark's Park
4812
1294
2106
2
Ayr United
Somerset Park
11998
2363
1057
1477
3
East Fife
Bayview Stadium
2000
1980
533
864
#*/
create table singer
Singer_ID ID number,
Name text, Country text,
Long_Name text,
Song_release__eee text, Age number, Is ms_e others, primary key (Singer_ID)
/
3 example rows from table singer:
Singer_ID Name Country
Song_
Song release_year
Age
Is_ Is_male
Joe Sharp
Netherlands
You
1992
52
F

---

2
Timbaland
United States
Dangerous
2008
32
T
3
Justin Brown
France
Hey Oh
2013
29
T
**/
create table concert
concert_In [_D number,
concert _ame text,
Theme text,
Stadium ID text,
Year text,
primary key (concert_ID) ID),
foreign key (Stadium_IDD ID) references stadium(Stadium __)
y
3 example rows from table concert:
concert_ID
TancertNNane Free choice
Theme
Stadium_Iu ID
Year
Auditions
2014
2
Super bootcamp
Free choice 2
2
2014
3
Home Visits
Bleeding Love
2015
#*/
create table singer_in in__cnccertttttteeeeeee
concert ID number,
Singer_ID ID text,
primary key (concert ID), foreign key (Singer_ ID) references singer(Singer_ID) [D),
foreign key (concert_ID) references concert(coneert_ ID)
/*
3 example rows from table
concert ID
Singer_ID ID
5
**/
Question: How many singers do have?
C.3.3 Auto-CoT
role: system
content:
Given the database schema, you need to translate the question into the SQL query.
role: user
content:
Database schema:
create table ACCOUNTS custid number,
name text,
primary key (custid)
/*
3 example rows from table ACCOUNTS:
custid name
Brown

---

2
Wang
3
O' O'mahony
**/
create table SAVINGS
custid number, balance number,
primary key (custid),
foreign key (custid) references ACCOUNTS(custii)
/
3 example rows from table SAVINGS:
custid
balance
200000.0
2
999999999.0
3
230000.0
**/
create table CHECKING custid number, balance number,
primary key (custid).
foreign key (custid) references ACCOUNTS(custid)
3 example rows from table CHECKING:
custid
balance
10000.0
2
2000.0
3
3000.0
#*/
Question: Find the name and savings balance the top accounts with the highest saving balance sorted by savings balance in descending order.
role: assistant
content:
Let's think step by step.
According to "savings balance", columns [SAVINGS balance] may be used.
According to "accounts". columns (ACCOUNTS.nam)) name) may be used.
Values [3] may be used.
So the final answer is: SELECT T1 nnaee balanee FROM accounts AS TI JOIN savings AS T2 ON Tl.custid T2.custid ORDER BY balanee DESC LIMIT 3
role: user
content:
Datahase schema: create table flight fino number, origin text, destination text, distance number, departure_ date time,

---

arrival_date time,
price number, aid number,
primary key (fino), foreign key (aid) references aircraft(aid)
3 example rows from table flight: origin
fino
destination
distance
departure_ date
arrival date
price
aid
99
Los Angeles
Washington D.C.
2308
04/12/2005 09:30
04/12/2005 09:40
235.98
13
Los Angeles
Chicago
1749
04/12/2005 08:45
04/12/2005 08:45
220.98
3 Los Angeles
346
Dallas
1251
04/12/2005 11:50
04/12/2005 07:05
182
2
#*/
create table aircraft aid number, name text, distance number, primary key (aid)
/*
3 example rows from table aircraft:
aid
name
distance
Boeing 747-400
8430
2
Boeing 737-800
3383
3
Airbus A340-300
7120
#a/
create table employee eid number, name text, salary number, primary key (eid)
f*
3 example rows from table employee:
eid
name
salary James Smith
242518965
120433
141582651
Mary Johnson
178345
11564812
John Williams
153972
**/
create table certificate
eid number, aid number, primary key (eid). foreign key (aid) references aircraft(aid), foreign key (cid) references employce(cid)

---

3 example rows from table certificate:
eid
aid
11564812
2
1 11564812
10
90873519
#*/
Question: Which destination has least number of flights?
role: assistant
content:
Let's think step by step.
According to "flights". columns [flight.destinationn] may be used.
Values [1] may be used.
So the final answer is: SELECT destination FROM Flight GROUP BY destination ORDER BY count(*) LIMIT
role: user
content:
Database schema:
create table Student
StuID number,
LName text,
Fname text,
Age number,
Sex text,
Major number,
Advisor number,
city_code text, primary key (StuID)
3 example rows from table Student:
StulD
LName
Fname
Age
Sex
Major
Advisor
city_code
1001
Smith
Linda
18
F
600
1121
BAL
1002
Kim
Tracy
19
F
600
7712
HKG
1003
Jones
Shiela
21
600
7792
WAS
#*/
create table Dorm
dormid number,
dorm_name name text,
student_cappan catppttccccyyyyaaaeaae number, gender text
/*
3 example rows from table Dorm:
dormid
dorm_name
student__cpacity
gender
100
Smith Hall
85
110
Bud Jones Hall
116
M
140
Fawlty Towers
355
X
#*/
create table Dorm amenity

---

amenid number,
amenity name text
3 example rows from table Dorm_amentn
amenid
aaecittynnamp
900
TV Lounge
901
Study Room
902
Pub in Basement
#*/
create table Has_ aas_amenity& (
dormid number, amenid number,
foreign key (amenid) references Dorm_ Dorm_amenity( (anaamimitt(((eni)))
foreign key (dormid) references Dorm(dormid)
y
3 example rows from table Has amenity:
dormid
amenid
109
900
109
901
109
903
#*/
create table Lives in (
stuid number, dormid number, room_number number, foreign key (dormid) references Dorm(dormid), forcign key (stuid) references Student(StulD)
3 example rows from table Lives_in in:
stuid
dormid
room room_number
1001
109
105
1002
100
112
1003
100
124
#*/
Question: How many dorms have amenities?
role: assistant
content:
Let's think step by step.
According to "dorms have amenities ?". columns [Has_ [llss_aieniy may be used.
So the final answer is:
SELECT count(DISTINCT dormid) FROM has has has_amynity
role: user
content:
Database schema:
create table Student
StulD number,

---

LName text,
Fname text, Age number, Sex text, Major number, Advisor number, city_code code text, primary key (StulD)
/
3 example rows from table Student:
StuID
LName
Fname
Age
Sex
Major
Advisor
city_code
1001
Smith
Linda
18
600
1121
BAL
1002
Kim
Tracy
19
F
600
7712
HKG
1003
Jones
Shiela
21
F
600
7792
WAS
**/
create table Dorm
dormid number,
dorm name text, student_ cacrentt_aacittytyteee number, gender text
y
3 example rows from table Dorm:
dormid
dorm, name
student student_capacity
gender
100
Smith Hall
85
X
110
Bud Jones Hall
116
M
140
Fawlty Towers
355
X
#*/
create table Dorm S
amenid number,
amenity_nan _ame text
y
3 example rows from table Dorm_ant _mmenit:::
amenid
amenity name
900
TV Lounge
901
Study Room
902
Pub in Basement
**/
create table Has _as_amtttyy ( dormid number,
amenid number, foreign key (amenid) references Dorm Dorm_ amenity(amenid),
foreign key (dormid) references Dorm(dormid)
y
3 example rows from table Has amenity:
dormid
amenid
109
900
109
901

---

109
903
#8/
create table Lives in ( stuid number, dormid number, room _ou__numeerrreeeeeeeeeeeeeeeeeer number, foreign key (dormid) references Dorm(dormid), foreign key (stuid) references Student(StulD)
3 example rows from table Lives_im in:
stuid
dormid
room comm_number
1001
109
105
1002
100
112
1003
100
124
**/
Question: How many diffrent dorm amenities there?
role: assistant
content:
Let's think step by step.
According to "dorm amenities are there ?", tables (Dorm_amenine may used. So the final answer is:
SELECT count(*) FROM dorm_amenity
role: user
content:
Datahase schema:
create table stadium (
Stadium _D number,
Location text,
Name text,
Capacity number,
Highest number,
Lowest number,
Average number, primary key (Stadium_Ii ID)
/
3 example rows from table stadium:
Stadium_ID ID
Location
Name
Capacity 10104
Highest
Lowest
Average
Raith Rovers
Stark's Park
4812
1294
2106
2
Ayr United
Somerset Park
11998
2363
1057
1477
3
East Fife
Bayview Stadium
2000
1980
533
864
*ay
create table singer
Singer_ID ID number,
Name text,
Country text,
Name text,
Song_release__eat text,

---

Age number,
Is male others.
primary key (Singer_ID)
/
3 example rows from table singer:
Singer_ID
Name
Country
Song Name
Song release_
Age
Is ms_male
Joe Sharp
Netherlands
You
1992
52
F
2
Timbaland
United States
Dangerous
2008
32
T
3
Justin Brown
France
Hey Oh
2013
29
T
**/
create table concert
concert_ ID number,
concert_Name text,
Theme text,
Stadium_ ID text,
Year text,
primary key (concert ID),
foreign key (Stadium ID) references Maddium(Stadiuua ID)
3 example rows from table concert:
concert_ip ID
concert -ame Free choice
Theme
Stadium ID
Year
Auditions
2014
2
Super bootcamp
Free choice 2
2
2014
3
Home Visits
Bleeding Love
2015
#*/
create table singer_in ienee__rn_concetttt
concert ID number,
Singer_ID ID text,
primary key (concert_II))
foreign key (Singer_ID) ID) references singer(Singer_ID)
foreign key (DD references concert(concert_LD)
/*
3 example rows from table singer_in, ii__iinnnooo concert:
concert ID
Singer_
**/
Question: How many singers do we have?


Reviews:
[{"review_id": "EPrEoedLFz", "reviewer": "Reviewer_5VBy", "summary": "", "questions": "1. Are table linking from line 325 to line 327 robust? Considering that some of the tables in JOIN part only serve as bridge function (connect two tables with foreign keys), the question may not explicitly mention those tables. In this case, enforcing the linking between tables and utterance chunks may not be reasonable.\n2. I found that mixing randomly selected exemplars and test-case-specific exemplars yields the best performance. Any insights on this? Why didn’t more test-case-specific exemplars improve the performance?\nAny evidence to show the claim from line 534-538 (in-context learning alleviate the performance gap issue between dev set and test set)\n3. Comments: In line 570, the authors claim that the proposed in-context learning method is comparable to finetuned models. I think the authors refer to GAZP+BERT, because RASAT+PICARD is much higher than in-context learning. However, in terms of model size, GAZP+BERT is much smaller than GPT-3.5. The author should make careful claims by providing accurate assumptions.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions.\n\n**Q1:** Are table linking robust?\n\n**A1:**\n\n**a)** In most cases, table names of bridge tables have semantics and can be mentioned in the question. Consider the following example. The bridge table \"student_course\" can still be mentioned in the question.\n\n**Question:** What courses does the student named Tom take?\n\n**SQL:** SELECT T3.course_name FROM student as T1 JOIN student_course as T2 JOIN course as T3 ON T1.student_id = T2.student_id AND T2.course_id = T3.course_id WHERE T1.student_name = \"Tom\"\n\n**b)** Since exemplars come from train set, their schema linking processes are completed with the rule-based method discussed in our paper. These exemplars are enforced to link tables with question chunks, which may lead to wrong schema linking. The following example shows the wrong table linking.\n\n**Database schema:**\n\ncreate table college (\n\n$\\quad$id text,\n\n$\\quad$name text,\n\n$\\quad$primary key (id)\n\n)\n\ncreate table department (\n\n$\\quad$id text,\n\n$\\quad$name text,\n\n$\\quad$cid text,\n\n$\\quad$primary key (id),\n\n$\\quad$foreign key (cid) references college(id)\n\n)\n\ncreate table student (\n\n$\\quad$id text,\n\n$\\quad$name text,\n\n$\\quad$did text,\n\n$\\quad$primary key (id),\n\n$\\quad$foreign key (did) references department(did)\n\n)\n\n**Question:** List all student names from the college named XXX.\n\n**CoT:**\n\nLet's think step by step.\n\nAccording to \"student names\", columns [student.name] may be used.\n\nAccording to \"college named\", columns [college.name] may be used.\n\nAccording to \"from\", tables [department] may be used.\n\nValues [XXX] may be used.\n\nSo the final answer is:\n\nSELECT T3.name FROM college as T1 JOIN department as T2 JOIN student as T3 ON T1.id = T2.cid AND T2.id = T3.did WHERE T1.name = \"XXX\"\n\nThe linking between the bridge table \"college\" and the question word \"from\" is wrong. However, during the testing phase, LLMs would generate the CoT and complete the inference for the testing case. Although wrong table linking may appear in exemplars, LLMs like GPT have the ability to avoid unreasonable schema linking. Following is the example completed by GPT where we add the wrong exemplar mentioned above.\n\n**Database schema:**\n\ncreate table continent (\n\n$\\quad$name text,\n\n$\\quad$primary key (name)\n\n)\n\ncreate table country (\n\n$\\quad$name text,\n\n$\\quad$continent_name text,\n\n$\\quad$primary key (name),\n\n$\\quad$foreign key (continent_name) references continent(name)\n\n)\n\ncreate table singer (\n\n$\\quad$id text,\n\n$\\quad$name text,\n\n$\\quad$country_name text,\n\n$\\quad$foreign key (country_name) references country(name)\n\n)\n\n**Question:** What are singer names from Asia?\n\n**CoT:**\n\nLet's think step by step.\n\nAccording to \"singer names\", columns [singer.name] may be used.\n\nAccording to \"from Asia\", columns [continent.name] may be used.\n\nAccording to \"singer names from\", tables [singer] may be used.\n\nValues [Asia] may be used.\n\nSo the final answer is:\n\nSELECT name FROM singer WHERE country_name IN (SELECT name FROM country WHERE continent_name = 'Asia')\n\n**Q2:** Why mixing exemplars but not more test-case-specific exemplars?\n\n**A2:**\n\n**a)** The previous work [1] has shown that exemplar selection strategy should balance diversity and similarity. The diversity would suffer if more test-case-specific exemplars are selected.\n\n**b)** Diversity is important for exemplar selection strategy. With different DB schema, similar questions can correspond to different SQL queries. Diverse exemplars can provide different SQL patterns.\n\n**Q3:** Does in-context learning alleviate the performance gap between dev set and test set?\n\n**A3:** Yes.\n\n**a)** The following table shows some previous works' performances on Spider dev set and test set. For finetuned approaches like RESDSQL-3B + NatSQL and Graphix-3B + PICARD, the performances both drop from dev set to test set. On the contrary, for in-context learning approaches like DIN-SQL and C3, the performances both increase from dev set to test set.\n\n**b)** After all, finetuned models are selected by the dev set performance, which would lead to the overfitting on dev set and the performance dropping on test set. For in-context learning approaches, dev set and test set are equal to the model. Performances between dev set and test set are only affected by the dataset feature.\n\n|      Previous Work      |     Method Type     | Spider Dev EX | Spider Test Ex |\n| :---------------------: | :-----------------: | :-----------: | :------------: |\n| RESDSQL-3B + NatSQL [2] | finetuned approach  |     84.1      |      79.9      |\n|  Graphix-3B+PICARD [3]  | finetuned approach  |     81.0      |      77.6      |\n|       DIN-SQL [4]       | in-context learning |     82.8      |      85.3      |\n|         C3 [5]          | in-context learning |     81.8      |      82.3      |\n\n**Q4:** ACT-SQL is comparable with GAZP+BERT but not RASAT+PICARD.\n\n**A4:** Sorry, we didn't make a clear statement in our paper. ACT-SQL is comparable with GAZP+BERT ($\\approx$ 0.2B params) but not RASAT+PICARD ($\\approx$ 3B params). Experiment results show that LLMs have the ability to handle the multi-turn text-to-SQL task. However, there is still a large room between our method and the SOTA model. Improving LLMs' performance on this difficult task is a challenging future work. We just complete the initial exploration.\n\n[1] Nan et al. 2023. Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies.\n\n[2] Li et al. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL.\n\n[3] Li et al. 2023. Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing.\n\n[4] Pourreza and Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction.\n\n[5] Dong et al. 2023. C3: Zero-shot Text-to-SQL with ChatGPT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Auto-generate Chain-of-Thought prompt is cost-saving.\n2. By combining traditional query rewriting strategy, the proposed method can be easily extended to multi-turn text-to-SQL task. \n3. Incremental contributions on studying prompting style and hybrid exemplar selection strategy.\n", "reasons_to_reject": "1. In terms of schema linking based chain-of-thought prompt design, there is no significant difference from (Pourreza and Rafiei, 2023). \n2. More on questions part.\n", "questions_to_authors": "1. Are table linking from line 325 to line 327 robust? Considering that some of the tables in JOIN part only serve as bridge function (connect two tables with foreign keys), the question may not explicitly mention those tables. In this case, enforcing the linking between tables and utterance chunks may not be reasonable.\n2. I found that mixing randomly selected exemplars and test-case-specific exemplars yields the best performance. Any insights on this? Why didn’t more test-case-specific exemplars improve the performance?\nAny evidence to show the claim from line 534-538 (in-context learning alleviate the performance gap issue between dev set and test set)\n3. Comments: In line 570, the authors claim that the proposed in-context learning method is comparable to finetuned models. I think the authors refer to GAZP+BERT, because RASAT+PICARD is much higher than in-context learning. However, in terms of model size, GAZP+BERT is much smaller than GPT-3.5. The author should make careful claims by providing accurate assumptions.\n", "paper_topic_and_main_contributions": "In this work, the authors proposed ACT-SQL that generates the Chain-of-Thought automatically for the training set. These examples are beneficial for the in-context learning. Only one LLM API call is required for this method compared to DIN-SQL’s 4 API calls. This method can be extended to multi-turn Text-to-SQL problems. To mitigate multi-turn text-to-SQL, the LLM is leveraged to translate the sequence of utterances into a single question to facilitate schema linking. The experiments have shown that the proposed method achieved state-of-the-art performance on Spider dataset under the in-context learning category.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 439 lower case `We`."}, "extra_text": {}}, {"review_id": "q7jRqVAevr", "reviewer": "Reviewer_sUqE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions.\n\n**Q1:** What is the novelty of this research?\n\n**A1:**\n\n**a)** The previous work DIN-SQL [1] has applied the chain-of-thought (CoT) method for text-to-SQL. However, DIN-SQL needs manually labeled exemplars. Manually labeled CoT has some disadvantages. First, it is cost-consuming and time-consuming. Second, it needs domain experts to design and write the CoT. Third, it is unfriendly for exemplar selection strategy, since dynamic exemplars lack CoTs unless the whole train set is labeled by human. We propose the method to automatically generate the CoT for the given exemplar, which is cost-saving and time-saving.\n\n**b)** When handling one testing case, ACT-SQL calls LLMs' API only once while DIN-SQL calls LLMs' API 4 times. Our approach is faster and cheaper. In real-world scenarios and applications, users are always very sensitive to their user experience. Therefore, optimizing the response time is an important novelty of our research. Our approach decreases the total response time for handling one user question. Thus real-world users would not be impatient when waiting for the response.\n\n**c)** Under the same LLM, we achieve the SOTA performance on Spider dev set with the in-context learning method. The TS score is 71.4 when using GPT-3.5-turbo. The TS score is 74.5 when using GPT-4. Furthermore, to the best of our knowledge, we are also the first to apply the in-context learning method on the multi-turn text-to-SQL task.\n\n**Q2:** Comparison with similar works?\n\n**A2:** Actually Table 3 in our paper has shown the comparison with related similar works. The following table shows the detailed comparison. Comparing with [2] and [3], ACT-SQL achieves the much better performance on Spider dev set since the CoT method is introduced. Comparing with DIN-SQL [1], ACT-SQL performs a bit better than DIN-SQL and uses less LLMs' API call, i.e. our approach is more effective and efficient. Furthermore, our approach can automatically generate the CoT for the given exemplar, which avoids the cost-consuming human labeling by domain experts.\n\n|           Approach            | Spider Dev EX/TS | # API Calls | Automatically | CoT  |\n| :---------------------------: | :--------------: | :---------: | :-----------: | :--: |\n|      Rajkumar et al. [2]      |    67.0/55.1     |      1      |      yes      |  no  |\n| Chang and Fosler-Lussier. [3] |      76.8/-      |      1      |      yes      |  no  |\n|          DIN-SQL [1]          |    82.8/74.2     |      4      |      no       | yes  |\n|        ACT-SQL (Ours)         |    82.9/74.5     |      1      |      yes      | yes  |\n\n**Q3:** The extent or significance of the performance improvement is not made clear.\n\n**A3:**\n\n**a)** Comparing with previous works, we achieve a better performance with less LLMs' API call. This indicates that we propose a approach to prompt GPT for the text-to-SQL task in a better, faster and cheaper way.\n\n**b)** We also complete some more experiments with several different random seeds and do the statistical significance test to reveal the performance improvement from the ordinary few-shot method to our ACT-SQL. The p-value in the null hypothesis when comparing few-shot and ACT-SQL in EM/EX/TS is 0.004/0.048/0.016. These p-values are lower than 0.05. Thus our approach is effective.\n\n**Q4:** No clear differentiation from existing works.\n\n**A4:** The novelty of our research and the comparison with similar works are stated in A1 and A2.\n\n[1] Pourreza and Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction.\n\n[2] Rajkumar et al. 2022. Evaluating the Text-to-SQL Capabilities of Large Language Models.\n\n[3] Chang and Fosler-Lussier. 2023. How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces chain-of-thought (CoT) prompt design, inspired by schema linking, represents a fresh perspective in the text-to-SQL task.\n2. The ACT-SQL method for automatically generating auto-CoT exemplars without manual labeling streamlines the process and enhances efficiency.\n3. The application of the approach to multi-turn text-to-SQL tasks demonstrates its versatility and potential adaptability to more complex dialog scenarios.\n4. By attaining SOTA performance on the Spider dev set among existing in-context learning approaches, the paper showcases the effectiveness of the proposed method.", "reasons_to_reject": "1. While the paper introduces the chain-of-thought (CoT) prompt design and in-context learning techniques, it does not appear to be the first work to utilize these concepts in the text-to-SQL task. Many similar works in the public domain have explored these ideas, leading to difficulty in distinguishing the novelty of this particular research.\n2. The paper seems to lack a comprehensive comparison with related similar works that have used CoT or in-context learning techniques. A detailed analysis of how the proposed method differs from or improves upon existing approaches would enhance the paper's credibility and relevance.\n3. While the paper claims state-of-the-art performance on the Spider dev set, the extent or significance of this performance improvement is not made clear. A more transparent presentation of the performance metrics and comparison with benchmarks would provide a robust validation of the claimed achievements.\n4. Without clear differentiation from existing works and a detailed comparison with similar techniques, the paper's contributions may be perceived as incremental or lacking substantial novelty. This could affect the reception and perceived value of the work within the NLP community.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on the challenge of prompt designing in the text-to-SQL task, aiming to enhance LLMs' reasoning capabilities when generating SQL queries. The authors introduce a method called ACT-SQL to automatically generate auto-CoT exemplars without manual labeling, making the process cost-saving. The approach is extended to multi-turn text-to-SQL tasks, and experimental results demonstrate that ACT-SQL achieves state-of-the-art performance on the Spider dev set among existing in-context learning approaches.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UzceZYB7VO", "reviewer": "Reviewer_tJPB", "summary": "", "questions": "1. See above for the statistical significance test. What's the p-value in the null hypothesis when comparing Few-shot and ACT-SQL in EM/EX/TS?\n2. The authors use a neural model + rule-based method to create chain-of-thought reasoning steps for demonstrations. How does it compare to the chain-of-thought reasoning steps obtained by using \"let's think step by step\" to prompt ChatGPT. [1]\n\n[1] Kojima, Takeshi, et al. \"Large language models are zero-shot reasoners.\" Advances in neural information processing systems 35 (2022): 22199-22213.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions.\n\n**Q1:** Statistical significance test between Few-shot and ACT-SQL?\n\n**A1:** We complete some more experiments with several different random seeds and then compute the p-value. Under the Spider dev set, the p-value in EM/EX/TS is 0.004/0.048/0.016. These p-values are lower than 0.05.\n\n**Q2:** How about using \"let's think step by step\" to prompt GPT?\n\n**A2:**\n\n**a)** Actually according to Figure 1 and Figure 2 in our paper, we have added \"let's think step by step\" into the prompt.\n\n**b)** If we only use \"let's think step by step\" to prompt GPT without any exemplar, the performance would definitely decrease, since exemplars play a very important role for the in-context learning. We attempt to do the experiment with GPT-3.5-turbo. The TS score is only 59.4, which is lower than ACT-SQL TS score 71.4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author presents a comprehensive comparison of various database prompt styles in the zero-shot as well as a comparison between using trivial few-shot examples and using ACT few-shot examples with different database prompt styles. \n2. The experiments cover multiple single-turn and multi-turn text-to-SQL datasets, where the proposed ACT-SQL outperforms the baseline Few-shot method in a majority of these datasets.\n3. The paper is well-written and easy to follow.", "reasons_to_reject": "1. My biggest concern is the effectiveness of the proposed ACT-SQL. In Table 10, the author presents a comparison between using trivial few-shot examples and using ACT few-shot examples with different database prompt styles. It seems the difference between Few-shot and ACT-SQL is very small (<1% for most cases) for Table(Column), Table(Column) PF, Create(NoPF), and relatively large (about 2%) for Create(EoC) and Create(EoT). Given that the Spider dev set only contains about 1K examples, it seems important to do a statistical significance test when comparing Few-shot and ACT-SQL. The same statistical significance test should be applied to Table 4,5,6,7,8 as the results of Few-shot and ACT-SQL are very close (ACT-SQL even underperforms Few-shot in Table 6,7,8 for TS accuracy).\nI believe addressing this concern during the rebuttal would be essential.", "questions_to_authors": "1. See above for the statistical significance test. What's the p-value in the null hypothesis when comparing Few-shot and ACT-SQL in EM/EX/TS?\n2. The authors use a neural model + rule-based method to create chain-of-thought reasoning steps for demonstrations. How does it compare to the chain-of-thought reasoning steps obtained by using \"let's think step by step\" to prompt ChatGPT. [1]\n\n[1] Kojima, Takeshi, et al. \"Large language models are zero-shot reasoners.\" Advances in neural information processing systems 35 (2022): 22199-22213.", "paper_topic_and_main_contributions": "The paper proposes to automatically generate the chain-of-of-thought reasoning steps for the text-to-SQL (ACT-SQL). The authors suggest using hybrid demonstration examples, which contains n_s static examples for all test case, and n_d dynamic examples selected based on specific test cases. Additionally, these demonstration examples are compounded with automatically generated chain-of-thought reasoning steps.\nThe paper compares different database prompt styles in the zero-shot and few-shot text-to-SQL.\nThe results demonstrate that automatically generated chain-of-thought steps benefit LLMs on text-to-SQL generation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a consensus among the reviewers that the proposed text-to-SQL method is worthwhile, the experiments are thorough and the paper is generally well written and clear. The reviewers had concerns on the novelty of this work with respect to the existing techniques (and the subsequent comparison with these) and statistical significance tests. The authors have provided detailed response to these. Also, some questions were raised by the reviewers; the authors are encouraged to clarify those as well in the paper.
Decision: Accept-Findings