DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
Authors: Mohammadreza Pourreza
Davood Rafiei
Venue: NeurIPS 2023 poster (2023)
Keywords: In-Context Learning
Text-to-SQL
Task Decomposition
Spider Challenge
Natural Language Interfaces to Databases
TL;DR: Significantly improving the performance of in-context learning approaches on the challenging task of text-to-sql through problem decomposition

Abstract: There is currently a significant gap between the performance of fine-tuned models and prompting approaches using Large Language Models (LLMs) on the challenging task of text-to-SQL, as evaluated on datasets such as Spider. To improve the performance of LLMs in the reasoning process, we study how decomposing the task into smaller sub-tasks can be effective. In particular, we show that breaking down the generation problem into sub-problems and feeding the solutions of those sub-problems into LLMs can be an effective approach for significantly improving their performance.  Our experiments with three LLMs show that this approach consistently improves their simple few-shot performance by roughly 10%, pushing the accuracy of LLMs towards SOTA or surpassing it. On the holdout test set of Spider, the SOTA, in terms of execution accuracy, was 79.9 and the new SOTA at the time of  this writing using our approach is 85.3. Our approach with in-context learning beats many heavily fine-tuned models by at least 5%. Additionally, when evaluated on the BIRD benchmark, our approach achieved an execution accuracy of 55.9%, setting a new SOTA on its holdout test set.

DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
Mohammadreza Pourreza Department of Computer Science University of Alberta Edmonton, CA pourrera@ualberta.cc
Davood Rafiei Department of Computer Science University of Alberta Edmonton, CA Brafiei@ualberta.cc
Abstract
There is currently significant gap between the performance of fine-tuned models and prompting approaches using Large Language Models (LLMs) on the challeng- ing task of text-to-SQL, evaluated performance of LLMS in the reasoning process, we study how decomposing the be effective. In particular,
datasets such Spider. To improve the
task into smaller sub-tasks down the generation problem into sub-problems and feeding the solutions of those sub-problems into LLMs can be effective approach for significantly improving their performance. Our experiments with three LLMs show that this approach consistently improves their simple few-shot performance by roughly 10%, pushing the accuracy of LLMs towards SOTA or surpassing it. On the holdout test set of Spider, the SOTA, terms of execution accuracy, was 79.9 and the new SOTA the time of this writing using our approach is 85.3. Our approach with in-context learning beats many heavily fine-tuned models by least 5%. Additionally, when evaluated on the BIRD benchmark, our approach achieved an execution accuracy of 55. 5%,, setting new SOTA on holdout test set.
show that breaking
Introduction
Natural language interfaces to databases aim at making easier for end users access data relational database. For example, given the utterance "find employees who make more than their managers" and the schema tables employees and manages, one may want to generate a query in SQL that retrieves those employees from database. Over the past two decades, research in this field has progressed through several phases, with early systems being domain domain-specific, supporting controlled natural language [Popescu al.) 2003 2004 Li et al. 2007 Li and Jagadish] 2014) or relying on based approaches [Stratica et al. 2005 while more recent systems offering greater domain-independence using supervised models trained on diverse domains and datasets (Zhong
et al. 2017 Yu et al. [0188] and more recently deep neural models trained on large text and code
repositories Dong and Lapata 2016 Devlin al. 2018).
The latest development in this progression is the use Large Language Models (LLMs) under zero-shot and few-shot prompting [Rajkumar et al. 2022 Liu et al. 2023a It has been shown that
LLMs provide strong baselines using only few demonstrations and no fine-tuning [Chen et al.
2021
Brown et al. 2020 Liu al. 2023b). However, these models fall behind commonly used benchmarks (e.g., Spider) compared to well-designed and fine-tuned models. Table shows the performance of two latest LLMs, CodeX and GPT-4, on the development set of the Spider dataset.
Despite strong performance, LLMs fall behind, compared to existing methods (Scholak et al. [02T]
Li al. 2023a especially medium and complex queries. The question investigated in this paper
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

is where these LLMs fail and if some of the problems that they are facing can be mitigated to push the performance to reach or surpass fine-tuned SOTA models.
Prompting has several advantages over traditional approaches using pretraining or fine-tuning. The main benefit is that L.MS can perform prediction tasks without requiring large task-specific training data. Training models from scratch or fine-tuning them is resource-intensive process, often requiring a large number of training samples and machine resources, which may not be available. Additionally, few-shot prompting has been shown to outperform previous state-of-the-ar methods on several benchmark datasets and can achieve high accuracy even with limited training examples [Brown et al. 2020 Wei et al. 2022b].
It has been recently shown that the performance of LLMs can be improved on more complex tasks (e.g., math word problems, compositional navigation steps) using ap- proaches such as chain-of-thought [Wei et al. 20226],
Fine-tuning approaches
Method
Execution accuracy
RED-SQL. 38 + NaIQQI. Li al. 20230
84.5
least-to-most [Zhou et al. 2022], and decomposed [Khot et al. 2022] prompting techniques where a task is broken down into multiple steps and the intermediate results are used to generate a final answer. Unlike algebraic expres- sions, which consist of clear steps or operations, breaking a complex SQL query can be a more daunting task be- cause of the declarative structure of the language and the complex relationships between query clauses.
T5-3B PICARD Scholak et al. 2021
79.3
Inference-only approaches
Method
Execution accuracy
Zero-shot GPT-4 (Ours)
64.9
In this paper, we propose novel method based on few- shot prompting that decomposes the task of natural lan- guage text to SQL (referred to text-to-SQL) into mul-
Fhw-shot GPT-4 (Ours)
67.4
Zero-shot Code Rajkumar et al, 2022
55.1
tiple sub-tasks. Previous works on text-to-SQL prompting using LLMs are only evaluated in zero-shot setting (Ra- ikumar et al.] 2022 Liu et al. 2023a However, zero-shot prompting only provides lower bound on the potential
Few-shor CodeX (Ours)
61.5
Table I: Zero-shot and few-shot prompt- ing compared to fine-tuned approaches on the dev set of Spider
power of LLMs for most tasks [Zhang et al., [222 Ko-
2022. Wei et al., 2022b 2021 Brown et al., 2020 We show that our proposed method outperforms the few-shot prompting method by a large margin. We also
jima et al.
compare our method with previous approaches on two cross-domain challenging benchmarks, Spider and BIRD. For Spider dataset, we use the two official evaluation metrics of execution accuracy and exact set match accuracy [Zhong et al. 2020). We utilize two variants of the CodeX family, namely Davinci and Cushman (Chen et al. 2021 and the GPT-4 model for prompting. On the holdout test set of Spider, our method achieves execution accuracy of 85.3% and 78.2% respectively using GPT-4 and CodeX Davinci models and an exact set match accuracy of 60% and 57% respectively using the same models. The large gap between the exact match and exccution accuracies is due to the few-shot in-context nature of our method. Pretrained and fine-tuned approaches are more likely to generate SQL queries with higher exact set match accuracy simply because these models have seen many examples during training that follow the composition style of the queries in the test set (queries in both sets are often written by the same people). Before our work, the SOTA on the test set had an execution accuracy of 79.9% [Li et al. 2023a and an exact set match accuracy of 74% (Li al. 2023b), and our method sets a new ground in terms of the execution accuracy. On the BIRD benchmark, our approach achieves a new SOTA result, attaining an execution accuracy of 55.9% on the holdout test set and 50.72% on the development set when employing GPT-4 Moreover, using the valid efficiency score introduced in this benchmark, our approach outperformed a GPT-4 baseline, demonstrating a 9% improvement on the development set. This highlights the efffcctiveness of our method.
Our contributions can be summarized follows: (1) improving the performance of -LM--ased text- to-SQL models through task decomposition. (2) introducing adaptive prompting strategies tailored to task complexity, (3) addressing schema linking challenges in the context of prompting, and (4) using LLMs for self correction. To replicate the reported results, visit our GitHub repository for access to the prompts, results, and the code.
prroppting

---

2 Related Work
Sequence-to-sequeee models [Sutskever et al. 2014] have shown great potential in code generation tasks including text-to-SQL. The key idea is to jointly encode a given natural language question and the database schema and leverage decoder to predict the target SQL.
On the encoder side, learning a representation for the question and the database schema is carried out using bidirectional LSTM in IRNet [Graves and Graves 2012]. convolutional neural networks in RYANSQL [Choi et al., 202T pretrained language models such as BERT in SQLova (Hwang et al.
2019] and graph neural networks in RATSQL [Wang et al. 2019 SADGA (Cai et al. 202 J. and
LGESQL [Cao et al., 2021 Gan al. [021 propose an intermediate representation to bridge the gap between the natural language question and SQL statements. There has been also work on tabular
language models that encode both tables and text such as TaBERT Yin et al. 2020 TaPas [Herzig
et al. 2020 and Grappa (Yu et al. 2020).
The methods on the decoder side can be categorized into sketch-based slot-filling and generation- based methods [Qin ct al.) 2022). Sketch- based methods break the problem into several slot prediction sub-problems and aggregate the predictions for the slots of the SQL query be generated [Hwang ct al. 2019 Xu et al. 2017 Hui et al.] 2021]. A drawback of these methods is that they cannot generalize to queries that do not follow the predefined templates.
The generation- -aasdd methods
[Guo et al. 2019| Wang et al. 2019 Cao et al. 2021 Huang et al. 2021 decode the SQL query as
an abstract syntax tree.
In contrast to pretrained and fine-tuned models, Rajkumar et al. 2022 and Liu et al. 2023a conduct
an evaluation of the zero-shot prompting capability of LLMS on text-to-SQL. using different prompts on the Spider dataset. Prompting techniques have been also used for tasks such as table understanding.
table reasoning, and table-to-text generation [Guo al.) 2023 Chen 2022]. , and some remarkable
results have been reported using LLMs with just a small number of examples given in the prompt.
3 Few-shot Error Analysis
To better understand where LLMs fail under few-shot setting, randomly sampled 500 queries from different databases in the training set of the Spider dataset, excluding all databases used in our prompts. We searched for the queries that produced results different than those of gold queries, hence failing the execution accuracy. We manually examined these failures and classified them into six categories shown in Figure | and discussed next.
Schema linking This category contained the largest number of failed queries and included instances where the model failed to identify col- umn names, table names, entities mentioned in questions. In some cases, the query required an aggregation function, but matching column name was chosen instead. For instance, the database schema for question "What are the erage and maximum capacities for all stadiums?" included column named "average", which was selected by the model instead of taking the av- erage of the capacity column.
Wrong cols 15%
Wrong tables 12%
Schema linking 37%
Wrong cols
Wrong entities 10%
JOIN 21%
Fallures
Invalid 3%
Dp Cord
Wrong tables 8%
Other 13%
JOIN This was the second largest category and included queries that needed JOIN but the model was unable to identify all the tables re- quired or the correct foreign keys to join the tables.
GROUP-BY 13%
DESC 4%
Wrong cols
13%
DISTICT
Not detected 5%
Set Op
Wrang sub-query
966
GROUP BY This category included cases where the SQL statement required a GROUP BY clause, but the model either did not recog- nize the need for grouping or wrong columns were used for grouping the results.
Figure 1: Statistics of simple few-shot failures using CodeX Davinci (Op refers operators, Cond refers to conditions, and cols refers to columns)
3

---

SQL generation
singer, columns [singer -.. Table concert, columns- = [concert_DD ID,
Easy
Select name from singer
Where Citizenshipiz 'French'
Q: "What are the names of the singers
Nested Complex
Classification
Schema linking
Non-Nested Complex
Self-correction
Decomposition
Figure 2: An overview of the proposed methodology including all four modules
Queries with nesting and set operations
For this category, the gold query used nesting or set operations but the model did not recognize the nested structure or unable to detect the correct nesting or set operation.
Invalid SQL small set of the generated SQL statements had syntax errors and could not be executed.
Miscellaneous This category included cases that did not fit under any of the previously mentioned categories. Examples included SQL queries that contained extra predicates, missed predicate, or had missing redundant DISTINCT or DESC keywords. This category also included cases where the WHERE clause was missing or the query had redundant aggregation functions.
4 Methodology
Despite improvements over zero-shot, few-shot models struggle more complex queries including those where schema linking is less trivial and the queries that use multiple joins or have nested structure, discussed in Section
Our approach to address these challenges is to break down the problem into smaller sub- sub-problems, solve each sub- tebbprrblem, and use those solutions to construct solution for the original problem. Similar approaches (e.g., chain-of-thought prompting (Wei et al. 2022b) and least-to-most prompt- ing (Zoou et al. 2022) have been taken improve the performance of LLMSs tasks that be broken down into multiple steps such as math word problems and compositional generaliza- tion (Cobbe al., 202 Lake and Baroni 2018 Unlike these domains where the tasks have procedural structure with one step directly feeding into the next step. SQL queries in most parts are less clear. However, the thought process for writing SQL queries may be broken down to (1) detecting database tables and columns that are relevant the query, (2) identifying the general query structure for more complex queries (e.g., group by, nesting, multiple joins, set operations, etc.), (3) formulating any procedural sub-components if they can identified, and (4) writing the final query based on the solutions of the sub-problems.
declarative and the possible steps and their boundaries
Based on this thought process, our proposed method for decomposing a text-to-SQL. task consists of four modules (as depicted in Figure 2 (1) schema linking, (2) query classification and decomposition, (3) SQL generation, and (4) ceff-correction, which are explained detail in the following sub-sections. While these modules may be implemented using techniques from the literature, we implement them all using prompting techniques to show that LLMs are capable of solving them all if the problems are simply broken down the right level of granularity. The few-shot examples used in the prompts obtained from the training set of the respective benchmarks.
4.1 Schema Linking Module
Schema linking is responsible for identifying references to database schema and condition values in natural language queries. shown help with the generalizability across domains and the synthesis of complex queries
[Lei et al. 2020], making it critical preliminary step in almost all

---

10 task-specific demonstrations
10 task-specific demonstrations
concert, [concert ID, Naree
Table advisor, columns [D_ LID,
Foreign. keys Stander ID Stadium ID, between 5000 and 10000." A: Let's think step step.
Forelgn_kep keys= [managementhead ID= head.head ID, ...]
name stadiums capacity
"How many courses that do have prerequisite?" schema_ links: [course." " ,course.course__K
prerequisite.co coccqusstte.couree id]
Let's think step step.
.....
linking
Classification
www.ereeooooee.................. wo arrenteeeennnn in the question "Show location and name for stadiums with
Decomposition
capacity between 5000 and 10000.
asked:
all stadiums' so we [stadium.Location Location, stadium.Nan Namm]
"location
query question "How many courses do not have prerequisita? ?" needs these tables [course, prerequisite]. so we need JOIN.
"stadiums with capacity" [stadium. |stadium.Capacity)
need column
Plus, requires nested queries with (INTERSECT, UNION, EXCEPT, IN, NOT IN), and we need the answer to the questions ("Which courses have preroquisite?") So, need JOIN and need nested querles, then the the SQL query can be classified as "NESTED".
Based on the columns and tables, we need these Foreign keys
on the tables, columns, Foreign keys, set possible cell values are = 15000,10000).
So the Schema. links Schema, links: Location, Name, stadium Capacily, Reveemeee
10000] com www.iieeee......................
Label: "NESTED"
(a) schema linking module
(b) classification and decomposition module
Figure Examples showing the input and output of schema linking (left) and classification and decomposition (right)
2019 Xuan al. 2021]. This was also a single category with the largest number of fai ures made by the LLM in our case (Figure
existing text-to-SQL methods (Cao al. 2021 Wang al. 2019 Guo et al.
We designed a prompt- prompt-based module for schema linking. The prompt includes ten randomly selected samples from the training set of the Spider dataset. Following the chain-of-thour template (Weil et al. 20226). the prompt begins with "Let's think step by step." as suggested by Kojima et al. [2022] For each mention of a column name in the question, the corresponding columns and their tables
selected from the given database schema. Possible entities and cell values are also extracted from the
question. Figure 3a illustrates an example and the full prompt can be found in Appendix A.3
4.2 Classification & Decomposition Module
For each join, there is some chance that correct table or join condition is not detected. As the number of joins in a query increases, the chance that at least one join fails to generate correctly increases. One way to alleviate the problem is introduce a module that detects the tables to be joined. Also some queries have procedural components such as uncorrelated sub-queries, which may be generated independently and be merged with the main query.
To address these issues, we introduce query classification and decomposition module. The module classifies each query into one the three classes: easy, non-nested complex and nested complex. The easy class includes single-table queries that can be answered without join nesting, The non-nested class includes queries that require join but no sub-queries, and the queries in the nested class contain joins, sub-queries and set operations. The class labels important for query generation module, which uses different prompts for each query class. In addition to class labels, query classification and decomposition also detects the set of tables to be joined for both non-nested and nested queries as well as any sub-queries that may be detected for nested queries. Figure shows example input given to the model and the output that the model generates.
4.3 SQL Generation Module
As the queries become more complex, additional intermediate steps must incorporated to bridge the gap between the natural language question and the SQL statement. This gap. known as the mismatch problem the literature [Guo al. 2019), poses significant challenge to SQL generation, which stems from the fact that SQL is primarily designed for querying relational databases and not
5

---

representing the meaning in natural language K 2008]. While more complex queries can benefit from listing the intermediate steps in a chain of thought style prompting, such listings can degrade the performance for simpler tasks (Wei et al. 2022b] On the same basis, our query generation comprises of three modules, each geared toward different classes.
For questions in our easy class, a simple few-shot prompting with no intermediate steps is adequate. The demonstration for an example E of this class follows the format <Q,, Sz, A,>, where Q2 and Aj give the query text in English and SQL respectively and S, indicates the schema links.
Our non-nested complex class includes queries that require join. Our error analysis ($ () revealed that finding the right columns and foreign keys to join two tables can be challenging for LLMs under simple few-shot prompting, especially when the query requires joining multiple tables. To address this issue, we resort to an intermediate representation to bridge the gap between queries and SQL. statements. Various intermediate representations have been introduced in the literature. In particular, SemQL. (Guo et al., 2019] removes operators JOIN ON, FROM, and GROUP BY, which have no clear counterparts in natural language queries, and merges the HAVING and WHERE clauses. NatSLL [Gan et al.) [021] builds upon SemQL and removes the set operators. Expressions in natural language queries may not clearly map a unique SQL clause or they may map to multiple clauses, so removing operators makes the transition from natural language to SQL easier. As our intermediate representation, we use NatSQL, which is shown to have a state-of-the oof-the-art performance when combined with other models [Li et al. 2023a The demonstration for an example E, of the non-nested complex class follows the format Sq JJIJAA,7, where S, and 15 respectively denote the schema links and the intermediate representation for the jth example.
Lastly, the nested complex class is the most sophisticated type and requires several intermediate steps before generating the final answer. This class can contain queries that not only require sub-queries using nesting and set operations such as EXCEPT, UNION, and INTERSECT but also multiple table joins, same as the previous class. To break down the problem further into multiple steps, our prompt for this class is designed in a way that the LLM should first solve the sub-queries, generated from the previous module, and then use them to generate the final answer. The prompt for this class follows the format <Q2, Sy, <Q : .j..... Q3x A,. Aj}, where denotes the number of sub-questions. and Qj and Aj, respectively denote the i-th sub-question and the i-th sub-query. As before. Q, and A, denote the query in English and SQL respectively, S gives the schema links and T, is a NaiSQL. intermediate representation.
Full prompts for all three query classes are provided in Appendix A.4 and all examples for the three classes are obtained from the exact same training set database chosen for the classification prompt.
4.4 Self-correction Module
The generated SQL queries can sometimes have missing or redundant keywords such as DESC, DISTINCT and aggregation functions. Our experience with multiple LLMMs indicates that these issues are less common in larger L.MSs (e.g., queries generated by GPT-4 have less bugs than those from CodeX) but are still present. To address this, we propose a self-correction module where the model is instructed to correct those minor mistakes. This is achieved in zero-shot setting, where only the buggy code is provided to the model and it is asked to fix the bugs. We propose two different prompts for the self-correctio to identify and correct the errors in the "BUGGY SQL". The gentle prompt, on the other hand, does not assume the SQL query is buggy, and instead asks the model to check for any potential issues and provides some hints on the clauses to be checked. Our evaluation indicates that a generic prompt can yield a better result with the CodeX model, while a gentle prompt is more effective for the GPT-4 model. Unless explicitly stated otherwise, the default self-correction prompt in DIN-SQL is set to gentle for GPT-4 and generic for CodeX. Examples of both generic and gentle self-correction prompts can be found in Appendix A.6
module: generic and gentle. With generic prompt,
request the
Experiments
5.1 Models
We evaluated the proposed method using two variants of the CodeX family (Davinci and Cushman variants) and the GPT-4 model. These are the largest open-access LLMs at the time of writing this

---

paper. Smaller models are less applicable since prompting is believed to be an emergent ability of the LLMs with the number of parameters in the scale of billions [Wei et al. 2022a
5.2 Hyperparameter
All models were accessed via the OpenAl API. Greedy decoding was used to generate the output by setting the temperature 7ero. The max tokens 350 for the self-correction module and 600 for all other modules. The stopping token sequence was set to "#AAn \n" for the self-correction module and "Q:" for all other modules.
5.3 Dataset
Our evaluation was conducted on two cross- domain challenging datasets, Spider and BIRD. Spider consists of 10,181 questions and 5,693
Model
EX
RM
unique complex SQL queries across 200 databases, covering 138 domains, each contain- ing multiple tables. The standard protocol for this dataset divides into 8,659 training exam- ples across 146 databases, 1,034 development examples across 20 databases, and a holdout of 2, 2,147 test examples across 34 databases. The databases used in each of these sets are non- overlapping. SQL queries are categorized into four difficulty levels, based on the number of SQL keywords used, the presence of nested subqueries, and the usage of column selections and aggrega- tions. BIRD comprises an extensive dataset with 12,751 unique question-SQL. pairs, encompass- ing 95 large databases totaling 33.4 GB in size. It spans a wide array of more than 37 profes- sional domains, including blockchain, hockey, healthcare, and education. BIRD also introduces external knowledge as an additional resource to assist models in generating accurate SQL queries. Specifically four sources of external knowledge were introduced: numeric reasoning knowledge, domain knowledge, synonym knowledge, and value illustration. Notably, the SQL queries in the BIRD dataset tend to be more intricate than those in the Spider dataset. Language models without access to database content often encounter challenges with schema linking. Therefore, our prompts for the BIRD dataset include sample rows from each table to aid the model in schema linking. Furthermore, we have concatenated the provided external knowledge for each question as hint, placed immediately after each question. However, due to constraints such as limited context window size, the presence of external knowledge, and the inclusion of sample rows, have had reduce the number of demonstrations within the prompts for the BIRD dataset.
(Ours)
RESDSQL 3B + NaiSOL (DB content used)
79.9
72
Li at 20236
DIN SQL + CodeX davinci (Ours)
78.2
57
Graphix33B+PICAr PICARRD (DB content used)
77.6
74
Li
2026
SHiP-PICARD (DB content used)
76.6
73.1
N best Rerankers PICARD (DB content used)
75.9
72.2
Zeng et 2022
RASAT-PICARD (DB content used)
75.5
70.9
Qi et al. 2022
15-3B+PICARD (DB content used)
75.1
71.9
Scholak nl. 2021
RATSQL+GAP+NatSQL (DB content used)
73.3
68.7
Gan et al., 2021
RYANSQL v2
60.6
Choo et al. 202]
SmBoP
60.5
Rubin and Berant 2020
Table 2: Execution accuracy (EX) and exact set match accuracy (EM) on the holdout test set of Spider
5.4 Metrics
The performance of models are evaluated using the official metrics of cach dataset: exact-set- match accuracy (EM) and execution accuracy (EX) for Spider and valid efficiency score (VES) and execution accuracy (EX) for BIRD.
The sxact-set-mateh accuracy (EM) treats each clause as a set and compares the prediction for each clause to its corresponding clause in the reference query. A predicted SQL query is considered correct only if of its components match the ground truth. This metric does not take values into account. The execution accuracy (EX) compares the execution output of the predicted SQL query with that of the ground truth SQL. query on some database instances. Execution accuracy provides more precise estimate of the model's 's performance since there may be multiple valid SQL queries
7

---

for given question, and exact set match accuracy only evaluates the predicted SQL against one of them. The Valid Efficiency Score (VES) is metric designed measure the efficiency of running the generated SQL queries. This metric is meaningful if the generated queries are correct, meaning their result matches that of the reference query. Therefore, the VES metric takes into account both the accuracy of the generated queries and their efficiency in terms of the execution time.
5.5 Results
5.5.1 Test set results
As shown in Table 2 for the holdout test set of Spider, method achieves the highest execution accuracy using GPT-4 and the third-highest execution accuracy using CodeX Davinci among all officially published results the time of this writing. This is achieved without database content. In terms of exact set match accuracy, our method achieves comparable results to previous works that do not utilize database content. As demonstrated in Table 3a in the of the BIRD dataset, our method using GPT-4 achieved a test set execution accuracy of 55.9%, setting a new SOTA.
utilizing the
5.5.2 Development set results
Most of our evaluation during development was conducted on the development set of Spider which was easily accessible unlike the test set that was only accessible through evaluation server provided by Yu et al. [2088] Table 4a
Few-shot prompt
DIN-SQL
shows the performance of method using dif- ferent LLMs, compared to zero-shot prompting
60
of Rajkumar al. [2022] and Liu al. 2023a
and our own few-shot prompting. To ensure fair comparison for the few-shot prompting,
Number
we incorporate all the examples utilized for our three classes (easy, non non-nested complex, and nested complex) inside the prompt. Given that the CodeX Cushman model has smaller input context size than the CodeX Davinci and the GPT-4 models, we only use examples from each class (for total of 6 examples).
10
Schemaa
JOIN
categories
Figure 4: The break-down failure cases for DIN- SQL (green) and the basic few-shot prompting (blue) across different categories
Our method significantly outperforms both sim- ple few-shot prompting and zero-shot prompt- ing, in terms of both exact match and execu- tion accuracies, and the improvement is consistent across all models despite their sizes. For example, compared few-shot prompting, our method improves the execution accuracy for all models by at least 10%.
On the development set of BIRD, our approach demonstrates substantial improvement, achieving 4% gain in execution accuracy and a remarkable 9% improvement valid efficiency score over a GPT-4 baseline Li et al. 2023c], establishing new SOTA. These and other results are reported in able 3N
The performance of our method on the test set (as reported in Tables and 3a is higher than that on the development set for both Spider and BIRD. It is hard to pinpoint the exact reason when the test set is hidden, but speculate that fewer questions in the test set may require the knowledge of the database content, making it casier for our method to predict a correct SQL query. Furthermore, the development set has schema ambiguity (e.g., query entity can be mapped to multiple database entities but only one considered correct), and it is possible that the test set has less ambiguity.
We further analyzed the performance of our proposed method on queries with different levels of difficulty. Table 4b presents the performance of our proposed method compared a basic few-shot prompting the development set of Spider. Our proposed method outperforms the basic few-shot prompting across all difficulty levels, with the greatest improvement performance observed for the extra hard and hard classes where the fho-shot prompting performed poorly. Our improvement

---

Model
VES
EX
Model
VES
EX
DIN-SQL GPT-4 (Ours)
59.44
55.9
DIN-SQL GPT-4 4PT-4 (Ours)
58.79
50.72
GPT-4
60.77
54.89
GPT-4
49.77
46.35
Claude-2
49.02
Claude-2
42.70
ChatGPT CoT
56.56
40.08
ChatGPT CoT
42.30
36.64
Li et al. 2023c
Li et all 20236
ChatGPT
51.40
39.30
CharGPT
43.81
37.22
Codex
41.60
36.47
Codex
34.35
Palm-2
33.04
Palm-2
27.38
T5-3B
27.80
24.05
TS-3B
25.57
23.34
Large
25
20.94
Large
22.74
19.75
TS Base
14.7
12.89
Base
12.90
11.54
(a) Exccution accuracy (EX) and Valid Efficiency
(b) Execution accuracy (EX) and Valid Efficiency
Score (VES) on the holdout test set of BIRD
Score (VES) on the development set of BIRD
Table 3: Performance of DIN-SQL on BIRD development set and test set.
on the easy class (compared to basic few-shot) is due to incorporating schema links in the prompt, highlighting the importance our schema- schema-linking module.
Prompting
Model
EX
EM
GPT-4
74.2
60.1
Execation accurney (EX)
DIN-SQL (Ours)
CodeX Davinci
69.9
57.2
Prompting
Model
Easy
Medium
Hard
Extra
All
CodeX Cushman
47.6
35.7
GPT-4
67.4
DIN-SQI.
GPF4
91.1
79.8
64.9
43.4
74.2
Few-shot (Ours)
CodeX Davinci
61.5
50.2
CodeX Cushman
43.1
30.9
Few-shot
GPT4
86.7
73.1
59.3
31.9
67.4
Zero-shot (Ours)
GPT-4
64.9
40.4
Few-shor
CodeX Davinci
84.7
67.3
47.1
26.5
61.5
Liu et al. 20230
Zero-shot
ChatGPT
60.1
Prompting
Model
Esact set match accuracy (EM)
Easy
Medium
Hard
Extra
All
Rajkumar
Zero-shot
CodeX Davinci
47.5
DIN-SQL
GPT-
82.7
65.5
30.7
60.1
CodeX Davinei
53.1
DIN-SQI.
CodeX Davinei
78.6
67.3
38.5
17.5
57.2
Zero-shot
used)
Rajkumar et al. 2022 2022
CodeX Cushman
53
GPT4
GPT3
21.7
Few-shot
Davinei
77
127
(a) Performance compared zero-shot and few-shot prompting using different L.MSs on the dev set Spider
(b) Performance compared to our basic few-shot prompting across different query difficulty levels
Table 4: Performance of DIN-SQL against other in-context learning approaches
5.5.3 Error improvements
In Section 3 we did an eтror analysis of basic few-shot prompting on 500 queries randomly chosen from the training set. To understand the degree those errors are resolved, we ran DIN-SQL. on the same 500 queries. As shown in Figure g our proposed approach improves the performance for all categories with the largest improvement seen for the JOIN and Nested categories. Despite having an explicit module for schema-linking, the largest portion of failure cases still belong to this category.
5.6 Ablation study
In an ablation study, we evaluated our approach with and without each of the four modules. As shown in Table for the CodeX Davinci model, excluding any of the modules leads to an overall decrease in performance, terms of the execution accuracy.
More details emerge as we study the effectiveness of each module across different query classes. Schema linking helps all query classes with the least improvement for the hard class. Our inspection of a sample of the failed cases reveals that schema linking sometimes finds redundant links due to an ambiguity in the question schema, and this can introduce redundant joins or output columns.
Without classification, we had to use either a simple few-shot prompting or a decomposed chain- of-thought (COT) prompting for all queries. The reported performance without classification

---

Prompting
Model
Easy
Medium
Hard
Extra
All
DIN-SQL (generie self-corr)
CodeX Davinci
89.1
75.6
38.6
69.9
DIN-SQL (gentle self-corr)
CodeX Davinci
87.5
76.9
51.7
36.1
68.7
DIN-SQL w/o self-corr
CodeX Davinci
83.9
75.4
52.3
36.1
67.3
DIN-SQL w/o schema-linking
CodeX Davinci
87.3
70.6
57.6
27.1
65.9
DIN-SQL w/o classification
CodeX Davinci
87.9
68.2
51.7
27.1
63.1
(simple few-shot prompting)
DIN-SQL w/o classification
CodeX Davinci
84.2
71.2
54.3
38.6
68.2
(dccomposed COT prompting)
DIN-SQL (gentle self-cumr)
GPT-4
91.1
79.8
64.9
43.4
74.2
DIN-SQI. (generie self-corr)
GPT-4
89.9
76.5
59.2
34.3
70.0
DIN-SQL w/o self-corree
GPT-4
91.1
79.1
63.2
41.6
73.3
Table 5: Performance of our method, in terms of execution accuracy, on the dev with and without each module
module in Table 5 is for our comprchensive framework that includes all our components except classification. This means that the approach contains not only COT prompting but also Schema Linking, Self-Correction, and NatSQL Intermediate Representation, all of which are significant contributions of our work. The decomposed chain-of-though result presented in this table refers to employing the most complex prompt, developed for the nested complex class, for all questions instead of adopting a classification--ased approach to determine prompt complexity based on the question's level of difficulty. In contrast, the result for the DIN-SQL with simple few-shot prompting refers to using the simplest prompting class, easy class, for all questions across different level's 's of difficulty. As expected, decomposed chain-of- thain-of-thought prompting works better for hard and extra hard queries whereas a simple few-shot works better for the easy class.
For seff-correction, we ran our study using both CodeX Davinci and GPT-4. For CodeX Davinci, a generic self- self-correction prompt helps the model across all query classes. gentle self-correction prompt is also helpful but the gain is smaller than generic one for CodeX Davinci. However, there less chance that GPT-4 generates a buggy code, and giving a generic prompt of "Buggy SQL:... Fixed SQL:.." can hurt the performance. A gentle prompt work better for GPT-4 and improves the perfromance across all of the classes except the easy class.
6 Conclusions
Prompting has enabled large language models to achieve impressive performance on numerous NLP tasks across different domains, without requiring large training set. Prior to research, the effectiveness of prompting methods utilizing LLMs for the text-to-SQL. task was inferior to that of models fine-tuned for the task. To bridge this gap, have devised decomposition technique to tackle some of the challenges that caused this disparity. Our extensive experiments on two challenging datasets of Spider and BIRD show that our method significantly improves the performance of prompting across all query classes, producing comparable or even superior results to state-of-the-art fine-tuned approaches.
7 Limitations
There are some limitations or areas of improvement to this work. Our manually constructed demon- strations are fixed for each query class. Future research may explore adaptive and automated methods for generating demonstrations finer granularities, which can further enhance the performance of our approach. Additionally, as of the time of writing this paper, our proposed approach, characterized by its decomposed and step-by-step structure, incurs a cost of approximately $0.5 and exhibits a latency of approximately 60 seconds when responding to natural language question from the Spider dataset using GPT-4. We anticipate that as L.MMs continue to advance, these costs and latencies should decrease, but reducing the cost is another possible direction.
10


Reviews:
[{"review_id": "hXS2HAW3Rh", "reviewer": "Reviewer_kanX", "summary": "The paper's contributions are the following:\n- The authors examined the common failure modes of doing text-to-SQL with few-shot prompting of LLMs: schema linking, JOIN, GROUP BY, nested queries and set operations, invalid SQL and miscellaneous.\n- The authors propose a method to do text-to-SQL by decomposing the task into 4 sub-problems, and solving each with a few-shot / zero-shot prompt to an LLM: schema linking, classification and decomposition, SQL generation, and self-correction. Their in-context learning method is reported to attain the highest execution accuracy on the test set of the Spider dataset, without making use of database content. \n- The work shows that LLMs can be used for text-to-SQL via prompting with performance equivalent to or better than methods that make use of fine-tuning. \n", "questions": "- How is the intermediate representation obtained? I refer to Appendix A.5.2 to see that it is inserted into the prompt, but it is not clear where this is generated.\n- How do the authors think the removal of operators like GROUP BY or the WHERE clause from the syntax of the intermediate representation help the LLM still generate the right SQL statements?\n- Can the authors illustrate how a query is decomposed into sub-queries for the nested complex class? It is not clear to me still in Appendix A.5.3.\n- What is the latency of the proposed decomposed in-context learning method, and how does it compare to other methods (like RESDSQL-3B + NatSQL or Graphix-3B+PICARD)?", "limitations": "Yes, limitations are adequately addressed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is a good application of chain-of-thought style problem decomposition to in-context learning techniques for text-to-SQL. \n- The method is well motivated by first conducting an examination of common failure modes when using LLMs for text-to-SQL in the few-shot setting. Investigating the error improvements using their method in Figure 4 is a very clear way to show how their method helps with better in-context learning. \n- The paper is written in a largely clear manner that makes it easy to follow.\n- The paper's results are a relevant contribution to the text-to-SQL community, in that their performance is significantly better than other LLM techniques like those in Rajkumar et al., 2022 and Liu et al., 2023. ", "weaknesses": "- There could have been more discussion about how the proposed four modules have been implemented without using prompting, to better situate the work in the literature.\n- The paper should have been clearer about how the intermediate representation of NatSQL bridges the gap between queries and SQL statements. An example each for the non-nested complex class and nested complex class would have been helpful in the main paper, instead of leaving it to the Appendix. In particular, it is not clear in the paper 1) how the intermediate representation is obtained, 2) how the removal of operators like GROUP BY or the WHERE clause from the syntax of the intermediate representation can help the LLM still generate the right SQL statements and 3) how the LLM is induced to solve sub-queries for the nested complex class.\n- The paper could have been clear about the latency of its proposed method. While high-performing, it is likely that making several sequential calls to an LLM will be high latency. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Question: How is the intermediate representation obtained (ref Appendix A.5.2)? \n\nAnswer: For our few-shot examples, we used the intermediate representation from the NatSQL Github repo. The repo gives the intermediate representation for all queries in the training set of Spider. \n\nQuestion: How does removing operators in the intermediate representation help?\n\nAnswer: Removing/merging operators makes the transition from natural language to SQL easier and is part of our problem decomposition. Expressions in natural language queries may not clearly map to a unique SQL clause or they may map to multiple clauses. For example, some conditions are mapped to the WHERE clause whereas others are mapped to the HAVING clause. Some SQL clauses do not have a clear counterpart in text descriptions (e.g. JOIN and GROUP BY). Dispensing or merging the operators makes the generation task easier and pushes the LLM to focus on correctly predicting the query structure before refining it in the next step.\n\nQuestion: Can the authors illustrate how a query is decomposed into sub-queries for the nested complex class? It is not clear to me still in Appendix A.5.3.\n\nAnswer: The sub-questions are extracted in the classification and decomposition module. For example, as shown in Fig 3-b, for the question “how many courses that do not have prerequisites,” the sub-question: “which courses have prerequisites” is extracted. In SQL generation, SQL is generated for sub-questions in a step-by-step reasoning process before generating the whole query. For example, consider the last demonstration in the prompt in Appendix A.5.3, where the question “Give the title and credits for the course that is taught in the classroom with the greatest capacity” has the sub-question “What is the capacity of the largest room?”. The mapping of the sub-question to SQL, i.e. “ (SELECT max(capacity) FROM classroom)”, is provided in the reasoning process before generating the final answer.\n\nQuestion: Latency compared to other methods?\n\nAnswer: Our latency heavily relies on that of the OpenAI API calls which varies a lot over time. For GPT-4, this was typically under 1min. This time can significantly be improved using the LLM on Microsoft Azure. The latency of other approaches also depends heavily on the hardware and the GPU."}, {"review_id": "YAuA4DGuGP", "reviewer": "Reviewer_XH1V", "summary": "This paper proposes to improve few-shot prompting LLMs for text-to-SQL task. It first provides detailed error analysis on existing few-shot prompting LLM approaches, into six categories. Then the paper proposes a new approach to decompose the task into a few sub-tasks, solve each task individually, and compose the subtask for the final answer. \n\nExperiments show the proposed approach achieve state-of-the-art on the Spider benchmark dataset. ", "questions": "1. What's the difference on errors for Codex and GPT4\n2. What's the result on chain-of-thought promoting for GPT4. \n3. What do you think is the upper bound for this dataset, and do you think scaling LLM can achieve the upper bound?", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The result is solid, as it achieves state-of-the-art on the challenging Spider benchmark dataset. In addition, this paper provides detailed ablation study and analysis, therefore it's straightforward to understand the strength and weakness of the proposed approach. \n2. The paper is well-written, easy to follow the motivation and details. ", "weaknesses": "Overall, my biggest concern is there's little additional novelty beyond \"just another prompt engineering paper\". It's expected that with larger LM, few-shot approaches could outperform existing fine-tuning based approaches. However, after reading this paper, it's still unclear whether scaling LLM can solve this text-to-SQL task and if not, where is the bottleneck. \n\nMore specific concerns:\n1. The manual error analysis (Section 3) only applies to Codex, it's unclear whether these errors still exist for larger language models such as GPT4. \n2. There's only marginal improvement compared with chain-of-thought prompting (Table 4). And I guess the margin would be even smaller if  we use GPT4 as LLM.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A detailed explanation of the novelties of our prompting method is provided in the general response to the reviewers.\nThe argument regarding marginal improvement over the Chain-of-thought method is invalid because the cited performance “decomposed COT prompting” is not for the chain-of-thought method alone. The reported performance here is for our comprehensive framework that includes all our components except classification. This means that the approach contains not only COT prompting but also Schema Linking, Self-Correction, and NatSQL Intermediate Representation, all of which are significant contributions of our work. It is essential to consider the entire framework's performance, as each module contributes to the overall effectiveness of our approach.\n\n\nQuestion: What's the difference on errors for Codex and GPT4\n\nAnswer: Our error analysis reveals that the majority of errors in the area of schema-linking remains in transitioning from Codex to GPT-4. Ambiguities within the database schema pose a significant challenge, even for advanced models like GPT-4, in accurately identifying correct columns and tables. While using a larger model addresses certain issues, particularly in cases where the model struggles to generate accurate SQL queries (miscellaneous class of errors), schema-linking complexities remain the primary obstacle.\n\nIn summary, larger models generally exhibit enhanced SQL query generation capabilities, yet the challenges arising from schema-linking ambiguities remain prominent. Addressing these challenges will be a key focus for further improvement and refinement of our approach.\n\n\nQuestion: What's the result of chain-of-thought promoting for GPT4.\n\nAnswer: We did not specifically test the pure chain-of-thought approach on the Spider dataset with GPT-4. However, we evaluated the performance of the chain-of-thought approach when integrated with three other modules proposed by us. Using the chain-of-thought approach uniformly for all questions, irrespective of their complexity, resulted in a performance degradation, as highlighted in Table 4. The decomposed chain-of-thought result presented in this table refers to employing the most complex prompt, developed for the nested complex class, for all questions instead of adopting a classification-based approach to determine prompt complexity based on the question's level of difficulty.\n\nQuestion: What do you think is the upper bound for this dataset, and do you think scaling LLM can achieve the upper bound?\n\nAnswer: We think we are approaching the upper bound on the Spider dataset because there are certain ambiguities in the database, which make it challenging to achieve a flawless performance even if larger LLMs are employed. Some natural language questions have multiple interpretations, and relying on a single reference query as the gold standard is insufficient to address these diverse interpretations. While scaling LLMs can address some issues related to generating complex SQL queries, the primary challenge persists in the area of schema linking, where comprehending database schema based on less-descriptive table and column names remains a difficult task."}, {"review_id": "A0pBvfGStM", "reviewer": "Reviewer_VYLB", "summary": "The paper proposed to decompose the text-to-SQL reasoning into multiple steps and solve them with large language models. The authors began by conducting an error study of LLMs with few-shot learning and identified the common errors, such as \"schema linking\" and \"JOIN\". To address these common errors, they proposed a new method called DIN-SQL which breaks the text-to-SQL task down into four modules: (1) schema linking, (2) query classification and decomposition, (3) SQL generation, and (4) self-correction. \n\nIn the schema linking and query classification modules, examples from the Spider dataset's training set are used to guide LLMs in identifying the relevant database schema and potential SQL structure. The SQL generation modules leverage different examples to handle various SQL structures. For more complex SQL structures, such as non-nested complex and nested-complex queries, demonstration examples with hand-written intermediate reasoning steps are utilized. At last, the self-correction module asks the LLMs to correct the predicted SQL if they realize that their initial generation was incorrect.\n\nThe proposed framework demonstrates remarkable performance, surpassing state-of-the-art models on the Spider test set.", "questions": "I am somewhat confused by the disparity between the results of DIN-SQL in the Spider development set and the Spider test set. On the test set, DIN-SQL demonstrates superior performance compared to the state-of-the-art model [1], achieving a margin of 5.4 points higher (85.3 vs. 79.9). However, on the development set, it falls short by 9.9 points (74.2 vs. 84.1) and is outperformed by other in-context learning methods [2]. Do you know what factors contribute to DIN-SQL's high performance on the test set but comparatively low performance on the development set?\n\n[1] Li, Haoyang, et al. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n[2] Ni, Ansong, et al. \"Lever: Learning to verify language-to-code generation with execution.\" arXiv preprint arXiv:2302.08468 (2023).", "limitations": "Looks good to me.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The proposed method DIN-SQL demonstrates very impressive results on the hold-out test set in the Spider dataset, showing the effectiveness of DIN-SQL on Spider.\n2. The error analysis reveals the drawbacks of LLMs with standard few-shot demonstrations and the proposed method are designed to address these drawbacks.\n", "weaknesses": "1. The proposed methods seem highly tailored to the examples in the Spider dataset. Previous work [1-2] has highlighted the spurious correlations in the Spider dataset, such as the strong lexical matching between the database and the question. It seems important to evaluate the proposed method on other datasets such as Spider-syn [1], Spider-DK [2], Dr. Spider [3], or KaggleDBQA [4].\n\n2. The proposed SQL generation modules require hand-written intermediate reasoning steps. I don't think it is a problem given the limited number of examples requiring annotation but my concern is whether these examples will still serve as good demonstrations for the datasets other than Spider. I hope this can be addressed along with 1.\n\n[1] Gan, Yujian, et al. \"Towards robustness of text-to-SQL models against synonym substitution.\" arXiv preprint arXiv:2106.01065 (2021).\n[2] Deng, Xiang, et al. \"Structure-grounded pretraining for text-to-sql.\" arXiv preprint arXiv:2010.12773 (2020).\n[3] Chang, Shuaichen, et al. \"Dr. Spider: A diagnostic evaluation benchmark towards text-to-SQL robustness.\" arXiv preprint arXiv:2301.08881 (2023).\n[4] Lee, Chia-Hsuan, Oleksandr Polozov, and Matthew Richardson. \"KaggleDBQA: Realistic evaluation of text-to-SQL parsers.\" arXiv preprint arXiv:2106.11455 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The first weakness highlighted by the reviewer has been thoroughly addressed in the general response to all reviewers.\n\nRegarding the second weakness raised in the review, we would like to clarify that our examples were chosen from the training set of Spider and were selected deliberately to include at least one example for each of the widely used SQL keywords. Additionally, the cross-domain nature of the Spider dataset ensures that achieving the highest performance on this benchmark translates to strong performance across various domains. As a result, we believe that the examples used in our prompts are highly applicable to a wide range of existing benchmark datasets.\n\nQuestion: Why is there a disparity between the results of DIN-SQL in the Spider dev set and the test set?\nAnswer: The test set of Spider, which is intended to be hidden, has not been published, making it challenging to determine the exact reason for the performance disparity. However, we have some speculations that may shed light on potential contributing factors:\n\n1) Our proposed approach relies solely on the database schema to generate answers for given questions. This strategy poses challenges in cases where the model needs knowledge of how values are stored in the database. For instance, when the question is \"Give me the name of female users with age over 40,\" the model may not know whether the stored values in the gender column are \"female\" or \"F.\" Consequently, it relies on the question's context and chooses \"female.\" We speculate that the number of questions requiring knowledge of specific database values is lower in the test set compared to the dev set of Spider.\n\n2) Ambiguities exist in the database schema of the dev set of Spider. In some databases within the dev set, tables have columns that store the same information about entities but in a slightly different format. While using any of those columns may be treated as correct, a generated query is deemed correct if its column choice matches that of the reference query. The higher performance of our model on the test set might be attributed to the test set having a more well-structured database schema, reducing such ambiguities.\n\nWhile these speculations offer insights into potential reasons for the observed performance disparity, the lack of access to the test set hinders us from confirming these hypotheses conclusively.\n"}, {"review_id": "GCy91LWhX7", "reviewer": "Reviewer_DUDq", "summary": "This paper addresses the task of text-to-SQL prediction using large language model (LLM) prompting. The authors propose DIN-SQL, a chain-of-thought (CoT) prompting method which decomposes the SQL prediction process into four substeps: schema linking, complexity classification, SQL prediction, and self-correction. For each step, a manually designed prompt with few-shot in-context learning samples are provided to the LLM to obtain the prediction. Especially, after complexity classification, samples with different complexity levels are addressed with different prompts. Experiments on the Spider dataset demonstrate the effectiveness of the proposed method.\n", "questions": "- Around L231: The SQL-prediction step prompt for nested complex questions are designed in a comprehensive way, allowing multiple sub-questions to be included. However, looking at the samples in the appendix, it seems that at most one sub-question is predicted. Is the method actually able to handle very complex queries by generating >1 sub-questions? \n", "limitations": "Several limitations are discussed, although I think there are additional points as I mention above.\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The overall pipeline is conceptually simple but practically effective. It appears to successfully leveraged the power of LLMs and achieved promising results.\n\n- The idea of applying different prompts for samples with different complexity is justifiable. It is based on prior observations that CoT prompts are effective on harder samples but can hurt the performance on easy samples. The ablation studies also echoed this observation, and verified the usefulness of this proposed mechanism.\n", "weaknesses": "- The proposed method does not carry much techinical innovation, as CoT is already a widely studied and applied technique, and the intermediate representation is imported from previous work NatSQL.\n\n- The complexity categorization and corresponding prompt designs are more or less tailored for the Spider dataset. It might be unclear whether the method is directly applicable in real-world scenarios, where the variety of target SQL might be much larger, and may include additional mechanisms such as self-join.\n\n- Based on my observations on the \"sub-questions\" generated for nested complex samples, it seems that many of them (though not all) are identical to the original questions. This might not be the desired behavior and leaves room for further improvements. For example:\n\t- \"Find the title, credit, and department name of courses that have more than one prerequisites?\" can be solved by knowing the answer to the following sub-question \"What is the title, credit value, and department name for courses with more than one prerequisite?\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The first weakness raised in the review has been thoroughly addressed in the general response to all reviewers.\n\nRegarding the second weakness concerning the applicability of our proposed approach to real-world scenarios with more complex SQL query structures, our current classification structure is effective for many benchmarks in the text-to-SQL domain, which aim to resemble queries in real-world applications. However, it is fair to say that the queries in real world applications can be more complex. To handle more complex scenarios which fall outside our current query classes, we may simply add more query classes to our query classification module and provide more relevant steps to each class in our query generation module. We could only evaluate our model on existing benchmarks, and Spider, at the time of writing this paper, stood out as the most complex text-to-SQL benchmark we could use.\n\nThe third weakness pertains to identical questions and sub-questions. During our experiments with query decomposition, we noticed instances where the query did not have a clear subquery or could not be broken down to independent components. In those cases, the model generated a paraphrased version of the original question. To promote this behavior especially when a question cannot be broken to sub-questions, we included examples of such cases in our prompt. Our evaluation shows that including such cases overall helps the model.\n\nQuestion: Is the method actually able to handle very complex queries by generating >1 sub-questions?\n\nAnswer: The presence of only one sub-question in all the few-shot demonstrations can be attributed to the database selection process from the training set of Spider. For the samples in our chosen database, we did not encounter queries that required more than one sub-question. However, during the inference stage, we noticed that the model was capable of generating multiple sub-questions, even though some were unnecessary. It was evident that often one sub-question, along with a SQL query utilizing JOIN operators, proved to be sufficient."}, {"review_id": "8TxfLIzdNR", "reviewer": "Reviewer_Qajd", "summary": "In their study on the Text2SQL task, the authors demonstrate the efficacy of breaking down the generation problem into sub-problems when utilizing LLMs to improve performance on the Spider dataset. The approach is simple yet effective on the Spider dataset. It yielded consistent improvements across three LLMs: GPT-4, Codex Davinci, and Cushman. Notably, the paper achieves a new SOTA performance on the Spider holdout test set.", "questions": "1. How did the authors come up with the self-correction prompts? Was it through a lot of trial and error from the training and validation set? \n2. How did the author decide on this set of in-context learning exemplars? \n", "limitations": "No negative societal impacts\n", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- SOTA on Spider holdout test set\n- Approach consistently improve results on three OpenAI LLMs\n", "weaknesses": "- The analysis and experiments conducted in the study were solely focused on a single dataset, namely Spider.\n- The study does not provide sufficient evidence to support the generalizability of the proposed approach to Text2SQL tasks in general.\n- It is important to highlight the existence of several large-scale and robust Text2SQL datasets that could have strengthened the results in terms of generalizability. Notably, the BIRD-bench dataset (https://bird-bench.github.io/) provides an additional resource for evaluating the proposed approach. Furthermore, the Spider dataset has various variants, such as Spider-SYN, Dr.Spider, Spider-realistic, and Spider-DK, which offer opportunities to assess the effectiveness of the approach in different contexts and scenarios. Conducting experiments on these datasets would have enhanced the overall robustness and applicability of the findings.\n- The foundation models utilized in the study are exclusively based on OpenAI models.\n- While the classification step for SQL difficulty proves effective on the Spider dataset, it is unlikely to exhibit strong generalization capabilities to other real-world Text2SQL data. The classification of SQL difficulty may be influenced by annotation artifacts specific to the Spider dataset.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The first, second, and third weaknesses mentioned by the reviewer have been thoroughly addressed in the general response to all reviewers.\n\nRegarding the weakness related to other LLMs beside OpenAI models, at the time of writing this paper, GPT-based models, PaLM, and the OPT model were the widely used language models. Access to other LLMs besides the GPT family was limited as they were not accessible through API calls. Moreover, due to the large number of parameters, we faced difficulties in loading those models onto our available infrastructure.\n\nConcerning the last weakness regarding the applicability of our proposed approach to real-world scenarios with more complex SQL query structures, our current classification structure effectively handles numerous benchmarks in the text-to-SQL domain and real-world applications. To accommodate more complex scenarios, we can readily add another class for intermediate steps and include additional steps in the existing most complex class. As the Spider dataset already contains the most complex SQL queries, it serves as a valuable benchmark for assessing intricate SQL structures.\n\n\n\nQuestion: How did the authors come up with the self-correction prompts? Was it through a lot of trial and error from the training and validation set?\n\nAnswer:  The concept of using LLMs to correct themselves was inspired by an application mentioned on the OpenAI website. LLMs were suggested as a tool for debugging codes. Recognizing the LLMs' potential in debugging, we proposed utilizing them to self-correct their own responses. This approach aligns with the ideas presented in independent work conducted around  the same time as ours, such as the paper \"self-debug\" by Google DeepMind, where LLMs were employed to debug their generated SQL queries and Python codes.\n\nQuestion: How did the author decide on this set of in-context learning exemplars?\n\nAnswer: The examples in our prompts are deliberately selected to include at least one instance for almost all widely used SQL keywords. This careful selection ensures that the LLM has access to all the necessary information before generating the SQL query."}]

Author rebuttal:
Our method draws inspiration from chain-of-thought and decomposed prompting techniques and brings valuable contributions to prompting across various domains, including text-to-SQL. These contributions are as follows:
1) Adaptive Prompting Based on Task Complexity: Our technique involves classifying the input task complexity and adjusting the prompt complexity accordingly. Tailoring prompts based on input question complexity outperforms using generic or overly complex prompts, as demonstrated in Table 4. Moreover, this classification based on task complexity can reduce the number of tokens that are used for simple questions, hence reducing the cost.
2) Schema-Linking Module: Our work introduces a schema-linking module, which has not been utilized in the context of prompting approaches. Inspired by our work, a few other prompting methods (e.g. C3 paper and LangChain SQLAgent) boost their performance using schema linking. The schema-linking template we propose is optimized through numerous iterations to mimic human thought processes.
3) LLMs for Self-Correction: Our research is among the first to propose using Language Models (LLMs) to self-correct their generated responses. The effectiveness of this method is demonstrated by other independent work conducted around the same time as ours, including the self-debug paper, across various domains, not limited to text-to-SQL. Our approach influenced the widely used text-to-SQL agent of LangChain, with practical applications in the industry.
In addition to the aforementioned contributions in the prompting techniques domain, our method stands out with the highest performance, surpassing not only fine-tuning approaches but also other prompting methods.

For our evaluation, we utilized the Spider dataset, a comprehensive cross-domain benchmark with databases from various domains. Given its extensive coverage, a successful approach on this dataset is expected to perform well across all domains. There are also a few other datasets that are derived from or are based on Spider:
1) Spider-Syn replaces question terms in Spider with synonyms. As reported in the Spider-Syn paper, 99% of those modifications are done on Schema words and the remaining 1% on cell values.
2) Spider-DK modifies the questions in Spider by adding domain knowledge or question paraphrases. For example, "in the order of birth date" in Spider is replaced with "order of their birth date from old to young" in Spider-DK, and "dog" is replaced with "abandoned dogs".
3) Spider-realistic modifies the questions in the Spider dataset to remove the explicit mentions of column names.
4) Dr.Spider applies perturbations to Spider tabases, natural language questions, and SQL queries to measure the robustness of the models.

We did not evaluate our work on these variants for two reasons: 
1) Many of these benchmarks share a similar SQL query structure with the Spider dataset but extend the benchmark in one direction. Based on our experience with these datasets, the modified queries in these variants tend to be less complex, and our classification and decomposition techniques are expected to do well on these variants. Also, with the wealth of information stored within the parameters of LLMs, and the less reliance of our method on the training data, our model is expected to do well on these variants compared to fine-tuned models that heavily rely on the training data. For example, LLMs have demonstrated exceptional paraphrasing capabilities, making them resilient to synonym substitutions. Experimental results in the paper "A comprehensive evaluation of ChatGPT’s zero-shot Text-to-SQL capability" support these claims, as the model's performance remained stable under various perturbations to the natural language questions.
2) Cost was the main factor in evaluating our model. We had to use the OpenAI API calls in our evaluation, and the cost of making those API calls with the large number of queries in the Spider dataset was not cheap. Running our model on other variants would have doubled or tripled the cost.

Although we could not provide results for these other datasets due to the expensive nature of employing GPT-4, the resilience of the previous prompting method on these variants suggests a similar trend to hold for our method.

BIRD, another cross-domain dataset similar to Spider, has been very recently proposed.  At the time of writing this paper, the BIRD dataset was not published, hence it was not included in our study. This dataset intentionally focuses on database values and external knowledge provided in queries, whereas our queries are generated independent of database values and only based on database schema for generality reasons (as discussed in the paper). Our prompts for this dataset will need to include such information to be effective. We are currently working with the authors of BIRD to evaluate our model on their datasets.

Meta-review: This paper presents a group of techniques using few-shot prompting LLMs for text-to-SQL by decomposing the task into smaller sub-problems including schema linking, query classification and decomposition, SQL generation, and self-correction. It achieves state-of-the-art execution accuracy on Spider, a well-studied established benchmark for text-to-SQL. While reviewers pointed out that this is a combination of known techniques for a single task with limited novelty and insights for other problems, it shows the great potential of using LLMs with prompting to solve text-to-SQL tasks better than specialized fine-tuning models. During rebuttal, the author further provided results on BIRD-SQL, a more recent and realistic text-to-SQL benchmark, to dismiss reviewers' concerns on its generalizability and usability when evaluating only on Spider, which is appreciated by reviewers.
Decision: Accept (poster)