Title: DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
Venue: NeurIPS 2023
Accepted: yes
Keywords: In-Context Learning

Text-to-SQL

Task Decomposition

Spider Challenge

Natural Language Interfaces to Databases
Abstract: There is currently a significant gap between the performance of fine-tuned models and prompting approaches using Large Language Models (LLMs) on the challenging task of text-to-SQL, as evaluated on datasets such as Spider. To improve the performance of LLMs in the reasoning process, we study how decomposing the task into smaller sub-tasks can be effective. In particular, we show that breaking down the generation problem into sub-problems and feeding the solutions of those sub-problems into LLMs can be an effective approach for significantly improving their performance.  Our experiments with three LLMs show that this approach consistently improves their simple few-shot performance by roughly 10%, pushing the accuracy of LLMs towards SOTA or surpassing it. On the holdout test set of Spider, the SOTA, in terms of execution accuracy, was 79.9 and the new SOTA at the time of  this writing using our approach is 85.3. Our approach with in-context learning beats many heavily fine-tuned models by at least 5%. Additionally, when evaluated on the BIRD benchmark, our approach achieved an execution accuracy of 55.9%, setting a new SOTA on its holdout test set.

DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction

Abstract

There is currently a significant gap between the performance of fine-tuned models and prompting approaches using Large Language Models (LLMs) on the challenging task of text-to-SQL, as evaluated on datasets such as Spider. To improve the performance of LLMs in the reasoning process, we study how decomposing the task into smaller sub-tasks can be effective. In particular, we show that breaking down the generation problem into sub-problems and feeding the solutions of those sub-problems into LLMs can be an effective approach for significantly improving their performance. Our experiments with three LLMs show that this approach consistently improves their simple few-shot performance by roughly 10%, pushing the accuracy of LLMs towards SOTA or surpassing it. On the holdout test set of Spider, the SOTA, in terms of execution accuracy, was 79.9 and the new SOTA at the time of this writing using our approach is 85.3. Our approach with in-context learning beats many heavily fine-tuned models by at least 5%. Additionally, when evaluated on the BIRD benchmark, our approach achieved an execution accuracy of 55.9%, setting a new SOTA on its holdout test set.

1 Introduction

Natural language interfaces to databases aim at making it easier for end users to access data in a relational database. For example, given the utterance 'find employees who make more than their managers' and the schema of tables employees and manages , one may want to generate a query in SQL that retrieves those employees from a database. Over the past two decades, research in this field has progressed through several phases, with early systems being domain-specific, supporting controlled natural language [Popescu et al., 2003, 2004, Li et al., 2007, Li and Jagadish, 2014] or relying on rule-based approaches [Stratica et al., 2005] while more recent systems offering greater domain-independence using supervised models trained on diverse domains and datasets [Zhong et al., 2017, Yu et al., 2018] and more recently deep neural models trained on large text and code repositories [Dong and Lapata, 2016, Devlin et al., 2018].

The latest development in this progression is the use of Large Language Models (LLMs) under zero-shot and few-shot prompting [Rajkumar et al., 2022, Liu et al., 2023a]. It has been shown that LLMs provide strong baselines using only a few demonstrations and no fine-tuning [Chen et al., 2021, Brown et al., 2020, Liu et al., 2023b]. However, these models fall behind on commonly used benchmarks (e.g., Spider) compared to well-designed and fine-tuned models. Table 1 shows the performance of two latest LLMs, CodeX and GPT-4, on the development set of the Spider dataset. Despite a strong performance, LLMs fall behind, compared to existing methods [Scholak et al., 2021, Li et al., 2023a], especially on medium and complex queries. The question investigated in this paper

is where these LLMs fail and if some of the problems that they are facing can be mitigated to push the performance to reach or surpass fine-tuned SOTA models.

Prompting has several advantages over traditional approaches using pretraining or fine-tuning. The main benefit is that LLMs can perform prediction tasks without requiring large task-specific training data. Training models from scratch or fine-tuning them is a resource-intensive process, often requiring a large number of training samples and machine resources, which may not be available. Additionally, few-shot prompting has been shown to outperform previous state-of-the-art methods on several benchmark datasets and can achieve high accuracy even with limited training examples [Brown et al., 2020, Wei et al., 2022b].

It has been recently shown that the performance of LLMs can be improved on more complex tasks (e.g., math word problems, compositional navigation steps) using approaches such as chain-of-thought [Wei et al., 2022b], least-to-most [Zhou et al., 2022], and decomposed [Khot et al., 2022] prompting techniques where a task is broken down into multiple steps and the intermediate results are used to generate a final answer. Unlike algebraic expressions, which consist of clear steps or operations, breaking a complex SQL query can be a more daunting task because of the declarative structure of the language and the complex relationships between query clauses.

In this paper, we propose a novel method based on fewshot prompting that decomposes the task of natural language text to SQL (referred to as text-to-SQL) into multiple sub-tasks. Previous works on text-to-SQL prompting using LLMs are only evaluated in a zero-shot setting [Rajkumar et al., 2022, Liu et al., 2023a]. However, zero-shot prompting only provides a lower bound on the potential power of LLMs for most tasks [Zhang et al., 2022, Kojima et al., 2022, Wei et al., 2022b, 2021, Brown et al., 2020]. We show that our proposed method outperforms the few-shot prompting method by a large margin. We also

compare our method with previous approaches on two cross-domain challenging benchmarks, Spider and BIRD. For Spider dataset, we use the two official evaluation metrics of execution accuracy and exact set match accuracy [Zhong et al., 2020]. We utilize two variants of the CodeX family, namely Davinci and Cushman [Chen et al., 2021], and the GPT-4 model for prompting. On the holdout test set of Spider, our method achieves an execution accuracy of 85.3% and 78.2% respectively using GPT-4 and CodeX Davinci models and an exact set match accuracy of 60% and 57% respectively using the same models. The large gap between the exact match and execution accuracies is due to the few-shot in-context nature of our method. Pretrained and fine-tuned approaches are more likely to generate SQL queries with a higher exact set match accuracy simply because these models have seen many examples during training that follow the composition style of the queries in the test set (queries in both sets are often written by the same people). Before our work, the SOTA on the test set had an execution accuracy of 79.9% [Li et al., 2023a] and an exact set match accuracy of 74% [Li et al., 2023b], and our method sets a new ground in terms of the execution accuracy. On the BIRD benchmark, our approach achieves a new SOTA result, attaining an execution accuracy of 55.9% on the holdout test set and 50.72% on the development set when employing GPT-4. Moreover, using the valid efficiency score introduced in this benchmark, our approach outperformed a GPT-4 baseline, demonstrating a 9% improvement on the development set. This highlights the effectiveness of our method.

Our contributions can be summarized as follows: (1) improving the performance of LLM-based textto-SQL models through task decomposition, (2) introducing adaptive prompting strategies tailored to task complexity, (3) addressing schema linking challenges in the context of prompting, and (4) using LLMs for self correction. To replicate the reported results, visit our GitHub repository 1 for access to the prompts, results, and the code.

2 Related Work

Sequence-to-sequence models [Sutskever et al., 2014] have shown great potential in code generation tasks including text-to-SQL. The key idea is to jointly encode a given natural language question and the database schema and leverage a decoder to predict the target SQL.

On the encoder side, learning a representation for the question and the database schema is carried out using bidirectional LSTM in IRNet [Graves and Graves, 2012], convolutional neural networks in RYANSQL [Choi et al., 2021], pretrained language models such as BERT in SQLova [Hwang et al., 2019] and graph neural networks in RATSQL [Wang et al., 2019], SADGA [Cai et al., 2021], and LGESQL [Cao et al., 2021]. Gan et al. [2021] propose an intermediate representation to bridge the gap between the natural language question and SQL statements. There has been also work on tabular language models that encode both tables and text such as TaBERT [Yin et al., 2020], TaPas [Herzig et al., 2020], and Grappa [Yu et al., 2020].

The methods on the decoder side can be categorized into sketch-based slot-filling and generationbased methods [Qin et al., 2022]. Sketch-based methods break the problem into several slot prediction sub-problems and aggregate the predictions for the slots of the SQL query to be generated [Hwang et al., 2019, Xu et al., 2017, Hui et al., 2021]. A drawback of these methods is that they cannot generalize to queries that do not follow the predefined templates. The generation-based methods [Guo et al., 2019, Wang et al., 2019, Cao et al., 2021, Huang et al., 2021] decode the SQL query as an abstract syntax tree.

In contrast to pretrained and fine-tuned models, Rajkumar et al. [2022] and Liu et al. [2023a] conduct an evaluation of the zero-shot prompting capability of LLMs on text-to-SQL using different prompts on the Spider dataset. Prompting techniques have been also used for tasks such as table understanding, table reasoning, and table-to-text generation [Guo et al., 2023, Chen, 2022], and some remarkable results have been reported using LLMs with just a small number of examples given in the prompt.

3 Few-shot Error Analysis

To better understand where LLMs fail under a few-shot setting, we randomly sampled 500 queries from different databases in the training set of the Spider dataset, excluding all databases used in our prompts. We searched for the queries that produced results different than those of gold queries, hence failing the execution accuracy. We manually examined these failures and classified them into six categories as shown in Figure 1 and discussed next.

Schema linking This category contained the largest number of failed queries and included instances where the model failed to identify column names, table names, or entities mentioned in questions. In some cases, the query required an aggregation function, but a matching column name was chosen instead. For instance, the database schema for question 'What are the average and maximum capacities for all stadiums?' included a column named 'average', which was selected by the model instead of taking the average of the capacity column.

JOIN This was the second largest category and included queries that needed a JOIN but the model was unable to identify all the tables required or the correct foreign keys to join the tables.

GROUP BY This category included cases where the SQL statement required a GROUP BY clause, but the model either did not recognize the need for grouping or wrong columns were used for grouping the results.

Wrong cols

15%

Wrongtables

12%

Schema-linking

37%

Wrong

Wrong

cols

entities

13%

10%

JOIN

21%

Failures

Invalid 3%

Op 1%

Wrong

Cond

tables

Other

4%

8%

GROUP-BY

13%

DESC

13%

Nested

4%

Wrongcols

13%

DISTICT

8%

Not

Wrong

5%

detected

5%

Set

Op

sub-query

4%

9%

SQL generation

Table singer, columns=[singer_ID,..]

Easy

Table concert,columns=[concert_ID,...]]

Selectnamefromsinger

WhereCitizenship!=‘French'

Q:"whatarethenamesofthesingers

whoarenotFrenchcitizens?"

NestedComplex

Classification

Schema linking

&

Non-Nested Complex

Self-correction

Decomposition

Queries with nesting and set operations

For this category, the gold query used nesting or set operations but the model did not recognize the nested structure or was unable to detect the correct nesting or set operation.

Invalid SQL A small set of the generated SQL statements had syntax errors and could not be executed.

Miscellaneous This category included cases that did not fit under any of the previously mentioned categories. Examples included SQL queries that contained extra predicates, missed a predicate, or had missing or redundant DISTINCT or DESC keywords. This category also included cases where the WHERE clause was missing or the query had redundant aggregation functions.

4 Methodology

Despite improvements over zero-shot, few-shot models struggle on more complex queries including those where schema linking is less trivial and the queries that use multiple joins or have a nested structure, as discussed in Section 3.

Our approach to address these challenges is to break down the problem into smaller sub-problems, solve each sub-problem, and use those solutions to construct a solution for the original problem. Similar approaches (e.g., chain-of-thought prompting [Wei et al., 2022b] and least-to-most prompting [Zhou et al., 2022]) have been taken to improve the performance of LLMs on tasks that can be broken down into multiple steps such as math word problems and compositional generalization [Cobbe et al., 2021, Lake and Baroni, 2018]. Unlike these domains where the tasks have a procedural structure with one step directly feeding into the next step, SQL queries in most parts are declarative and the possible steps and their boundaries are less clear. However, the thought process for writing SQL queries may be broken down to (1) detecting database tables and columns that are relevant to the query, (2) identifying the general query structure for more complex queries (e.g., group by, nesting, multiple joins, set operations, etc.), (3) formulating any procedural sub-components if they can be identified, and (4) writing the final query based on the solutions of the sub-problems.

Based on this thought process, our proposed method for decomposing a text-to-SQL task consists of four modules (as depicted in Figure 2): (1) schema linking, (2) query classification and decomposition, (3) SQL generation, and (4) self-correction, which are explained in detail in the following sub-sections. While these modules may be implemented using techniques from the literature, we implement them all using prompting techniques to show that LLMs are capable of solving them all if the problems are simply broken down to the right level of granularity. The few-shot examples used in the prompts are obtained from the training set of the respective benchmarks.

4.1 Schema Linking Module

Schema linking is responsible for identifying references to database schema and condition values in natural language queries. It is shown to help with the generalizability across domains and the synthesis of complex queries [Lei et al., 2020], making it a critical preliminary step in almost all

(a) schema linking module

(b) classification and decomposition module

10 task-specific demonstrations

10task-specificdemonstrations

+

+

Table concert, columns = [concert_ID, concert_Name, ...]

Table advisor, columns = [s_ID, i_ID, ...]

Foreign_keys=[concert.Stadium_ID=stadium.Stadium_ID,...]

Foreign_keys = [management.head_ID = head.head_ID, ...]

Q:"Showlocationandnameforallstadiumswithacapacity

Q: "How many courses that do not have prerequisite?"

between5000and10000."

schema_links:[course.*,course.course_id =

A:Let'sthinkstepbystep.

prerequisite.course_id]

A: Let's think step by step.

Schema linking

Classification

&

Decomposition

Ilnthequestion"Showlocationandnameforallstadiumswitha

capacitybetween5000and100o0.",weareasked:

"location and name forall stadiums"so we need column =

The SQL query for the question "How many courses that do

[stadium.Location,stadium.Name]

not have prerequisite?"needs these tables=[course,

stadiumswitha capacity"soweneed column=

prerequisite], so we need JoIN.

[stadium.Capacity]

Plus, it requires nested queries with (INTERSECT, UNION,

Based on the columns and tables,we need these Foreign_keys =

EXCEPT, IN, NOT IN), and we need the answer to the

10.

questions =[which courses have prerequisite?"].

Basedonthetables,columns,andForeign_keys,Thesetof

possible cell values are = [5000,10000].

So, we need JoIN and need nested queries, then the the

So the Schema_links are: Schema_links: [stadium.Location,

SQLquery canbeclassified as"NESTED".

stadium.Name,stadium.Capacity,5000,10000]

Label:"NESTED"

existing text-to-SQL methods [Cao et al., 2021, Wang et al., 2019, Guo et al., 2019, Xuan et al., 2021]. This was also a single category with the largest number of failures made by the LLM in our case (Figure 2).

We designed a prompt-based module for schema linking. The prompt includes ten randomly selected samples from the training set of the Spider dataset. Following the chain-of-thought template [Wei et al., 2022b], the prompt begins with 'Let's think step by step,' as suggested by Kojima et al. [2022]. For each mention of a column name in the question, the corresponding columns and their tables are selected from the given database schema. Possible entities and cell values are also extracted from the question. Figure 3a illustrates an example and the full prompt can be found in Appendix A.3.

4.2 Classification & Decomposition Module

For each join, there is some chance that a correct table or join condition is not detected. As the number of joins in a query increases, the chance that at least one join fails to generate correctly increases. One way to alleviate the problem is introduce a module that detects the tables to be joined. Also some queries have procedural components such as uncorrelated sub-queries, which may be generated independently and be merged with the main query.

To address these issues, we introduce a query classification and decomposition module. The module classifies each query into one of the three classes: easy, non-nested complex and nested complex. The easy class includes single-table queries that can be answered without join or nesting. The non-nested class includes queries that require join but no sub-queries, and the queries in the nested class can contain joins, sub-queries and set operations. The class labels are important for our query generation module, which uses different prompts for each query class. In addition to class labels, query classification and decomposition also detects the set of tables to be joined for both non-nested and nested queries as well as any sub-queries that may be detected for nested queries. Figure 3b shows an example input given to the model and the output that the model generates.

4.3 SQL Generation Module

As the queries become more complex, additional intermediate steps must be incorporated to bridge the gap between the natural language question and the SQL statement. This gap, known as the mismatch problem in the literature [Guo et al., 2019], poses a significant challenge to SQL generation, which stems from the fact that SQL is primarily designed for querying relational databases and not

representing the meaning in natural language [Kate, 2008]. While more complex queries can benefit from listing the intermediate steps in a chain-of-thought style prompting, such listings can degrade the performance for simpler tasks [Wei et al., 2022b]. On the same basis, our query generation comprises of three modules, each geared toward different classes.

For questions in our easy class , a simple few-shot prompting with no intermediate steps is adequate. The demonstration for an example E j of this class follows the format < Q j , S j , A j >, where Q j and A j give the query text in English and SQL respectively and S j indicates the schema links.

Our non-nested complex class includes queries that require join. Our error analysis (§ 3) revealed that finding the right columns and foreign keys to join two tables can be challenging for LLMs under simple few-shot prompting, especially when the query requires joining multiple tables. To address this issue, we resort to an intermediate representation to bridge the gap between queries and SQL statements. Various intermediate representations have been introduced in the literature. In particular, SemQL [Guo et al., 2019] removes operators JOIN ON, FROM, and GROUP BY, which have no clear counterparts in natural language queries, and merges the HA VING and WHERE clauses. NatSQL [Gan et al., 2021] builds upon SemQL and removes the set operators. Expressions in natural language queries may not clearly map to a unique SQL clause or they may map to multiple clauses, so removing operators makes the transition from natural language to SQL easier. As our intermediate representation, we use NatSQL, which is shown to have a state-of-the-art performance when combined with other models [Li et al., 2023a]. The demonstration for an example E j of the non-nested complex class follows the format < Q j , S j , I j , A j >, where S j and I j respectively denote the schema links and the intermediate representation for the jth example.

Lastly, the nested complex class is the most sophisticated type and requires several intermediate steps before generating the final answer. This class can contain queries that not only require sub-queries using nesting and set operations such as EXCEPT, UNION, and INTERSECT but also multiple table joins, same as the previous class. To break down the problem further into multiple steps, our prompt for this class is designed in a way that the LLM should first solve the sub-queries, generated from the previous module, and then use them to generate the final answer. The prompt for this class follows the format < Q j , S j , < Q j 1 , A j 1 , ..., Q j k , A j k > , I j , A j >, where k denotes the number of sub-questions, and Q j i and A j i respectively denote the i -th sub-question and the i -th sub-query. As before, Q j and A j denote the query in English and SQL respectively, S j gives the schema links and I j is a NatSQL intermediate representation.

Full prompts for all three query classes are provided in Appendix A.4, and all examples for the three classes are obtained from the exact same training set database chosen for the classification prompt.

4.4 Self-correction Module

The generated SQL queries can sometimes have missing or redundant keywords such as DESC, DISTINCT and aggregation functions. Our experience with multiple LLMs indicates that these issues are less common in larger LLMs (e.g., queries generated by GPT-4 have less bugs than those from CodeX) but are still present. To address this, we propose a self-correction module where the model is instructed to correct those minor mistakes. This is achieved in a zero-shot setting, where only the buggy code is provided to the model and it is asked to fix the bugs. We propose two different prompts for the self-correction module: generic and gentle . With a generic prompt, we request the model to identify and correct the errors in the 'BUGGY SQL'. The gentle prompt, on the other hand, does not assume the SQL query is buggy, and instead asks the model to check for any potential issues and provides some hints on the clauses to be checked. Our evaluation indicates that a generic prompt can yield a better result with the CodeX model, while a gentle prompt is more effective for the GPT-4 model. Unless explicitly stated otherwise, the default self-correction prompt in DIN-SQL is set to gentle for GPT-4 and generic for CodeX. Examples of both generic and gentle self-correction prompts can be found in Appendix A.6.

5 Experiments

5.1 Models

We evaluated the proposed method using two variants of the CodeX family (Davinci and Cushman variants) and the GPT-4 model. These are the largest open-access LLMs at the time of writing this

paper. Smaller models are less applicable since prompting is believed to be an emergent ability of the LLMs with the number of parameters in the scale of billions [Wei et al., 2022a].

5.2 Hyperparameter

All models were accessed via the OpenAI API. Greedy decoding was used to generate the output by setting the temperature at zero. The max tokens was set to 350 for the self-correction module and 600 for all other modules. The stopping token sequence was set to '#;\n \n' for the self-correction module and 'Q:' for all other modules.

5.3 Dataset

Our evaluation was conducted on two crossdomain challenging datasets, Spider and BIRD. Spider consists of 10,181 questions and 5,693 unique complex SQL queries across 200 databases, covering 138 domains, each containing multiple tables. The standard protocol for this dataset divides it into 8,659 training examples across 146 databases, 1,034 development examples across 20 databases, and a holdout of 2,147 test examples across 34 databases. The databases used in each of these sets are nonoverlapping. SQL queries are categorized into four difficulty levels, based on the number of SQL keywords used, the presence of nested subqueries, and the usage of column selections and aggregations. BIRD comprises an extensive dataset with 12,751 unique question-SQL pairs, encompassing 95 large databases totaling 33.4 GB in size. It spans a wide array of more than 37 professional domains, including blockchain, hockey, healthcare, and education. BIRD also introduces external knowledge as an additional resource to assist models in generating accurate SQL queries. Specifically four sources of external knowledge were introduced: numeric reasoning knowledge, domain knowledge, synonym knowledge, and value illustration. Notably, the SQL queries in

the BIRD dataset tend to be more intricate than those in the Spider dataset. Language models without access to database content often encounter challenges with schema linking. Therefore, our prompts for the BIRD dataset include sample rows from each table to aid the model in schema linking. Furthermore, we have concatenated the provided external knowledge for each question as a hint, placed immediately after each question. However, due to constraints such as limited context window size, the presence of external knowledge, and the inclusion of sample rows, we have had to reduce the number of demonstrations within the prompts for the BIRD dataset.

5.4 Metrics

The performance of our models are evaluated using the official metrics of each dataset: exact-setmatch accuracy (EM) and execution accuracy (EX) for Spider and valid efficiency score (VES) and execution accuracy (EX) for BIRD.

The exact-set-match accuracy (EM) treats each clause as a set and compares the prediction for each clause to its corresponding clause in the reference query. A predicted SQL query is considered correct only if all of its components match the ground truth. This metric does not take values into account. The execution accuracy (EX) compares the execution output of the predicted SQL query with that of the ground truth SQL query on some database instances. Execution accuracy provides a more precise estimate of the model's performance since there may be multiple valid SQL queries

for a given question, and exact set match accuracy only evaluates the predicted SQL against one of them. The Valid Efficiency Score (VES) is a metric designed to measure the efficiency of running the generated SQL queries. This metric is meaningful if the generated queries are correct, meaning their result matches that of the reference query. Therefore, the VES metric takes into account both the accuracy of the generated queries and their efficiency in terms of the execution time.

5.5 Results

5.5.1 Test set results

As shown in Table 2 for the holdout test set of Spider, our method achieves the highest execution accuracy using GPT-4 and the third-highest execution accuracy using CodeX Davinci among all officially published results at the time of this writing. This is achieved without even utilizing the database content. In terms of exact set match accuracy, our method achieves comparable results to previous works that do not utilize database content. As demonstrated in Table 3a, in the case of the BIRD dataset, our method using GPT-4 achieved a test set execution accuracy of 55.9%, setting a new SOTA.

5.5.2 Development set results

Most of our evaluation during development was conducted on the development set of Spider which was easily accessible unlike the test set that was only accessible through an evaluation server provided by Yu et al. [2018]. Table 4a shows the performance of our method using different LLMs, compared to zero-shot prompting of Rajkumar et al. [2022] and Liu et al. [2023a] and our own few-shot prompting. To ensure a fair comparison for the few-shot prompting, we incorporate all the examples utilized for our three classes (easy, non-nested complex, and nested complex) inside the prompt. Given that the CodeX Cushman model has a smaller input context size than the CodeX Davinci and the GPT-4 models, we only use 2 examples from each class (for a total of 6 examples).

Our method significantly outperforms both simple few-shot prompting and zero-shot prompting, in terms of both exact set match and execu-

Few-shotprompt

DIN-SQL

70

60

Number of data samples

50

40

30

20

10

0

Schema

NIOr

GROUP

Nested

Invalid

other

Failure categories

tion accuracies, and the improvement is consistent across all models despite their sizes. For example, compared to few-shot prompting, our method improves the execution accuracy for all models by at least 10%.

On the development set of BIRD, our approach demonstrates a substantial improvement, achieving a 4% gain in execution accuracy and a remarkable 9% improvement in valid efficiency score over a GPT-4 baseline Li et al. [2023c], establishing a new SOTA. These and other results are reported in able 3b.

The performance of our method on the test set (as reported in Tables 2 and 3a) is higher than that on the development set for both Spider and BIRD. It is hard to pinpoint the exact reason when the test set is hidden, but we speculate that fewer questions in the test set may require the knowledge of the database content, making it easier for our method to predict a correct SQL query. Furthermore, the development set has schema ambiguity (e.g., a query entity can be mapped to multiple database entities but only one is considered correct), and it is possible that the test set has less ambiguity.

We further analyzed the performance of our proposed method on queries with different levels of difficulty. Table 4b presents the performance of our proposed method compared to a basic few-shot prompting on the development set of Spider. Our proposed method outperforms the basic few-shot prompting across all difficulty levels, with the greatest improvement in performance observed for the extra hard and hard classes where the few-shot prompting performed poorly. Our improvement

(a) Execution accuracy (EX) and Valid Efficiency Score (VES) on the holdout test set of BIRD

(b) Execution accuracy (EX) and Valid Efficiency Score (VES) on the development set of BIRD

on the easy class (compared to basic few-shot) is due to incorporating schema links in the prompt, highlighting the importance of our schema-linking module.

(a) Performance compared to zero-shot and few-shot prompting using different LLMs on the dev set of Spider

5.5.3 Error improvements

In Section 3, we did an error analysis of basic few-shot prompting on 500 queries randomly chosen from the training set. To understand the degree those errors are resolved, we ran DIN-SQL on the same 500 queries. As shown in Figure 4, our proposed approach improves the performance for all categories with the largest improvement seen for the JOIN and Nested categories. Despite having an explicit module for schema-linking, the largest portion of failure cases still belong to this category.

5.6 Ablation study

In an ablation study, we evaluated our approach with and without each of the four modules. As shown in Table 5 for the CodeX Davinci model, excluding any of the modules leads to an overall decrease in performance, in terms of the execution accuracy.

More details emerge as we study the effectiveness of each module across different query classes. Schema linking helps all query classes with the least improvement for the hard class. Our inspection of a sample of the failed cases reveals that schema linking sometimes finds redundant links due to an ambiguity in the question or schema, and this can introduce redundant joins or output columns.

Without a classification, we had to use either a simple few-shot prompting or a decomposed chainof-thought (COT) prompting for all queries. The reported performance without a classification

module in Table 5 is for our comprehensive framework that includes all our components except classification. This means that the approach contains not only COT prompting but also Schema Linking, Self-Correction, and NatSQL Intermediate Representation, all of which are significant contributions of our work. The decomposed chain-of-thought result presented in this table refers to employing the most complex prompt, developed for the nested complex class, for all questions instead of adopting a classification-based approach to determine prompt complexity based on the question's level of difficulty. In contrast, the result for the DIN-SQL with simple few-shot prompting refers to using the simplest prompting class, easy class, for all questions across different level's of difficulty. As expected, a decomposed chain-of-thought prompting works better for hard and extra hard queries whereas a simple few-shot works better for the easy class.

For self-correction, we ran our study using both CodeX Davinci and GPT-4. For CodeX Davinci, a generic self-correction prompt helps the model across all query classes. A gentle self-correction prompt is also helpful but the gain is smaller than generic one for CodeX Davinci. However, there is less chance that GPT-4 generates a buggy code, and giving a generic prompt of 'Buggy SQL: . . . Fixed SQL: . . . ' can hurt the performance. A gentle prompt work better for GPT-4 and improves the perfromance across all of the classes except the easy class.

6 Conclusions

Prompting has enabled large language models to achieve impressive performance on numerous NLP tasks across different domains, without requiring a large training set. Prior to our research, the effectiveness of prompting methods utilizing LLMs for the text-to-SQL task was inferior to that of models fine-tuned for the task. To bridge this gap, we have devised a decomposition technique to tackle some of the challenges that caused this disparity. Our extensive experiments on two challenging datasets of Spider and BIRD show that our method significantly improves the performance of prompting across all query classes, producing comparable or even superior results to state-of-the-art fine-tuned approaches.

7 Limitations

There are some limitations or areas of improvement to this work. Our manually constructed demonstrations are fixed for each query class. Future research may explore adaptive and automated methods for generating demonstrations at finer granularities, which can further enhance the performance of our approach. Additionally, as of the time of writing this paper, our proposed approach, characterized by its decomposed and step-by-step structure, incurs a cost of approximately $0.5 and exhibits a latency of approximately 60 seconds when responding to a natural language question from the Spider dataset using GPT-4. We anticipate that as LLMs continue to advance, these costs and latencies should decrease, but reducing the cost is another possible direction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn their study on the Text2SQL task, the authors demonstrate the efficacy of breaking down the generation problem into sub-problems when utilizing LLMs to improve performance on the Spider dataset. The approach is simple yet effective on the Spider dataset. It yielded consistent improvements across three LLMs: GPT-4, Codex Davinci, and Cushman. Notably, the paper achieves a new SOTA performance on the Spider holdout test set.\n\nSTRENGTHS:\n- SOTA on Spider holdout test set\n- Approach consistently improve results on three OpenAI LLMs\n\nWEAKNESSES:\n- The analysis and experiments conducted in the study were solely focused on a single dataset, namely Spider.\n- The study does not provide sufficient evidence to support the generalizability of the proposed approach to Text2SQL tasks in general.\n- It is important to highlight the existence of several large-scale and robust Text2SQL datasets that could have strengthened the results in terms of generalizability. Notably, the BIRD-bench dataset (https://bird-bench.github.io/) provides an additional resource for evaluating the proposed approach. Furthermore, the Spider dataset has various variants, such as Spider-SYN, Dr.Spider, Spider-realistic, and Spider-DK, which offer opportunities to assess the effectiveness of the approach in different contexts and scenarios. Conducting experiments on these datasets would have enhanced the overall robustness and applicability of the findings.\n- The foundation models utilized in the study are exclusively based on OpenAI models.\n- While the classification step for SQL difficulty proves effective on the Spider dataset, it is unlikely to exhibit strong generalization capabilities to other real-world Text2SQL data. The classification of SQL difficulty may be influenced by annotation artifacts specific to the Spider dataset.\n\nQUESTIONS:\n1. How did the authors come up with the self-correction prompts? Was it through a lot of trial and error from the training and validation set? \n2. How did the author decide on this set of in-context learning exemplars?\n\nLIMITATIONS:\nNo negative societal impacts", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of text-to-SQL prediction using large language model (LLM) prompting. The authors propose DIN-SQL, a chain-of-thought (CoT) prompting method which decomposes the SQL prediction process into four substeps: schema linking, complexity classification, SQL prediction, and self-correction. For each step, a manually designed prompt with few-shot in-context learning samples are provided to the LLM to obtain the prediction. Especially, after complexity classification, samples with different complexity levels are addressed with different prompts. Experiments on the Spider dataset demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n- The overall pipeline is conceptually simple but practically effective. It appears to successfully leveraged the power of LLMs and achieved promising results.\n\n- The idea of applying different prompts for samples with different complexity is justifiable. It is based on prior observations that CoT prompts are effective on harder samples but can hurt the performance on easy samples. The ablation studies also echoed this observation, and verified the usefulness of this proposed mechanism.\n\nWEAKNESSES:\n- The proposed method does not carry much techinical innovation, as CoT is already a widely studied and applied technique, and the intermediate representation is imported from previous work NatSQL.\n\n- The complexity categorization and corresponding prompt designs are more or less tailored for the Spider dataset. It might be unclear whether the method is directly applicable in real-world scenarios, where the variety of target SQL might be much larger, and may include additional mechanisms such as self-join.\n\n- Based on my observations on the \"sub-questions\" generated for nested complex samples, it seems that many of them (though not all) are identical to the original questions. This might not be the desired behavior and leaves room for further improvements. For example:\n\t- \"Find the title, credit, and department name of courses that have more than one prerequisites?\" can be solved by knowing the answer to the following sub-question \"What is the title, credit value, and department name for courses with more than one prerequisite?\".\n\nQUESTIONS:\n- Around L231: The SQL-prediction step prompt for nested complex questions are designed in a comprehensive way, allowing multiple sub-questions to be included. However, looking at the samples in the appendix, it seems that at most one sub-question is predicted. Is the method actually able to handle very complex queries by generating >1 sub-questions?\n\nLIMITATIONS:\nSeveral limitations are discussed, although I think there are additional points as I mention above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed to decompose the text-to-SQL reasoning into multiple steps and solve them with large language models. The authors began by conducting an error study of LLMs with few-shot learning and identified the common errors, such as \"schema linking\" and \"JOIN\". To address these common errors, they proposed a new method called DIN-SQL which breaks the text-to-SQL task down into four modules: (1) schema linking, (2) query classification and decomposition, (3) SQL generation, and (4) self-correction. \n\nIn the schema linking and query classification modules, examples from the Spider dataset's training set are used to guide LLMs in identifying the relevant database schema and potential SQL structure. The SQL generation modules leverage different examples to handle various SQL structures. For more complex SQL structures, such as non-nested complex and nested-complex queries, demonstration examples with hand-written intermediate reasoning steps are utilized. At last, the self-correction module asks the LLMs to correct the predicted SQL if they realize that their initial generation was incorrect.\n\nThe proposed framework demonstrates remarkable performance, surpassing state-of-the-art models on the Spider test set.\n\nSTRENGTHS:\n1. The proposed method DIN-SQL demonstrates very impressive results on the hold-out test set in the Spider dataset, showing the effectiveness of DIN-SQL on Spider.\n2. The error analysis reveals the drawbacks of LLMs with standard few-shot demonstrations and the proposed method are designed to address these drawbacks.\n\nWEAKNESSES:\n1. The proposed methods seem highly tailored to the examples in the Spider dataset. Previous work [1-2] has highlighted the spurious correlations in the Spider dataset, such as the strong lexical matching between the database and the question. It seems important to evaluate the proposed method on other datasets such as Spider-syn [1], Spider-DK [2], Dr. Spider [3], or KaggleDBQA [4].\n\n2. The proposed SQL generation modules require hand-written intermediate reasoning steps. I don't think it is a problem given the limited number of examples requiring annotation but my concern is whether these examples will still serve as good demonstrations for the datasets other than Spider. I hope this can be addressed along with 1.\n\n[1] Gan, Yujian, et al. \"Towards robustness of text-to-SQL models against synonym substitution.\" arXiv preprint arXiv:2106.01065 (2021).\n[2] Deng, Xiang, et al. \"Structure-grounded pretraining for text-to-sql.\" arXiv preprint arXiv:2010.12773 (2020).\n[3] Chang, Shuaichen, et al. \"Dr. Spider: A diagnostic evaluation benchmark towards text-to-SQL robustness.\" arXiv preprint arXiv:2301.08881 (2023).\n[4] Lee, Chia-Hsuan, Oleksandr Polozov, and Matthew Richardson. \"KaggleDBQA: Realistic evaluation of text-to-SQL parsers.\" arXiv preprint arXiv:2106.11455 (2021).\n\nQUESTIONS:\nI am somewhat confused by the disparity between the results of DIN-SQL in the Spider development set and the Spider test set. On the test set, DIN-SQL demonstrates superior performance compared to the state-of-the-art model [1], achieving a margin of 5.4 points higher (85.3 vs. 79.9). However, on the development set, it falls short by 9.9 points (74.2 vs. 84.1) and is outperformed by other in-context learning methods [2]. Do you know what factors contribute to DIN-SQL's high performance on the test set but comparatively low performance on the development set?\n\n[1] Li, Haoyang, et al. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n[2] Ni, Ansong, et al. \"Lever: Learning to verify language-to-code generation with execution.\" arXiv preprint arXiv:2302.08468 (2023).\n\nLIMITATIONS:\nLooks good to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes to improve few-shot prompting LLMs for text-to-SQL task. It first provides detailed error analysis on existing few-shot prompting LLM approaches, into six categories. Then the paper proposes a new approach to decompose the task into a few sub-tasks, solve each task individually, and compose the subtask for the final answer. 

Experiments show the proposed approach achieve state-of-the-art on the Spider benchmark dataset.

STRENGTHS:
1. The result is solid, as it achieves state-of-the-art on the challenging Spider benchmark dataset. In addition, this paper provides detailed ablation study and analysis, therefore it's straightforward to understand the strength and weakness of the proposed approach. 
2. The paper is well-written, easy to follow the motivation and details.

WEAKNESSES:
Overall, my biggest concern is there's little additional novelty beyond "just another prompt engineering paper". It's expected that with larger LM, few-shot approaches could outperform existing fine-tuning based approaches. However, after reading this paper, it's still unclear whether scaling LLM can solve this text-to-SQL task and if not, where is the bottleneck. 

More specific concerns:
1. The manual error analysis (Section 3) only applies to Codex, it's unclear whether these errors still exist for larger language models such as GPT4. 
2. There's only marginal improvement compared with chain-of-thought prompting (Table 4). And I guess the margin would be even smaller if  we use GPT4 as LLM.

QUESTIONS:
1. What's the difference on errors for Codex and GPT4
2. What's the result on chain-of-thought promoting for GPT4. 
3. What do you think is the upper bound for this dataset, and do you think scaling LLM can achieve the upper bound?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper's contributions are the following:\n- The authors examined the common failure modes of doing text-to-SQL with few-shot prompting of LLMs: schema linking, JOIN, GROUP BY, nested queries and set operations, invalid SQL and miscellaneous.\n- The authors propose a method to do text-to-SQL by decomposing the task into 4 sub-problems, and solving each with a few-shot / zero-shot prompt to an LLM: schema linking, classification and decomposition, SQL generation, and self-correction. Their in-context learning method is reported to attain the highest execution accuracy on the test set of the Spider dataset, without making use of database content. \n- The work shows that LLMs can be used for text-to-SQL via prompting with performance equivalent to or better than methods that make use of fine-tuning.\n\nSTRENGTHS:\n- The proposed method is a good application of chain-of-thought style problem decomposition to in-context learning techniques for text-to-SQL. \n- The method is well motivated by first conducting an examination of common failure modes when using LLMs for text-to-SQL in the few-shot setting. Investigating the error improvements using their method in Figure 4 is a very clear way to show how their method helps with better in-context learning. \n- The paper is written in a largely clear manner that makes it easy to follow.\n- The paper's results are a relevant contribution to the text-to-SQL community, in that their performance is significantly better than other LLM techniques like those in Rajkumar et al., 2022 and Liu et al., 2023.\n\nWEAKNESSES:\n- There could have been more discussion about how the proposed four modules have been implemented without using prompting, to better situate the work in the literature.\n- The paper should have been clearer about how the intermediate representation of NatSQL bridges the gap between queries and SQL statements. An example each for the non-nested complex class and nested complex class would have been helpful in the main paper, instead of leaving it to the Appendix. In particular, it is not clear in the paper 1) how the intermediate representation is obtained, 2) how the removal of operators like GROUP BY or the WHERE clause from the syntax of the intermediate representation can help the LLM still generate the right SQL statements and 3) how the LLM is induced to solve sub-queries for the nested complex class.\n- The paper could have been clear about the latency of its proposed method. While high-performing, it is likely that making several sequential calls to an LLM will be high latency.\n\nQUESTIONS:\n- How is the intermediate representation obtained? I refer to Appendix A.5.2 to see that it is inserted into the prompt, but it is not clear where this is generated.\n- How do the authors think the removal of operators like GROUP BY or the WHERE clause from the syntax of the intermediate representation help the LLM still generate the right SQL statements?\n- Can the authors illustrate how a query is decomposed into sub-queries for the nested complex class? It is not clear to me still in Appendix A.5.3.\n- What is the latency of the proposed decomposed in-context learning method, and how does it compare to other methods (like RESDSQL-3B + NatSQL or Graphix-3B+PICARD)?\n\nLIMITATIONS:\nYes, limitations are adequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}