Exploring Chain of Thought Style Prompting for Text-to-SQL
Authors: Chang-Yu Tai
Ziru Chen
TIANSHU ZHANG
Xiang Deng
Huan Sun
Venue: EMNLP 2023 Main (2023)
Keywords: Text-to-Sql
Prompt engineering
Chain-of-Thought
Least-to-Most
Question Decomposition.
TL;DR: We systematically study how to enhance the reasoning ability of large language models for text-to-SQL parsing through chain of thought style prompting.

Abstract: In-context learning with large language models (LLMs) has recently caught increasing attention due to its superior few-shot performance on various tasks. However, its performance on text-to-SQL parsing still has much room for improvement. In this paper, we hypothesize that a crucial aspect of LLMs to improve for text-to-SQL parsing is their multi-step reasoning ability. Thus, we systematically study how to enhance LLMs' reasoning ability through chain of thought (CoT) style prompting, including the original chain-of-thought prompting and least-to-most prompting. Our experiments demonstrate that iterative prompting as in least-to-most prompting may be unnecessary for text-to-SQL parsing, and using detailed reasoning steps tends to have more error propagation issues. Based on these findings, we propose a new CoT-style prompting method for text-to-SQL parsing. It brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reasoning steps; 2.4 and 1.5 point absolute gains, compared to the least-to-most prompting method.

Exploring Chain of Thought Style Prompting for Text-to-SQL.
Chang-You You Tai", Ziru Chen"i, Tianshu Zhang, Xiang Deng, Huan Sun
The Ohio State University
(tai. 97, chen. 8336, Shang.115353 deng. 595, sun. sun.397} Cosu. edu
Abstract
costly obtain because annotating SQL queries requires programming expertise.
In-context learning with large language models (LLMs) has recently caught increasing atten- tion due to its superior few-shot performance on various tasks. However, its performance on text-to-SQL parsing still has much improvement. In this paper, we hypothesize that crucial aspect LLMs improve text-to-SQL parsing is their multi-step reason- ing ability. Thus, we systematically study how enhance LLMs' reasoning ability through chain of thought (CoT) style prompting, includ- ing the original chain-of- offtthought prompting (Wei al., 2022b) and least-to-man prompting (Zhou et al., 2023). Our experiments demon- strate that iterative prompting as Zhou et al. (2023) may be unnecessary for text-to-SQL parsing, and using detailed reasoning steps tends to have more error propagation issues. Based on these findings, we propose new CoT-style prompting method for text-to-SQL. parsing. It brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reason- ing steps; 2.4 and 1.5 point absolute gains, com- pared to the least-to-mos to-tost prompting method'.
As alternative to supervised learning, in- context learning (Brown al., 2020), emergent capability of large language models (LLMs), alle- viates the need for large-scale data. With only a few examples, in-context learning enables LLMs to demonstrate performance comparable to or even better than fully supervised models on many NLP tasks, such as question answering, machine transla- tion, and natural language inference (Chowdhery et al., 2022; Kojima et al., 2022; Wei al., 2022b.a; Brohan al., 2023). When applied to text-to-SQL. parsing, in-context learning has also shown encour- aging results (Rajkumar et al., 2022; Chang et al., 2023b; Liu et al., 2023a), but there is still much room for improvement.
for
We hypothesize that crucial aspect of LLMs to improve for text-to-SQL parsing is their multi- step reasoning ability. Even for a seemingly simple question, such as "What the ID of Kyle," a model has to ground it to the given database schema, infer the relational algebra among schema items, and construct syntactically correct SQL clauses. To enhance LLMs' reasoning capabilities, chain of thought (CoT) style prompting methods (Wei al., 2022b; Zhou et al., 2023) are proposed and have shown promising results. However, how to ap- ply CoT--tyle prompting to text-to-SQL parsing remains under-explored, and we fill this gap by sys- tematically exploring CoT-style prompting for text- to-SQL parsing. Specifically, we seek to answer two research questions: (RQI) Which prompting style better, generating all reasoning steps in one pass, or iterative prompting and problem solving? (RQ2) Do more detailed reasoning steps lead to better results for text-to-SQL. parsing?
1 Introduction
Text-to-SQL. parsing, the task of mapping natu- ral language utterance to a SQL query, has found wide applications in building language agents for databases and piqued significant research interest in recent years (Deng al., 2021; Yu al., 2021; Ra- jkumar et al., 2022: Hongjin et al., 2023; Ni et al., 2023). To develop a text-to-SQU. parser, a prevalent approach is to collect labeled data and train a model via supervised learning (Shaw et al., 2021; Scholak et al., 2021). While effective, this approach necessi- tates considerable amount of training data, which
To address these questions, we adapt two widely used prompting methods for text-to-SQI. parsing. As the first method, we apply chain-of ofin-oof-tuouhh prompting (Wei et al., 2022b) by drawing an anal-
"Equal Contribution.
"Corresponding authors.
We include all our prompts in the appendix.

---

Model Input (API Doo Format)
eeW SQUie SQL tables, with their properties:
P allegyy lype (olleryy, allegytype)
hoee, .ilerrrrr (stard, alberg))
student (slund, Iname, thome, ago, sex, major, advisor, city code)
#44 Show firl reme, lost reme, for all femella sludentss. Their sec F.
Expected Model Output Different Prompting Methods
Single-Pass Prompting
Iterative Prompting
age for A. Chalin-of-Thuur Thought
B. Least-to Most Prompting
This choeses the records from student table, followed WHERE that records Inams,
the Problom Reduction:
column
Ni answerr the question "Show first for femmae students, Their F., know: "Show nome, age all students."
Thus, arswer the question Show flrst famale studants, Thair attaac
<Model Imput>
inamn,
ODecomp (Ours)
Problom Solving:
1. Show firet name, lsol name, age for students.
2. Show first name,, bet neme, age for femede students, Their sux F.
<Modeel Input>
Thus, the enswer for the question Show first name, last name, age for female students. Their sox
Q: Show frrst name, lest name. age for all students.
LLM
SELECT inarme, Inare, agn FRON etucent
sElsci Iearso, inarso, aqu EACK studonz
WHERE sex 1pr
D.e.ammp InterCOL (Ours)
<Model Input>
Show
name, age
Q: Show finil name,, lest name, A:
bed rome, age table (solumn): student (vex)
table (column) student (fname, Iname,, ege)
students. Their
si students.
BELSCT age
age
Thus, answer the question Show first name, name, age female students. sex
Show name, name, aga all famalo studiants.
RELECT fnano, Inans, agc FRCK
student
sox
Their sei
Figure 1: Example model input and expected outputs for four CoT style prompting methods applied to text-to-SQI. parsing: A. Chain-of- Thoough,, B. Srast-to-Most. C. QDecomp, and D. QDecomp + InterCOL, where C and are our proposed methods.
ogy between its problem-solving process and the execution procedure of SQL query (Figure IA). Referring the logical execution order of SQL clauses (Narechania et al., 2021), we compose the intermediate execution steps in natural language and prompt LLMS to derive them before generating the SQL query. For the second method, we follow Zhou et al. (2023) to apply least-to-most prompting in two stages: (1) problem reduction: generate se- ries of sub-questions from the original question and (2) problem solving: iteratively translate each sub- question into its corresponding SQL query, with the original question as the last sub-question, as shown in Figure 1B. With a careful analysis (Sec- tion 5.2), we find that directly applying these two methods for text-to-SQL parsing tends to introduce error propagation issues frequently. Also, the itera- tive process in least-to- mosst prompting incurs more computational costs generate each SQL query.
language question in one pass. Instead of generat- ing the intermediate execution steps, we instruct LLMs to decompose the original complex question, akin to the problem reduction stage in least-to-most prompting. Furthermore, to help LLMs ground database schemas, we design a variant of question decomposition prompting (QDecomp+InterCOL, Figure ID) by incrementally including the table and column names involved in each sub sub-question.
We conduct comprehensive evaluations two cross-domain text-to-SQL datasets, Spider (Yu et al., 2018) and Spider Realistic (Deng al., 2021). Compared the standard prompting method without reasoning steps, QDecomp In- terCOL brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively. It also brings 2.4 and 1.5 point absolute gains compared to least-to-mon prompt- ing. Our results suggest that it may be unneces- sary to perform iterative prompting, which is also computationally costly (R2/). Besides, our anal- ysis shows that our @Decomp+IntrCCOL method reduces the chance of error propagation by provid- ing less detailed reasoning steps and generating
Therefore, we propose a new CoT-style prompt- ing method called question-decomposiiion prompt- ing (QDecomp, Figure IC). Similar chain- of-thought prompting, QDecomp generates a se- quence of reasoning steps followed by the natural

---

the SQL query in one pass (RQ2). Meanwhile, it includes key schema information reasoning, which is still beneficial to database grounding. Fur- ther, we evaluate the robustness of our proposed methods by varying the number, selection, and format of in-context examples, providing useful guidelines for designing text-to-QQL prompting strategies. We also extend our evaluation to three single-domain datasets (Zelle and Mooney, 1996; Iyer et al., 2017; Yaghmazadeh et al., 2017) and show our proposed method can achieve strong per- formance consistently across different datasets.
of-thought prompting and least-to-most prompt- ing on several text-to-SQL datasets. Our exper- iments validate our hypothesis that text-to-SQL parsing indeed requires multi-step reasoning, and carefully designed CoT-sttyle prompting can help LLMs achieve higher parsing accuracy.
Text-to-SQI. Semantic Parsing. Text-to-SQL. semantic parsing has long been studied build language agents for database applications (Dahl et al., 1994; Zelle and Mooney, 1996). Since the release of Spider (Yu et al., 2018), cross-database text-to-SQL benchmark, many parsers have been developed on top of language models to better un- derstand various database schemas (Wang et al., 2020; Yu et al., 2021; Deng al., 2021). Recent work starts to explore the potential of LLMs, such as Codex (Chen et al., 2021), in text-to-SQL pars- ing by including database schemas in the prompts (Rajkumar et al., 2022) or retrieving similar ques- tions as few-shot examples (Hongjin et al., 2023). Orthogonal to these methods, our question decom- position prompting teaches LLM to perform multi- step reasoning for text-to-SQL parsing without ad- ditional engineering efforts. With few in-context examples, an LLM, such as Codex in our experi- ments, can learn decompose natural language questions and predict table and column names (Sec- tion 3) incrementally in each step.
2 Related Work
LLM and CoT-Style Prompting. As large lan- guage models (LLMs) advance (Brown et al., 2020; Chowdhery et al., 2022), in-context learn- ing emerged as a new paradigm in natural language processing (Liu et al., 2023b). Although LLMs can achieve outstanding performance by prompting them with few-shot examples in context, they strug- gle with tasks that require multi-step reasoning. As a solution, Wei et al. (2022b) proposed chain-of- thought prompting. By explicitly describing inter- mediate reasoning steps answer complex ques- tion in the prompts, chain-of thain-of-thour of-thought prompting improves the accuracy of LLMs by a large mar- gin across many natural language reasoning tasks. Besides, Zhou et al. (2023) proposed least-to-most prompting to solve complex problems in two stages. The method first prompts L.MSs to generate list of sub-questions as decomposition of the given prob- lem. Then, it uses the sub-questions guide LLMs incrementally solve each of them and derive a correct final answer. However, how to apply these two CoT-style prompting methods to text-to-SQL. parsing remains expereex.lored.
Our method demonstrates comparable perfor- mance to RASAT+PICARD (Qi et al., 2022), fine-tuned text-to-SQL. parser, on the Spider devel- opment set without using relational structures constrained decoding. Compared to other LLM- based methods, achieves better execution accu- racy than DIN-SQL (Pourreza and Rafiei, 2023) on the Spider development set in a single pass, while DIN-SQL requires iterative prompting. Although our method shows lower execution accuracy than LEVER (Ni et al., 2023), we note that LEVER s verifier model is fine-tuned on the full Spider train- ing set, which may have extra advantages over our method. Also, LEVER uses the execution results of SQL queries, which provides extra information for better database grounding. We leave the incorpora- tion of database contents beyond table and column names into our method as future work.
We fill this gap by systematically exploring sev- eral CoT-style prompting methods for the task. In particular, we propose a new CoT--tyle prompting method that guides LLMs to perform reasoning via question decomposition. Question decomposition is a method that converts a complex problem into a sequence of simpler sub-questions (Gupta and Lewis, 2018; 88 Min et al., 2019). Our work refers to existing question decomposition methods for text- to-SQL parsing (Wolfson et al., 2020, 2022) and presents a novel CoT-style prompting method to improve I.MM'' performance. We conduct com- prehensive experiments and show that our ques- tion decomposition prompting outperforms chain-
3 Prompting for Multi-Step Reasoning in Text-to-SQL
In this section, we outline three CoT-style prompt- ing methods that teach an LLM to perform

---

multi-step reasoning. We first describe how we adapt chain-of-thought and least-to-most prompt- ing for text-to-SLL parsing. Then, we pro- pose a novel prompting method, question de- composition prompting (QDecomp), and its vari- ant QDecomp+InterCOL. Figure demonstrates different prompting methods, and we provide more examples in Appendix A. For all ex- periments, we use Codex (Chen al., 2021), code-davinci-000 as the LLM. The experi- ments were conducted between January and March 2023 through OpenAl API, using greedy decoding with temperature
(3) For each decomposition, we remove words and phrases that may leak the information in any subsequent questions. This segmentation allows the LLM to focus on parsing each sub- quesquuston,, thereby decreasing the complexity of the original problem (Wolfson et al., 2022).
For instance, the question "Show first name, last name, age for all female students. Their sex is F." in Figure 1B would derive two sub-questions: (a) "Show first name, last name, age for all students." (b) "Show first name, last name, age for all female students. Their sex is F." This decomposition fol- lows principle (1) and (3) by removing the second sentence and the information-leaking word "female" from the original question construct the first step. For the first sub-question, the LLM only needs to construct the SELECT and FROM clauses. Then for the second sub-question, the LLM can build upon the SQL query generated for the first sub- sub-question, and focus solely on the WHERE clause.
3.1 Chain-of-Thought Prompting
Chain-of-thought prompting (Wei et al.. 2022b) aims to improve LLMs' reasoning ability by gener- ating series of intermediate steps before predict- ing the final answer. For text-to-SQL tQL parsing, one challenge is how come up with the reasoning steps to predict the SQL query (i.e., final answer in our case). In our work, we use each clause in the SQL query to compose reasoning step in CoT prompting. Specifically, inspired by Narechania et al. (2021), we use natural language templates to describe each SQL clause and chain them in the logical execution order of the SQL query. For example, the logical exccution order for the SQL query in Figure 1A is first the FROM clause, then the WHERE clause, and finally the SELECT clause. Following this order, we assemble the natural lan- guage description of each clause in the query to compose its CoT reasoning steps.
3.3 Question Decomposition Prompting
We propose new prompting method, question de- composition prompting (QDecomp). Similar to chain-of thought, ohain-of-thought,, QDecmmp generates intermedi- ate reasoning steps and the final SQL query in a single pass. Instead of using the logical execution procedure of SQL as in CoT, we follow the problem reduction stage in teast-to-most prompting and in- struct the LLM to decompose the original complex question as the reasoning steps. Through this de- sign, we hope to explore (1) the potential advantage of using question decomposition over the logical execution procedure of SQL clauses for composing reasoning steps; (2) whether an iterative process in least-to-most prompting is necessary.
3.2 Least-to-Most Prompting
Unlike chain-of-thought prompting, which in- structs LLMs to generate all reasoning steps in a single pass, least-to-most prompting (Zhou et al.. 2023) tackles complex questions by prompting LLMs in two stages: problem reduction and prob- lem solving. During problem reduction, it prompts the LLM to generate a series of sub-questions from the original complex question. During problem solving, it prompts the LLM with one sub-question at a time and iteratively builds up the final solution.
On top of that, we propose a variant, QDe- Comp+InterCOL, alleviate the well-known ta- ble/column linking issue in text-to-SQL parsing (Wang et al., 2020). Specifically, we augment the in-context examples to prompt the LLM to identify any corresponding table/column names when generating each sub-question. Given a sub- question and its corresponding SQL parse, we anno- tate all table-column pairs mentioned in the parse as ground-truth. For star operators (x), we sam- ple random column from tables mentioned in the same (sub-)query. If table-column pair has been mentioned in the SQL parse of sub-question, we would exclude it from the annotations of all subsequent steps. If a sub-question does not have
To derive the sub-questions for problem reduc- tion, segment the original question following three principles: (1) If the question has multiple sentences, we treat each sentence as a sub-question. (2) We further decompose each sentence by con- junction words (such as "and," " "or," and "but") and prepositions (such as "for," " "with," " and "without").

---

any table-column pairs annotate, we randomly choose one pair from preceding steps.
diffieulty- selection. In our main results, we use random selection for its simplicity and case of replication. Additionally, in Section 5.3, we com- pare results obtained using random selection with those obtained using difficulty-based selection.
We include examples of these two methods in Figure IC and ID. Following the same decom- position method in least-to-most prompting, the example has two sub-questions, In Figure ID, for the first sub sub-question, "Show first name, last name, age for all students," we expect the model to high- light the table "student" and its columns "fname," " "Iname," and "age," as they appear in the SQL parse this sub- question. Then, for the follow-up ques- tion, the model is expected to identify the table "student" and its column "sex," which is not men- tioned in the previous step.
For random selection, we uniformly sample in- context examples from the Spider training set at random. For diffieulty selection, we first group the Spider training examples into four diffi- cuity levels, pre-defined by Yu et al. (2018), includ- ing easy, medium, hard, and extra-hard. Then, we devise three methods to randomly select in-context examples based on their difficulties: (G1) sampling an equal number of examples at each difficulty level, (G2) sampling the same number of exam- ples from the hard level and the extra-hard level respectively, and (G3) sample all examples from the extra-hard level.
In addition to the prompting methods mentioned above, we also include the standard prompting method as the baseline in our experiments. uses question-SQL, pairs as in-context examples to prompt LLMs to directly parse a natural language question to its corresponding SQL query without generating any intermediate reasoning step.
4.3 Prompt Formats
We also experiment with two prompt formats in- troduced by Rajkumar et al. (2022), API Does and Create Table + Select 3. Both formats have their own advantages and can be utilized together with any prompting method in Section 3.
4 Experimental Setup
4.1 Datasets
A Does format represents database schemas as Python API comments, which only includes the table and column names. This format reduces the prompt length for each example, so we may include more in-context demonstrations from databases in different domains to increase diversity. In compari- son, Create Table Select 3 format adheres more closely to the SQLite standards, but with much longer prompts represents a database schema using the CREATE TABLE command, which pro- vides more information, such as column data types and foreign key declaration. Besides, this format in- cludes the results of executing SELECT * FROM LIMIT for each table T in the database SQL comments. In Section 5.3, we show that API Does format can achieve competitive performance compared the Create Table Select format. Thus, we primarily use the APT Does format our experiments due to its efficiency.
Spider (Yu et al., 2018). Spider is commonly used benchmark to evaluate text-to-SQL. parsing a cross-database setting, which requires mod- els to generalize to novel database schemas. The dataset consists of 7,000 question-query pairs in the training set and 1,034 pairs in the development set, covering 200 different databases and 138 do- mains. In this paper, due to the unavailability of the test set, evaluate on the Spider development set to demonstrate the effectiveness of our question decomposition prompting methods.
Spider Realistic (Deng et al., 2021). Spider Re- alistic is a more challenging version of the Spider development set. It modifies the natural language questions in Spider by removing or paraphrasing explicit mentions of column names to generate a more realistic dataset that reflects real-world sce- narios, where questions rarely contain explicit men- tions of column names. The final dataset comprises a total of 508 cnestion-query pairs.
4.4 Evaluation Metric
We use test-suite execution accuracy (Zhong et al.. 2020) to evaluate different prompting methods, in- context example selection strategies, and prompt
4.2 In-context Example Selection
To show the robustness of question decomposi- tion prompting, we consider two ways choos- ing in-context examples: random selection and
com/ www.///ampppppooccceccc es/ de fefult--ae sql -transtatt

---

Spider Dev
Spider Realistic
Method
Easy
Medium
Hard
Extra Hard
Overall TS (Overall EX)
Overall TS (Overall EX)
Standard
86.8
65.3
50.3
36.0
63.2 + 2.51 (68.7 + 4.08)
51.0 + 4.29 (62.5 = 4.01)
Chain-of Thought
73.9
64.5
44.6
23.4
56.8 :: 5.83 (53.9 + 7.21)
50.3 + 4.94 (53.4 9.19)
Least-to-Most
88.1
68.7
52.9
39.5
66.0 1 2.48 (68.9 1 3.44)
55.0 1 2.51 (63.3 - 2.73)
Least-to- Beast-to-Most (G3)
80.3
64.6
52.8
45.3
63.3 + 1.95 (73.8 +: 1.72)
Qeeommp
89.8
71.3
53.1
38.6
67.4 1 1.89 (70.7 2.80)
55.8 + 2.01 (65.8 1 2.29)
InterCOL.
89.6
74.1
52.4
38.1
68.4 + 2.05 (69.7 + 5.82)
56.5 + 2.05 (63.3 = 4.19)
InterCOL (G3)
88.7
71.1
56.8
45.7
68.8 + 1.16 (78.2 + 1.07)
Table 1: 8-shot test-suite (TS) accuracy of Codex on Spider Dev and Spider Realistic using different prompting methods and API Doc format. In-context examples are randomly selected except for the two rows marked with G3, where we only use extra-hard SQL queries (Section 4.2). We also include the overall standard execution accuracy (EX) in parenthesis for reference. For each method, we repeat the experiments with 5 different sets of in-context examples and report the average performances with their standard deviation. Wee were not able to run G3 example selection on Spider Realistic before Codex became unavailable.
SELECT
WHERE
GROUP BY
ORDER BY
KEYWORDS
Standard
89.8
66.1
83.0
84.2
Chain-of- Thoughh
70.7
72.8
76.9
Least-to-Most
90.0
70.7
72.5
82.4
84.3
QDecomp InterCOL 91.4
91.2
70.7
77.2
85.1
86.4
72.4
76.6
85.3
86.0
Table 2: 8-shot component matching accuracy of Codex the Spider development set.
formats. Leveraging the idea of "code coverage" in software testing (Miller and Maloney, 1963), the metric synthesizes a large number of databases as "test cases" and compares the execution results of
prompting and its variant (QDecomp+InterCO)) consistently outperform two existing methods, thain-of-thoughe and lcast-to-most prompting. Specifically, QDecomp+InterCOU achieves 68 08.4% test-suite accuracy on the Spider development set and 56.5% on the Spider Realistic set. Compared to the standard prompting, it brings 5.2% and 6.5% point absolute gains, respectively. Compared least-to-most prompting (the second best method), brings 2.4% and 1.5% point absolute gains. Fur- thermore, when using extra-hard (G3) in-context examples, we can improve the execution accu- racy of QDecomp+InterCOL prompting to 78.2%, which is comparable to RASAT+PICARD (Qi et al., 2022), strong fine-tuned text-to-SQL parser, In contrast, least-to-mos prompting does not gain too much execution accuracy (73.8%) from G3 in- context examples and even has decreased test suite accuracy (63.3%). We will present more analysis on this contrast in Section 5.3.
the predicted and gold SQL queries on all of them. In this way, test-suite accuracy reduces the num- ber of false positives (i.e., semantically different SQL queries that happen to have the execu- tion result) in standard exccution accuracy, which compares execution results on only one database. As shown in Table standard prompting's 's test- suite accuracy falls behind least-to-most prompting. However, their standard execution accuracy results are very close, which might be misleading.
5 Results and Analysis
By analyzing our experimental results, we seek to answer the following two research questions:
RQI: Which prompting style is better, generat- ing all reasoning steps in one pass, or iterative prompting and problem solving?
Additionally, the experiments show that iter- atively solving a series of sub-questions may not be necessary for text-to-SQL. parsing (RQI).
RQ2: Do more detailed reasoning steps lead to better results for text-to-SQL parsing?
Although chain-of-thought prompting (56.8%)
underperforms least-to-most prompting (66.0%)
5.1 Main Results
on the Spider development set, these methods have several distinct designs other than iterative prompting, so we cannot directly answer RQ/ by comparing them. With our QDecmmp prompting,
Through comprehensive experiments on Spider Dev and Spider Realistic (Table 1), we show that our proposed question decomposition (QDecomp)

---

Question
Which dogs are of rarest breed? Show their names and treatment dates.
Chain-of-
# This query chooses records from the dogs table, followed by joining the breeds table on the breed code columu. then joins the treatments table the dog id column. then groups the results by breed_nane It then orders the results by COUNT(DISTINCT TI (1.dg_id)) in descending order. It then selects the breed name and date, of treatment columns.
Thought
# Thus, the answer for the question is: Which dogs are of the rarest breed? Show their names and treatment dates, SELECT
T2. 12.bbree__name
T3. wtw.dat____trrtmeeeseeeeeee
FROM dogs AS T1
JOIN treatments AS T3 ON T1, rl.dog_io _id T3.dog_id GROUP BY T2, breed name ORDER BY
JOIN breeds
T2 T1 www.eeeeeoooooooooooooooeeoooooe code T2 12.bbeed_code
COUNT (DTSTTNCT T1 .1_dd__1o) DRSC
Gold
SELECT Ta.name, 22. date_aaaooooooo of_ of_treatment FROM dogs AS T1 JOIN Treatments AS 12 ON 11. v1.dog__ id - T2. d2. id WHERE "1.breed_cee code (SELECT breed_code
FROM Dogs GROUP BY greed_coct ORDER BY count ASC T.MMT
Table 3: An example error made by Codex with chain-of-though prompting.
Question
How many United Airlines flights to City 'Aberdeen??
Least-to-
Q: How many United Airlines flights go to City?
Most
A:SELECT count (+) FROM flights WHERE airline 'United Airlines' AND
destairport 'City' Q: How many United Airlines flights go to City 'Aberdeen'?
A:SELECT count (+) FROM flights WHERE airline 'United Airl ines' AND destairport = 'Aberdeen'
Gold
Q: How many United Airlines flights?
A:SELECT count (+) FROM flights AS T1 JOIN airlines AS T2 T2.uid fl.airline WHERE T2. airlins United Airlines' How many United Airlines flights go City 'Aberdeen'?
A:SELECT count (+) FROM flights AS T1 JOIN airports
desstairport 12. a2.airportccde JOIN airlines AS т3 ON 13.uid
m1.airlnne WHERE T2.city 'Aberdean'
a3. ine 'United
Airlines'
Table 4: example error made Codex with least-to-most prompting.
we show that generating sub-questions and the SQL query a single pass can also achieve improved ac- curacy. Thus, iterative prompting, which is compu- tationally costly, is not necessary when prompting LLMs to reason for text-to-SQL parsing.
also consider a component be correct if the whole SQL query's test-suite accuracy 1. As shown Table 2, our QDecomp and QDe- comp+InterCOL. prompts achieve better perfor- mance than other CoT-styye prompting methods across all five SQL components. Further analy- sis shows that chain-of-thought prompting under- performs standard prompting because provides very detailed reasoning steps. Translating such detailed steps error-prone and incurs more er- ror propagation issues. For example, in Table 3, Codex follows its reasoning steps faithfully to gen- erate the corresponding SQL query, but the reason- ing steps themselves have several errors, such choosing the "breed "breed_name" column instead of the "name" column in the SELECT clause. Least-to- most prompting makes improvements by providing reasoning steps at a higher level (via the problem reduction phase). However, sometimes still can- not translate a sub-question into the correct SQL
Another interesting finding is that chain-of- thought prompting performs
than the standard prompting method. We analyze the reason in the next section, which helps answer RQ2.
5.2 Error Analysis
We conduct a quantitative error analysis of all four prompting methods with the component matching accuracy (Yu et al., 2018) on the Spider develop- ment set. Component matching accuracy is finc- grained exact match metric that evaluates five SQL components, including SELECT clauses, WHERE clauses, GROUP BY clauses, ORDER clauses, and KEYWORDS (all SQL keywords, operators, and column names). Since exact match too strict, we

---

Spider Dev
Selection Method
Easy
Medium
Hard
Extra Hard
Overall
Random
89.6
74.1
52.4
38.1
68.4 + 2,05
GI
89.8
75.6
51.7
38.8
69.0 = 2.18
G2
87.4
72.2
50.4
39.4
66.9 - 2.31
G3
88.7
71.1
56.8
45.7
68.8 = 1.16
Table 5: 8-shot test-suite accuracy of Codex Spider Dev using QDecomp-InterCOL prompting with different in contexxx example selection methods.
Random
G1
G2
G3
8
Standard 63.2
64.1
60.2
58.2
Standard 59.6
62.0
63,9
63.2
Least-to-Most
65.8
62.6
61.2
63.3
Least-to-Most
59.2
62.1
65.8
Qecommp 67.4
68.2
65,2
66.6
QDecomp
63.1
66.6
67.4
InterCOL.
68.4
69.0
66.9
68.8
+ +InterCOL
61.4
66.5
68.4
Table 6: 8-shot test-suite accuracy of Codex on the Spider dev set using different in-context example selection methods.
Table 7:
Test-suite accuracy of Codex on the Spider dev set using different numbers of in-context examples. We do not have O-shot results for the proposed methods as they need at least one example to learn how to solve the task step by step.
query, especially when involving hard components, such as JOIN clauses, GROUP BY clauses, and ORDER BY clauses (Table 2). We include an er- ror example in Table 4. As a result, the errors are propagated to subsequent reasoning steps, leading to an incorrect final SQL parse. In contrast, QDe- comp+InterCOL prompting outperforms these two methods because it does not instruct Codex to gen- erate detailed reasoning steps or intermediate SQL queries. In this way, it reduces the possibility of accumulating mistakes in reasoning steps.
Codex obtains the best performance for hard and extra-hard SQL queries. We conjecture that QDe- comp+InterCOL teaches Codex become better at generating SQL queries at difficulty levels similar to the in-context examples.
Base on the conjecture, we extend this experi- ment to compare QDecomp+InterCOL and other prompting methods. As shown in Table 6. QDe- comp+InterCOL. prompting achieves the best per- formance across all settings, demonstrating its bustness. However, least-to-most prompting does not benefit from GI or G3 examples and shows decrcased accuracy. We believe this performance drop is because its iterative prompting generates one reasoning step at a time, which is relatively independent of the overall reasoning step length.
5.3 Robustness to Prompt Design
To further validate our conclusions in the main ex- periments, we conduct additional experiments to test the robustness of all four prompting methods in this section. Because chain-of of-thuhh prompt- ing already under-performs the standard prompting without reasoning, we omit this method in this and the next section.
Number of In- mn-Coneext Examples. Intuitively, performances all prompting methods improve as the number of in-context examples increases (Ta- ble 7). We found that our QDecomp prompting is the most robust and consistently achieves better performance than standard prompting. However, least-to-most prompting underperforms standard prompting when the number of examples less than 8. In addition, we note that our preliminary experiments, further increasing the number of ex- amples only leads to minor gains. Hence, we use 8 in-context examples in our main experiments.
Selection of In-Context Examples. Besides ran- dom selection, we evaluate the efficacy of QDe- comp+InterCOL and other prompting methods with in-context examples at various difficulty lev- els. As Table 5 suggests, QDecomp+InterCOL enables Codex to learn to reason for SQL queries at different difficulty levels from in-context examples. When using G1 examples, Codex learns to generate SQL queries and reasoning steps of various lengths from G1 examples. Thus, it is less likely gener- ate redundant SQL. clauses or reasoning steps and achieves the highest accuracy for SQL queries at easy and medium level. When using G3 examples,
Format of In-Context Examples. Finally, we show the performance of Codex using two prompt formats, API docs and Create Table Select 3

---

does
Create Table + Select
GeoQuery
IMDB
Yelp
MacroAvg
Standard 63.9
64.1
Standard
60.99
73.28
45.31
59.86
Least-lo-Most
62.1
63.8
Least-to-Most
60.99
58.78
36.72
52.16
QDccomp 66.6
66.2
Decommp InterCOL
64. 84
77.86
48 44
63,71
+ InterCOL 66.5
64.3
75.82
73.28
49.22
66.11
Table 4-shot test-suite accuracy of Codex on three other text-to-SQL datasets across different prompting methods.
Table 8: 4-shot test-suite accuracy of Codex on the Spider dev set using different prompt formats.
derived, it maintains a global view of all reasoning steps and mitigates such error propagation issues.
(Table 8). Due OpenAl's 's prompt length restric- tions, we use in-context examples this exper- iment. Although Create Table + Selcet format includes foreign key information and database con- tent, compared with API docs, it brings a negligible improvement in performance for standard prompt- ing and (slight) decrease for QDecomp and QDe- Comp+InterCOL. prompting methods. Nonetheless, QDecomp is still the best prompting method under this format. Therefore, we use API docs as our de- fault format due to efficiency and leave further experiments for future work.
Conclusion and Future Work
In this paper, systematically explore CoT-style prompting enhance LLMs' reasoning capability for text-to-SQL parsing. We design reasoning steps in order to apply two existing methods, chain-of- thought and least-to-most prompting, and propose new question decomposition prompting methods. Through comprehensive experiments, we demon- strate: (1) Iterative prompting may be not necessary for reasoning in text-to-SQL parsing. (2) Using de- tailed reasoning steps (in CoT) or intermediate SQL queries (in least-to-most prompting) is error-prone and aggravates error propagation.
5.4 Results on Other Text-to-SQL Datasets
Besides the Spider datasets, further compare QDecomp (+InterCOL) to standard and least-to- most prompting on other datasets including Geo- Query (Zelle and Mooney, 1996; Iyer et al., 2017), IMDB (Yaghmazadeh et al., 2017). and Yelp (Yagh- mazadeh et al., 2017). Since the database schema and SQL queries in these datasets are more com- plex than those in the Spider datasets, we also use 4-shot in-context examples in this experiment. As shown Table 9, QDecomp (+InterCOL.) consistently achieves the best performance for all three datasets. Moreover, we observe that least-to- most prompting underperforms standard prompting on IMDB and Yelp, which may be related to both iterative prompting and error propagation (Section 5.2). For example, least-to-moss prompting would decompose the question "Find all movies that were produced by Netflix" into two sub-questions: 1) "Find all movies" and 2) "Find all movies that were produced by Netflix." " Then, in the iterative solving stage, there are many correct SQL queries using dif- ferent tables and columns for the first sub-question. Without seeing the second sub-question, it is hard for the LLM to pinpoint the correct ones. As result, the LLM would include redundant or wrong schema items in the SQL parse for the first sub- question, which are propagated subsequent steps. Since QDccmmp (+InterCOL) instructs the LLM to generate the SQL query after all sub-questions are
Our question decomposition prompting serves as one of the first attempts to mitigate the error prop- agation issue in LLMs' multi-step reasoning, and we highlight this problem as meaningful future di- rection. For example, we can further reduce errors in intermediate reasoning steps by incorporating method into interactive semantic parsing framework (Yao et al., 2019, 2020; Li al., 2020; Zeng al., 2020; Chen et al., 2023a,b). Since the decomposed sub-questions are natural language, this interactive approach enables database users to easily spot the errors in each sub-question. Then, they can collaborate with LLMs by editing the sub- questions directly providing natural language feedback (Elgohary al., 2020, 2021; Narechania et al., 2021: Mo al., 2022), which should further improve text-to-SQL parsing accuracy.
Limitations
Experiments on other large language models. Our study focused on conducting experiments us- ing Codex the LLM, since it was available at no cost and showed impressive performance in text- to-SQL parsing among LLMs before GPT-4 (Ra- jkumar et al., 2022). To gain comprehensive understanding of different CoT-style promptings for text-to-SQL, future research should explore the

---

effects of these promptings on more recent, more powerful LLM models, such as GPT-4 (if budget al- lows). By doing so, we can determine whether the improvements achieved by our proposed prompt- ings are consistent across different LLMs.
Drspider: A diagnostic evaluation benchmark wards text-to-SQL robustness.. In The Eleventh Inter- national Conference on Learning Repre rentatenttii
Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hambo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang. Joseph Lilien, et al. 2023b. Dr. spider: A diagnostic evaluation bench- mark towards text-to-sql robustness. arXiv preprint arXiv:2301.0888.
Experiments on robustness. In our work, we mainly test robustness from the prompt design per- spective such as how to select in-context examples, the number of in-context examples and the prompt format of in-context examples. It would also be valuable investigate our prompting methods un- der different databases, natural language questions, or SQL perturbations (Chang et al., 2023a). This broader exploration would enable us to evaluate the robustness of our prompting methods across diverse scenarios.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
Shijie Chen, Ziru Chen, Huan Sun, and Yu Su. 2023a. Error detection for text-to-sql sql semantic parsing.
Ziru Chen, Shijie Chen, Michael White, Raymond Mooney, Ali Payani, Jayanth Srinivasa. Yu Su, and Huan Sun. 2023b. Text-to-SQL. error correction with language models of code. In Proceedings of the 61st Annual Meeting of the Association for Compu- national Linguistics (Volume 2: Short Papers), pages 1359-133 1372, Toronto, Canada. Association for Com- putational Linguistics.
Acknowledgements
We would like to thank the anonymous review- ers and colleagues from the OSU NLP group for their thoughtful comments. This research was spon- sored in part by a sponsored research award by Cisco Research, NSF IS- 1815674, NSF CAREER #1 #1942980, NSF OAC-21 2666, and Ohio Super- computer Center (Center, 1987). The views and conclusions contained herein are those of the au- thors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copy- right notice herein.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint orXiv:2204.03311.
Deborah A Dahl, Madeleine Bates, Michael K Brown, William M Fisher, Kate Hunicke-Smin David S Pallett, Christine Pao, Alexander Rudnicky, and Eliz- abeth Shriberg. 1994. Expanding the scope of the atis task: The atis- corpus. Human Language Tech- nology: Proceedings of Workshop held at Plains- boro, New Jersey, March 8-11, 1994.
References
Xiang Deng, Ahmed Hassan, Christopher Meek, Olek- sandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure-grounded grounded pretraining for text-to-sql. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1337-1350.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. 2023. Do as can, not as say: Grounding language in robotic affordances. In Conference Robot Learn- ing, pages 287-318. PMLR.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-
Ahmed Elgohary, Saghar Hosseini, and Ahmed Has- san Awadallah. 2020. Speak to your parser: Interac- tive text-to-SQL with natural language feedback. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2065- 2077, Online. Association for Computational Lin- guistics.
Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang, Vit- torio Castelli, Patrick Ng, and Bing Xiang. 2023a.
Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. NL-EDIT: Correcting semantic parse errors through natural

---

language interaction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 5599-5610, Online. Association for Computational Linguistics.
Arpit Narechania, Adam Fourney, Bongshin Lee, and Gonzalo Ramos. 2021. Diy: Assessing the correct- ness of natural language to sql systems. In 26th Inter- national Conference on Intelligent User Interfaces, pages 597-607.
Nitish Gupta and Mike Lewis. 2018. Neural composi- tional denotational semantics for question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages
Ansong Ni, Srini Tyer, Dragomir Radev, Veselin Stoy- anov. Wen-Tai Yih, Sida Wang, and Xi Victoria Lin. 2023. LEVER: Learning to verify taguage-to-cooe generation with execution. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26106-26128 PMLR.
SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin. Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2023, Selec- tive annotation makes language models better few- shot learners. In The Eleventh International Confer- ence on Learning Representations.
Mohammadreza Pourreza and Davood Rafiei. 2023, Din-sql: Decomposed in-context learning of text- to-sql with self-correction.
Srinivasan Iyer, Toannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learn- ing a neural semantic parser from user feedback. In 55th Annual Meeting of the Association for Compu- tational Linguistics.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022. RASAT: Integrating relational structures into pretrained Seq2Seq model for text-to-SQL. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 3215-3229, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Takeshi Kojima, Shixiang Shane Gu Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabil- ities of large language models. arXiv preprint arXiv:2204.00490
Yuntao Li. Bei Chen, Qian Liu, Yan Gao. Jian- SaanGuung Lou, Yan Zhang, and Dongmei Zhang. 2020. "what do you mean by that?" inneee-indeeendentt interac- tive approach for enhancing text-to-SQI. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6913-6922, Online. Association for Computational Linguistics.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 9895-9901.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. 2023a, A comprehensive evaluation of chat- gpt's zero-shot text-to-sq. sqi capability. arXiv preprint arXiv:2303. 13547.
Peter Shaw, Ming-Wes Weig Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generaliza- tion and natural language variation: Can semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Vol- ume I: Long Papers), pages 922-938.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 55(9):1-35.
Joan C. Miller and Clifford J. Maloney. 1963. System- atic mistake analysis of digital computer programs. Commun. ACM, 6(2):58-63.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. Rat-sql: Relation- schema encoding and linking for text- to-sql parsers. In Proceedings the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019. Multi-hop reading compre- hension through question decomposition and rescor- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097-6109.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. 2022a. Emergent abilities large language models. Transactions on Machine Learning Research.
Lingbo Mo. Ashley Lewis, Huan Sun, and Michael White. 2022. Towards transparent interactive seman- tic parsing via step-by-step correction. In Findings of the Association for Computational Linguistics: ACL
2022, pages 322-342, Dublin, Ireland. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou,

---

al. 2022b. soning in large language models. In Advances in Neural Information Processing Systems.
prompting elicits rea-
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-sql with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396-411.
Tomer Wolfson, Daniel Deutch, and Jonathan Berant. 2022. Weakly supervised text-to-sql parsing through question decomposition. In Findings of the Associ- afion for Compitational Linguistics: NAACL 2022, pages 2528 2542.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, and Ed H. Chi. 2023. Least-to- to- most prompting enables com- plex reasoning in large language models. In The Eleventh International Conference on Learning Rep- resentations.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understand- ing benchmark. Transactions of the Association for Computational Linguisties, 8: 18188-998
Navid Yaghmazadeh, Yuepeng Wang. Isil Dillig, and Thomas Dillig. 2017. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Pro- gramming Languages, ((00PSLA):-266.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and text-to-SQL. case study. In Proceed- ings of the 2019 Conference Empirical Methods Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing SAMNLP-JJCLP" pages 5447-5458, Hong Kong, China. Association for Computational Linguis- tics.
Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. 2020. An imitation game for learning se- mantic parsers from user interaction. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6883-6902, Online. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang. Dragomir Radev, Caiming Xiong, et all 2021. Grappa: grammar-auumentee pre-training for table semantic parsing. International Conference on Learning Representations.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao. Shanelle Roman, et al. 2018. Spider: large-scale huben-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 391 1-3921.
John M Zelle and Raymond J Mooney. 1996. Learning parse database queries using inductive logic pro- gramming. Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Jichuan Zeng, Xi Victoria Lin, Steven C.H. Hoi, Richard Socher. Caiming Xiong, Michael Lyu, and Irwin King. 2020. Photon: robust cross-domain text- to-SQL. system. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics: System Demonstrations, pages 204-214, Online. Association for Computational Linguistics.

---

A Example Prompts
### sQLite SQL tables, with their properties:
#
# medicine (id, name, trade_namey name, fda_approved)
# enzyme (id, name, location, product, chromosome, onin, porphyria)
1 medicine mee_ennynneinteraacii
(enzyme_ia) id, medi ol nidddd, interact interaction_typel)
Figure 2: An example for API does prompt format, introduced by Rajkumar et al. (2022), on Spider.
CREATE TARGE grapes { ID INTEGER PRIMARY KEY, Grape TEXT UNIQUE,
Color TEXT
3 example rows:
SELECT FROM grapes LIMIT 3;
TD Grape Color
1 Barbera Red
2 Cabernet Franc Red
3 Cabernet Sauvingnon Red
CREATE TABLÉ appellations No INTEGER PRIMARY KEY, Appelation TEXT UNIQUE, County TEXT, state TEXT,
Area TEXT, isAVA TEXT
3 example
SETECT FROM appellations L.MTT
Appelation County State Area isAVA
exander Val Sonora California North coast
2 Amador County Amador California Sierra Foothills No
Amador-Mendoci no-Sooom Counties N/A californía N/A No
CREATE TABLE wine
No INTEGER,
Grape TEXT, Winery TEXT, Appelation TEXT,
State TEXT, Name TEXT,
Year INTEGER,
Price INTEGER,
Score INTEGER,
Cases INTEGER,
Drink TEXT, FOREIGN ΚEУ (Grape) REFERENCES grapes (Grape),
FOREIGN KEY (Appelation) REFERENCES appellation lations (Appe 1 i
++
3 example rows:
SELECT FROM wine LIMIT 3;
No Grape Winery Agpelation State Name Year Price Score Cases Drink
1 2infandel Robert Biale St. Belena California old Kraft Vineyard 2008 44 93 275
2 Zinfandel chiarello Family Napa Valley California Giana 2008 35 93 480 now
3 zinfandel Robert Biale Napa Valley California Black Chicken 2008 40 91. 2700 2012
Figure 3: An example for Create Table Select 3 prompt format, introduced by Rajkumar et al. (2022), on Spider.

---

SQLite SQL tables, with their properties:
medicine (id, name, trade name, fda approvedd)
enzyme (id, name, location, product, chromosome, omin, porphyria) medicinn_eenzye__ineeee interacttinnn senzyme_id, medicine_id, interaction_type)
#*# What is the total count of enzymes?
SETECT count (+) FROM enzyme
### SQLite SQL tables, with their properties:
# buildings (id, name, city, height, stories, status)
# companies (id, name, headquarters, industry, sales_billion, profits_billion, assets billicn, market_value
d affice_locations (building company_id, move_in_year)
and Show the irdustries shared by companies whose headquarters are "DSA" and companies whose headquarters are "China",
SELECT Industry FROM Companies WHERE Headquarters "USA" INTERSECT SELECT Industry
FRCM Companies WEERE Headquarters "China"
441 sQLite SOL tables, with their properties:
#
4 stadium astadium_ia location, name, capacity, highest, lowest, average)
#
singer (singer_id, name, country, song_name, song_roloase_yedt age, is_male)
1 concert (concert_id) concert_name, theme, stadium um__d, year)
# singer_in_concort (concort_id, singer_id)
### How many singers do have?
Figure 4: An example prompt under the standard API does prompting for 2-shot on Spider.

---

SQLite soL tables, with their properties:
# book club (book club_id, id, year, author cacorroo___dddee or_edio,r, book_title publisher, category, result)
# movie (movie_id, title, year, director, budget_ million, gress worldwide)
I culture_cymparyy (company_name, type, incorperated_ group_equiity__harhhoolle. book_club_id, movie_id)
### List categories that have at least two books after year 1989.
# Let's s think step by step
# This query chooses records from the Book Club table, folloved by WHERE clause that selects records where the year column is greater than 1989. It then groups the results by the category column. It then filters the results where the count of each category is greater than or equal to 2. Tt then se selects the category co column.
I Thus, the answer for the question on 18: List categories that have at least two books after year 1989.
SELECT category FROM club WHERE YEAR 1989 GROUP BY category HAVING count (+) >= 2
#*# SQLite SQL tables, with their properties:
4
# stadium (stadium_id, location, name, capacity, highest, Lowest, average)
8 singer (singer_io) name, country, song_name, song_release sear,,, age, (s_male)
# concert (concert_id, concort_name, theme, stadium_id, year) singer_in_coreee (concert_id, singer_id)
### How many singers do have?
Figure 5: An example prompt under chain-of-thought API does prompting for 1-shot Spider.

---

#+ se sgl tables, with their properties:
# class (class_code, cre_code, class_section class_time, class_room, prof_num)
# course (cre_cede, dept_codo, cre_doscription, crs_credit)
# department (dept_code, dept_name, school_code, emp_num, dept_address,
# employee (emp_num, num, emp iname, emp initial, emp jobcjod, emp hiredate, emp dcbb
# enroll _olas__code, stu_num, enroll_grade)
# professor (emp num, dept_ prof_office prof _tf__xxtesion, prof _igred
degree)
I student (stu_numg num, stu_Iname, stu_fname, stu_init, stu_doo, stu_hrs, stu_class, stu gpa, scu_transferr, dept_code, stu phone, pref_nun
d To answer the question "Find the first names and offices of all instructors who have taught some course and the course description and the department name." ", we need to know: "Find the rirst names and offices of all instructors.", "Fina the first names and offices of all instructors who have taught some course.", ", "Find the first names and offices of all instructers who have taught some course and the course description.".
HH SQLite SQL tables, with their properties:
# station (station_ia id, name, ancual_entryy exit, annual interchanges, total_-
passengers, location, main_services, number__f__latforms)
# train (train_id, name, time, service)
I train_station (train_id, station_id)
and times in stations in London in descending order train time.", nsed know:
# To answer the question "Show all train
Figure 6: An example prompt under least-to-most + API does prompting (problem reduction) for I-shot on Spider.
sQLite tables, with their properties:
(class_codde crs_code, # course code, dept__code,
tlass_time, class_room, prof_num) description, ers credit)
1 department (dept_code, dept_name, school_code, emp_ emp_num, dept_address)
# employce temp_num, emp_iname, emp_initial, emp_jobcode, emp_hiredato, emp_dob)
enroll (class_code stu_num, enroll_grade)
# professor (emp_num, dept_code, prof_office, prof_cxtension, prof_nigh_degren
# student (stu_aur, stu Iname, stu fname, stu_init, stu_dob, stu_hrs, stu_class, stu_gpa, stu_transfer, dept_cade, stu_phone, prof_nur)
#
0: Find the first names and offices all instructors.
SELECT emp fname 12.prof_office FROM employee AS JOIN professor AS 12 ON 12.emp_nat
Ti. Ti.emp_num
Q: Find the first names and offices of all instructors who have taught some course.
T4.prof_office FROM CLASS As т. JOIN employce AS T2 A: SELECT T2. o2.mppmname
CN T1. Tl.prof_num num T2, emp_pnum JOIN course AS T3 ON cl.crs_code T3. c3ddre_code JOIN professor AS т4 ON T2, e2.emp_num 74.emp_num
0: Find the first names and offices of all instructors who have taught some course and the course description.
A: SELECT 12. hmm__nnaee
cxprof_office T3.crs description FROM CLASS AS T1 JOTN emploуee AS T2 ON T1 T1.prot_num = emm_mmmn JOIN course AS т3 ON Ti.crs_code 13.ccr_ccode JOIN professor AS 14 ON T2.emp_naa num - T4.emp_non num
2: Find the first names and offices of all instructors who have taught some course and the course description and the department name. A: SELECT T2. emp_fname T4.prof_offfe ooffcceffeee__e CLASS AS тl JOIN employee AS 12 ON Ti.prof_nuu _um = :2. emp num JOIN coarse AS 13 ON Ti.ers_code 13.ors_cod JOIN professor AS T4 ON T2. 12.emp_nine num Té.emp num JCIN department AS 15 ON ré.dept_code - T5. r5..dep_codde
r3.crs_description
I5.dept_name FROM
441 sqLite SOL tables, with their properties:
i station (station_id, name, annual_ontry_oxx annual annuaLinterrchange,, total_- passengers, location, main_services, oumber_of__latffoms # train (train_id, name, time, service) train_station (train_id, staticn_id)
Show train names and times.
Figure 7: An example prompt under least-to-moss most API does prompting (problem solving) for 1-shot on Spider. The same prompt will be used to solve the next sub sub-question after we get the generated SQL query for the first sub-question.

---

sqLite sQL tables, with their properties:
# document_Eypes
document_descti descriptiio)rii
# documents (document id, ptppemeett_t_______ee__ee__ todeete___ee___eeeee__________ee code, grant id, sent_date, response_ received_
# grants (grant_id, organisaticn_id, grant_ammuns grant_stert_ date, grant_ date,
# rrgentssaiionttpes (organisation_
# organisat sat (organisation_ orgenisation organ|sactoon_eetalla
# project_outcomes (project_id, outcoms_code, outcome_details)
# project_staft (staff_id, project_id, id,
date_from, from, date_to, [etaild # projects (project_id, organisation_id, project_details)
# sereaarch-outccmer (outcome code, outcome outcome__desceee description)
1 research_staff (staff_id,
staff_details)
# staff_
code, role descrriptin
1 tasks (task_id, project_id, task_detall eg agree objectivee ives)
atd question: Find out the send dates of the documents with the grant amount of more than 5000 were granted by organisation type deseribed as "Research". decompose the question
1. Find out the send dates of the documents.
2. Find out the send dates of the documents with the grant amount of more than
5000.
3. Find out the send dates of the documents with the grant amount of more than 5000 granted by organisation type deseribed "Research".
# Thus, the with the grant "Research",
for the question is: Find the send dates of the documents of than 5000 granted by organisation type described
SELECT ri,sent date FROM documents AS JOIN Grants 12.grant_id JOIN Organisations JOIN organisation Types AS 13.organisation type organisation type WHERE T2. 12.grant__amount 5000 AND or.organisation, typeeonnn__p_peeeeeeeee_e descripptittt 'Research'
Tl. id or..orgaisatien_ id 13.organisation
SQLite SQL tables, with their properties:
# stadium (stadium_id, location, name, capacity, highest, lowest, average)
singer (singer_id, neme, country, seng_name, song_release_ year, age, is_male)
# concert (concert_id, concert_name, theme, stadium year)
# Lunger_in_concet (concert_id, singer_id! id)
#4 Question: How many singers do we have?
decompose the question
Figure 8: example prompt QDecomp + API does prompting for I-shot on Spider.

---

sqLite sQL tables, with their properties:
# document_Eypes
document_descti descriptiis)rii
# documents (document id, ptppeeeee___tt___________ code, grant id, sent_date, response_
received_
# grants (grant_id, organisation_id, grant_amouns, grant_stert, date, grant_ date,
# orgenisationn_yper (organisation
# organisat
(organtsation orgenisation organtsaction_eetalll
# project_outcomes (project_id, outcoms_code, outcome_details)
# project_staft (staff_id, project_id, id,
date_from, from, date_to, other (etaild # projects (project_id, organisation_id, project_details)
#
(outcome code, outcome outcome__desceee description) 1 research_starf (staff_id,
staff_details)
(role (role_code, code, role desccriptt)nn 1 tasks (task_id, project_id, task_detall eg agree objectivee ives)
# staff_
atd question: Find out the send dates of the documents with the grant amount of more than 5000 were granted by organisation type deseribed as "Research". decompose the question
1. Find out the send dates of the documents.
SQL table (colume): documents beet__date
2. Find out the send dates of the documents with the grant amount of more than 5000.
SQt table : grants (grant_amount, grant_id)
3. Find out the send dates of the documents with the grant amount of than 5000 granted by organisation type doseribed "Research". table (column): organisation
(organisation_tye) type_descrippiion organisation_type)) organisations (organisation_type, orgamisation_id))
Thus, answer for the question is: Find with the grant "Research".
send dates of the documents of than 5000 granted by organisation type described
71.grart_id ON wtw.organssaiion_.... 13. T2.grant_id JOTN Organisations sations r3.organisationntyye organnniiittion sorpttttttpteeee WHERE JOIN orgenisation__yper
SELECT T1.sont_date FROM documents AS JOIN Grants
T2. 22.grant_amount 5000 AND
"Research"
### SQLite SQL tables, with their properties:
# stadium (stadium id, location, name, capacity, highest, lowest, average)
# singer (singer_id) name, country, seng_name, song_ release yelr age, is ialmale)
# concert (concert_id, concert_nan name, theme, stadium id, year)
1 singer_in_coneen (concert_id, sf singer_id]
## Question: How many singers do we have?
decompose the question
Figure 9: An example prompt under QDecomp-InterCOL+ + API does prompting for 1-shot on Spider.


Reviews:
[{"review_id": "dlAwDmaX8P", "reviewer": "Reviewer_DzXr", "summary": "", "questions": "Have you conducted any significant tests and what are the results?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your time reviewing our work! We are delighted that the reviewer finds our “proposed method works well” and “the results show promising improvements.” Here we address each of the reviewer’s concerns below and will add related discussion in our revised paper:\n\n“If it is applicable to other datasets”\n\nThere might be some misunderstanding here. We agree with the reviewer that it is important to test how well our method generalizes across databases and datasets. To this end, we conducted our main experiments on Spider (Table 1), a text-to-SQL dataset with 200 databases. Moreover, Spider is a cross-database benchmark, which means that databases in its development set are not included in the training set. This evaluation setup exactly tests our method’s generalization across different databases. In addition to Spider, we also tested our method on three single-database datasets, GeoQuery, IMDB, and Yelp (Table 8). Thus, our work is not “based on one database” or one dataset.\n\n“Why InterCOL's impacts are not consistent”\n\nWe find that the selection of in-context examples (Section 4.2) has the largest impact on QDecomp + InterCOL’s performance. Here we include some new experiments to show that when using harder in-context examples (denoted as G3, Lines 370-371), QDecomp + InterCOL consistently achieves the best performance in the main experiment and robustness experiments for the other two factors (number and format of in-context examples):\n\n|   Method  (Spider Dev)                    | Simple           | Medium           | Hard             | Extra Hard       | Overall TS (Overall EX)                                   |   |\n|-------------------------|------------------|------------------|------------------|------------------|-----------------------------------------------------------|---|\n| Standard (G3)           | 82.9             | 56.6             | 44.1             | 43.3             | 58.7 $\\pm$ 3.32 (70.4 $\\pm$ 3.68)                         |                                                   |\n| Least-to-Most (G3)      | 80.3            | 64.6             | 52.8             | 45.3             | 63.3 $\\pm$ 1.95 (73.8 $\\pm$ 1.72)                         |\n| QDecomp (G3)            | $\\underline{87.9}$ | $\\underline{67.5}$ | $\\underline{54.1}$ | $\\underline{45.1}$ | $\\underline{66.5}$ $\\pm$ 0.63 ($\\underline{77.1}$ $\\pm$ 0.75) |\n| + InterCOL (G3)         | $\\textbf{88.7}$    | $\\textbf{71.1}$    | $\\textbf{56.8}$    | $\\textbf{45.7}$    | $\\textbf{68.8}$  $\\pm$ 1.16 ($\\textbf{78.2}$ $\\pm$ 1.07)       |\n\nNew Experiment 1: 8-shot test-suite (TS) and execution (EX) accuracy of Codex on the Spider development set using G3 examples and API Doc format. \n\n|               | API docs | Create Table + Select 3 |\n|:-------------:|:--------:|:-----------------------:|\n| Standard  (G3)    | 58.3     | 60.6                    |\n| Least-to-Most (G3) | 63.0     | 65.6                    |\n| QDecomp (G3)     | 65.5     | 66.9                    |\n| + interCOL (G3)   | 67.3     | 69.8                    |\n\nNew Experiment 2: 4-shot test-suite (TS) accuracy of Codex on the Spider development set using G3 examples and two different formats. \n\n\n|               | 0    | 1    | 4    | 8    |\n|:-------------:|:----:|:----:|:----:|:----:|\n| Standard  (G3)    | 59.6 | 62.2 | 58.3 | 58.7 |\n| Least-to-Most (G3) | -    | 63.7 | 63.0 | 63.3 |\n| QDecomp (G3)      | -    | 63.6 | 65.5 | 66.5 |\n| + interCOL (G3)    | -    | 66.4 | 67.3 | 68.8 |\n\nNew Experiment 3: Test-suite (TS) accuracy of Codex on the Spider development set with different numbers of in-context examples. We use G3 examples and API Doc format. \n\nWe hypothesize that InterCOL benefits from harder in-context examples (G3) because they contain more complex table and column linking demonstrations. Consequently, QDecomp + InterCOL learn to ground table and column names to each subquestion better, which can guide Codex to generate the correct SQL query. On the other hand, with easier in-context examples:\n\n(1) [Table 5 in the paper] When the number of examples is insufficient (1 or 4), they may not provide clear guidance for Codex to accurately deduce the correct table-column pairs for harder questions at inference time. So, adding InterCOL to QDecomp may introduce more error propagation issues, which leads to a slight performance decrease.\n\n(2) [Table 7 in the paper] When we use the CREATE TABLE + SELECT 3 format, the rich schema information in this format can be more helpful than the easy demonstrations for table and column linking by InterCOL. So, QDecomp alone can achieve decent performance, and easy InterCOL examples cannot help as much as harder examples.\n\nBesides the new results, we would also like to kindly remind the reviewer that our work’s primary goal is to explore different CoT style prompting methods for text-to-SQL and try to address the two identified research questions (Lines 73-77), so proposing new CoT style prompting methods is only part of our contribution. Additionally, we propose both QDecomp and QDecomp + InterCOL, where the latter is a variant of QDecomp. In our paper, we have shown that QDecomp (+ InterCOL) consistently achieves the best performance across all experiments.\n\nWe will add the new results and analysis in our appendix.  \n\n“Have you conducted any significant tests and what are the results?”\n\nWe perform McNemar’s Test on the median results out of the 10 runs in our main experiment (Table 1) to measure the statistical significance of the CoT-style prompting methods we have explored. Since chain-of-thought prompting performs worse than standard prompting, we focus on comparing the other three methods, least-to-most, QDecomp, and QDecomp + InterCOL, with standard prompting. By McNemar’s test, we find that all of the three methods achieve statistically significant improvement over standard prompting (p < 0.01). We will incorporate this result in our paper’s revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The propose method works well and the results show promising improvements.\n", "reasons_to_reject": "The work is based on one db and it is not clear if it is applicable to other datasets. The guess is yes but cannot be verified.\nAlso it is not clear why  InterCOL's impacts are not consistent. It seems to work for some settings but has negative impacts under other settings. It is better to see results of InterCoL only.", "questions_to_authors": "Have you conducted any significant tests and what are the results?", "paper_topic_and_main_contributions": "In this work, the authors studied how to enhance language models' reasoning ability through CoT-style prompting.\nCoT is enhanced by 1) QDecomp which decompose the complex query into simple ones and 2) InterCOL, incrementally including new data lines/rows.\n\nThis strategy is shown to be able to prevent error propagation and improve performances.\n\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ucYSNTxFML", "reviewer": "Reviewer_Cceb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. On the other hand, there might be some misunderstanding and please let us clarify as follows:\n\nWe want to first clarify that our paper's main contribution is not to “propose a CoT-style prompting method.” Instead, our work focuses on a novel exploration of diverse CoT prompting methods for text-to-SQL parsing. Specifically, we aim to tackle the two research questions identified in the paper (Lines 73-77). Not only do we adapt chain-of-thought prompting and least-to-most prompting to text-to-SQL parsing (which is the first effort for this task), but also propose a new question decomposition prompting method (QDecomp) and its variant (QDecomp + InterCOL). Hence, our work is not to “design a variant of question decomposition prompting” but to propose both QDecomp and its variant. With various experiments around the prompting methods, we offer valuable insights for the future development of effective prompting strategies for text-to-SQL parsing.\n\n“It is difficult to prove the validity and robustness of the method.”\n\nWe’d appreciate it if the reviewer could elaborate on this concern, but let us highlight three aspects of our experiments, which aim to establish the validity and robustness of our proposed method. First, in our main experiments (Section 5.1), we conduct the evaluation on both the Spider development set and the Spider Realistic dataset. Spider Realistic removes or paraphrases direct mentions of table and column names in the original Spider questions, which shows our methods’ robustness against language variations. Second, we undertook a series of experiments in Section 5.3 to test our methods’ robustness against three factors of prompting, including the number, selection, and format of in-context examples. Finally, we show that our methods are also effective on three other in-domain datasets (Section 5.4), GeoQuery, IMDB, and Yelp, to provide further evidence of the effectiveness and resilience of our methods.\n\n“The result of experiment is not very satisfactory.” and “Experimental results are fluctuate too much.”\n\nWe would greatly appreciate further details from the reviewer to elaborate on these comments. Currently, we are afraid that we don’t have a precise understanding of what the reviewer is concerned about. However, we attempt to explain as below:\n\nFor the first comment, we firmly believe that our obtained results effectively address the two identified research questions (Lines 73-77). Our findings contribute substantially to these inquiries, and we are eager to engage in a constructive discussion with the reviewer to ensure that this perspective is well understood.\n\nFor the comment “Experimental results are fluctuate too much”, we are uncertain which aspect the reviewer is referring to. Was the concern about the standard deviations in Table 1 being too high? If so, we believe this is largely due to the random selection of in-context examples. As Table 2 in [1] shows, the standard deviation on other NLP tasks can be as large as 14.7 when randomly selecting examples, not to mention that text-to-SQL parsing is a complex task that can result in large variances even for fine-tuned models (Table 3 in [2]). \n\nYour insights will greatly assist us in better understanding and addressing these concerns.\n\nReferences:\n\n[1] Zhang, Yiming, et al. “Active Example Selection for In-Context Learning.” Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[2] Deng, Xiang, et al. “Structure-Grounded Pretraining for Text-to-SQL.” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Propose a question-decomposition prompt.", "reasons_to_reject": "- The result of experiment is not very satisfactory. \n- Experimental results are fluctuate too much\n- It is difficult to prove the validity and robustness of the method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a CoT-style prompting method that improves LLMs’ multi-step reasoning for text-to-SQL task. They design a variant of question decomposition prompting by incrementally including the table and column names involved in each sub-question.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dbzEQB7hTU", "reviewer": "Reviewer_vQpJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate that the reviewer finds our method shows “an improvement in accuracy,” our analysis “useful for subsequent studies,” and our attempt to mitigate error propagation in CoT prompting “interesting.” To address the reviewer’s concern, we conduct a detailed comparison between our method and “fine-tuned methods” or “other LLM-based methods.” We will also add related discussions in our revised paper.\n\n\"Comparison with Fine-Tuned Methods\"\n\nIn our new experiments (response to Reviewer DzXr), we show that harder in-context examples can further improve our methods’ accuracy to an average of 78.2 execution accuracy and a maximum of 79.8 among the 10 runs. We think that our method can achieve comparable performance to RASAT+PICARD (79.8 v.s. 80.5 on the Spider development set) [1], a method fine-tuned with supervised learning. Meanwhile, we would like to gently remind the reviewer that our method is few-shot in-context learning. Thus, direct comparison with fine-tuned methods using the full Spider training set may be unfair. Reducing the number of annotated text-to-SQL data is also one of the motivations of our work.\n\n\"Comparison with other LLM-based Methods\"\n\nWe compare our method with other LLM-based methods [2][3]. As previously mentioned, the best execution accuracy our method can achieve is 79.8, when using hard in-context examples. Compared to [2], our method is 1.9 points lower than their execution accuracy (81.2). We hypothesize this is because [2] uses the execution results of SQL queries, which provides extra information for better database grounding. We would like to explore the use of database contents beyond table and column names in future work. Meanwhile, we note that LEVER’s verifier model is also fine-tuned on the full Spider training set, which may have extra advantages over our method. Our work is closest to DIN-SQL [3], which introduces a multi-stage prompting method for text-to-SQL parsing, and we can achieve better execution accuracy on the Spider development set (79.8 v.s 74.2; Tables 4 and 5 in [3]).\n\nReferences:\n\n[1] Qi, Jiexing, et al. \"RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[2] Ni, Ansong, et al. \"Lever: Learning to verify language-to-code generation with execution.\" International Conference on Machine Learning. 2023.\n\n[3] Pourreza, Mohammadreza and Rafiei, Davood. “DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction.” Arxiv Preprint. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies CoT style prompting within the domain of text-to-SQL parsing. The experimental findings provide two primary insights: (1) error propagation is introduced by these two methods frequently, and (2) iterative prompting may not be necessary when prompting LLMs for text-to-SQL parsing.\n- The proposed question decomposition prompting method demonstrates an improvement in accuracy for the text-to-SQL parsing task compared to both the CoT and least-to-most prompting strategies.\n- It is good to see experiments and analysis on the number, format, and selection strategy of in-context examples, which is useful for subsequent studies on designing prompting strategies for text-to-SQL parsing.\n- The attempt to address and mitigate the error propagation of CoT prompting in the context of text-to-SQL parsing is interesting.", "reasons_to_reject": "- Given the significant performance disparity observed between CoT-style prompting methods presented in this work and both fine-tuned methods [1] and other LLM-based methods [2], it is a lack of a detailed comparison providing insights into the performance differentials among the three.\n\n[1] Qi, Jiexing, et al. \"RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[2] Ni, Ansong, et al. \"Lever: Learning to verify language-to-code generation with execution.\" International Conference on Machine Learning. PMLR, 2023.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores enhancing the multi-step reasoning ability of LLMs in the context of text-to-SQL parsing. Specifically, this work applies CoT and least-to-most prompting to text-to-SQL parsing. In doing so, it finds issues of error propagation associated with these two prompting methods and highlights the unnecessity of iterative prompting. In response to these observations, the authors introduce a new CoT-style prompting method termed QDecomp, alongside its variant, QDecomp+InterCOL. The empirical evidence presented supports the enhanced accuracy of the proposed question decomposition prompting method. The experimental section offers a thorough analysis, shedding further light on the findings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper verifies the chain-of-thoughts prompting style in the context of text-to-SQL. Two enhancements are introduced to the original CoT, providing interesting insights. The reviewers agree that the experimental results are promising. The authors have provided detailed answers to the questions raised by the reviewers (+ new results), addressing their concerns with respect to statistical significance of the results, the impact of different choices, and the novelty of the proposal. Reviewer DzXr has kept the initial judgement for the "possibility of missing or misunderstanding some points" while the concerns of reviewer Cceb are either not backed up with details or addressed in the rebuttal.
Decision: Accept-Main