SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the BIRD dataset tend to be more intricate than those in the Spider dataset. Language models without access to database content often encounter challenges with schema linking. Therefore, our prompts for the BIRD dataset include sample rows from each table to aid the model in schema linking. Furthermore, we have concatenated the provided external knowledge for each question as a hint, placed immediately after each question. However, due to constraints such as limited context window size, the presence of external knowledge, and the inclusion of sample rows, we have had to reduce the number of demonstrations within the prompts for the BIRD dataset.
Model	EX	EM
DIN-SQL + GPT-4 (Ours)	85.3	60
RESDSQL-3B + NatSQL (DB content used) [Li et al., 2023a]	79.9	72
DIN-SQL + CodeX davinci (Ours)	78.2	57
Graphix-3B+PICARD (DB content used) [Li et al., 2023b]	77.6	74
SHiP+PICARD (DB content used) [Zhao et al., 2022]	76.6	73.1
N-best Rerankers + PICARD (DB content used) [Zeng et al., 2022]	75.9	72.2
RASAT+PICARD (DB content used) [Qi et al., 2022]	75.5	70.9
T5-3B+PICARD (DB content used) [Scholak et al., 2021]	75.1	71.9
RATSQL+GAP+NatSQL (DB content used) [Gan et al., 2021]	73.3	68.7
RYANSQL v2 + BERT [Choi et al., 2021]	-	60.6
SmBoP + BART [Rubin and Berant, 2020]	-	60.5
Table 2: Execution accuracy (EX) and exact set match accuracy (EM) on the holdout test set of Spider
5.4 Metrics
The performance of our models are evaluated using the official metrics of each dataset: exact-set-match accuracy (EM) and execution accuracy (EX) for Spider and valid efficiency score (VES) and execution accuracy (EX) for BIRD.
The exact-set-match accuracy (EM) treats each clause as a set and compares the prediction for each clause to its corresponding clause in the reference query. A predicted SQL query is considered correct only if all of its components match the ground truth. This metric does not take values into account. The execution accuracy (EX) compares the execution output of the predicted SQL query with that of the ground truth SQL query on some database instances. Execution accuracy provides a more precise estimate of the model's performance since there may be multiple valid SQL queries
for a given question, and exact set match accuracy only evaluates the predicted SQL against one of them. The Valid Efficiency Score (VES) is a metric designed to measure the efficiency of running the generated SQL queries. This metric is meaningful if the generated queries are correct, meaning their result matches that of the reference query. Therefore, the VES metric takes into account both the accuracy of the generated queries and their efficiency in terms of the execution time.
5.5 Results
5.5.1 Test set results
As shown in Table 2 for the holdout test set of Spider, our method achieves the highest execution accuracy using GPT-4 and the third-highest execution accuracy using CodeX Davinci among all officially published results at the time of this writing. This is achieved without even utilizing the database content. In terms of exact set match accuracy, our method achieves comparable results to previous works that do not utilize database content. As demonstrated in Table 3a in the case of the BIRD dataset, our method using GPT-4 achieved a test set execution accuracy of  $55.9\%$ , setting a new SOTA.
5.5.2 Development set results
Most of our evaluation during development was conducted on the development set of Spider which was easily accessible unlike the test set that was only accessible through an evaluation server provided by Yu et al. [2018]. Table 4a shows the performance of our method using different LLMs, compared to zero-shot prompting of Rajkumar et al. [2022] and Liu et al. [2023a] and our own few-shot prompting. To ensure a fair comparison for the few-shot prompting, we incorporate all the examples utilized for our three classes (easy, non-nested complex, and nested complex) inside the prompt. Given that the CodeX Cushman model has a smaller input context size than the CodeX Davinci and the GPT-4 models, we only use 2 examples from each class (for a total of 6 examples).
Our method significantly outperforms both simple few-shot prompting and zero-shot prompting, in terms of both exact set match and execu
Figure 4: The break-down of failure cases for DIN-SQL (green) and the basic few-shot prompting (blue) across different categories
tion accuracies, and the improvement is consistent across all models despite their sizes. For example, compared to few-shot prompting, our method improves the execution accuracy for all models by at least  $10\%$ .
On the development set of BIRD, our approach demonstrates a substantial improvement, achieving a  $4\%$  gain in execution accuracy and a remarkable  $9\%$  improvement in valid efficiency score over a GPT-4 baseline [Li et al. (2023)], establishing a new SOTA. These and other results are reported in able 3b.
The performance of our method on the test set (as reported in Tables 2 and 3a) is higher than that on the development set for both Spider and BIRD. It is hard to pinpoint the exact reason when the test set is hidden, but we speculate that fewer questions in the test set may require the knowledge of the database content, making it easier for our method to predict a correct SQL query. Furthermore, the development set has schema ambiguity (e.g., a query entity can be mapped to multiple database entities but only one is considered correct), and it is possible that the test set has less ambiguity.
We further analyzed the performance of our proposed method on queries with different levels of difficulty. Table 4b presents the performance of our proposed method compared to a basic few-shot prompting on the development set of Spider. Our proposed method outperforms the basic few-shot prompting across all difficulty levels, with the greatest improvement in performance observed for the extra hard and hard classes where the few-shot prompting performed poorly. Our improvement
Model	VES	EX
DIN-SQL + GPT-4 (Ours)	59.44	55.9
GPT-4	60.77	54.89
Claude-2	-	49.02
ChatGPT + CoT [Li et al., 2023c]	56.56	40.08
ChatGPT	51.40	39.30
Codex	41.60	36.47
Palm-2	-	33.04
T5-3B	27.80	24.05
T5-Large	25	20.94
T5-Base	14.7	12.89
(a) Execution accuracy (EX) and Valid Efficiency Score (VES) on the holdout test set of BIRD
Model	VES	EX
DIN-SQL + GPT-4 (Ours)	58.79	50.72
GPT-4	49.77	46.35
Claude-2	-	42.70
ChatGPT + CoT [Li et al. 2023c]	42.30	36.64
ChatGPT	43.81	37.22
Codex	43.41	34.35
Palm-2	-	27.38
T5-3B	25.57	23.34
T5-Large	22.74	19.75
T5-Base	12.90	11.54
(b) Execution accuracy (EX) and Valid Efficiency Score (VES) on the development set of BIRD
on the easy class (compared to basic few-shot) is due to incorporating schema links in the prompt, highlighting the importance of our schema-linking module.
Table 3: Performance of DIN-SQL on BIRD development set and test set.
Prompting	Model	EX	EM
DIN-SQL (Ours)	GPT-4	74.2	60.1
CodeX Davinci	69.9	57.2
CodeX Cushman	47.6	35.7
Few-shot (Ours)	GPT-4	67.4	54.3
CodeX Davinci	61.5	50.2
CodeX Cushman	43.1	30.9
Zero-shot (Ours)	GPT-4	64.9	40.4
Zero-shot
[Liu et al., 2023a]	ChatGPT	60.1	-
Zero-shot
[Rajkumar et al., 2022]	CodeX Davinci	47.5
Zero-shot (DB content used)
[Rajkumar et al., 2022]	CodeX Davinci	55.1
CodeX Cushman	53
GPT3	21.7
(a) Performance compared to zero-shot and few-shot prompting using different LLMs on the dev set of Spider
Execution accuracy (EX)
Prompting	Model	Easy	Medium	Hard	Extra	All
DIN-SQL	GPT-4	91.1	79.8	64.9	43.4	74.2
DIN-SQL	CodeX Davinci	89.1	75.6	58	38.6	69.9
Few-shot	GPT-4	86.7	73.1	59.2	31.9	67.4
Few-shot	CodeX Davinci	84.7	67.3	47.1	26.5	61.5
Exact set match accuracy (EM)
Prompting	Model	Easy	Medium	Hard	Extra	All
DIN-SQL	GPT-4	82.7	65.5	42	30.7	60.1
DIN-SQL	CodeX Davinci	78.6	67.3	38.5	17.5	57.2
Few-shot	GPT-4	87.9	54	47.1	12	54.3
Few-shot	CodeX Davinci	77	53.8	38.5	12.7	50.2
(b) Performance compared to our basic few-shot prompting across different query difficulty levels
Table 4: Performance of DIN-SQL against other in-context learning approaches
5.5.3 Error improvements
In Section 3 we did an error analysis of basic few-shot prompting on 500 queries randomly chosen from the training set. To understand the degree those errors are resolved, we ran DIN-SQL on the same 500 queries. As shown in Figure 4, our proposed approach improves the performance for all categories with the largest improvement seen for the JOIN and Nested categories. Despite having an explicit module for schema-linking, the largest portion of failure cases still belong to this category.
5.6 Ablation study
In an ablation study, we evaluated our approach with and without each of the four modules. As shown in Table 5 for the CodeX Davinci model, excluding any of the modules leads to an overall decrease in performance, in terms of the execution accuracy.
More details emerge as we study the effectiveness of each module across different query classes. Schema linking helps all query classes with the least improvement for the hard class. Our inspection of a sample of the failed cases reveals that schema linking sometimes finds redundant links due to an ambiguity in the question or schema, and this can introduce redundant joins or output columns.
Without a classification, we had to use either a simple few-shot prompting or a decomposed chain-of-thought (COT) prompting for all queries. The reported performance without a classification
Prompting	Model	Easy	Medium	Hard	Extra	All
DIN-SQL (generic self-corr)	CodeX Davinci	89.1	75.6	58	38.6	69.9
DIN-SQL (gentle self-corr)	CodeX Davinci	87.5	76.9	51.7	36.1	68.7
DIN-SQL w/o self-corr	CodeX Davinci	83.9	75.4	52.3	36.1	67.3
DIN-SQL w/o schema-linking	CodeX Davinci	87.3	70.6	57.6	27.1	65.9
DIN-SQL w/o classification (simple few-shot prompting)	CodeX Davinci	87.9	68.2	51.7	27.1	63.1
DIN-SQL w/o classification (decomposed COT prompting)	CodeX Davinci	84.2	71.2	54.3	38.6	68.2
DIN-SQL (gentle self-corr)	GPT-4	91.1	79.8	64.9	43.4	74.2
DIN-SQL (generic self-corr)	GPT-4	89.9	76.5	59.2	34.3	70.0
DIN-SQL w/o self-correc	GPT-4	91.1	79.1	63.2	41.6	73.3
Table 5: Performance of our method, in terms of execution accuracy, on the dev set with and without each module
module in Table 5 is for our comprehensive framework that includes all our components except classification. This means that the approach contains not only COT prompting but also Schema Linking, Self-Correction, and NATSQL Intermediate Representation, all of which are significant contributions of our work. The decomposed chain-of-thought result presented in this table refers to employing the most complex prompt, developed for the nested complex class, for all questions instead of adopting a classification-based approach to determine prompt complexity based on the question's level of difficulty. In contrast, the result for the DIN-SQL with simple few-shot prompting refers to using the simplest prompting class, easy class, for all questions across different level's of difficulty. As expected, a decomposed chain-of-thought prompting works better for hard and extra hard queries whereas a simple few-shot works better for the easy class.
For self-correction, we ran our study using both CodeX Davinci and GPT-4. For CodeX Davinci, a generic self-correction prompt helps the model across all query classes. A gentle self-correction prompt is also helpful but the gain is smaller than generic one for CodeX Davinci. However, there is less chance that GPT-4 generates a buggy code, and giving a generic prompt of "Buggy SQL: ... Fixed SQL:..." can hurt the performance. A gentle prompt work better for GPT-4 and improves the performance across all of the classes except the easy class.
Prompting has enabled large language models to achieve impressive performance on numerous NLP tasks across different domains, without requiring a large training set. Prior to our research, the effectiveness of prompting methods utilizing LLMs for the text-to-SQL task was inferior to that of models fine-tuned for the task. To bridge this gap, we have devised a decomposition technique to tackle some of the challenges that caused this disparity. Our extensive experiments on two challenging datasets of Spider and BIRD show that our method significantly improves the performance of prompting across all query classes, producing comparable or even superior results to state-of-the-art fine-tuned approaches.
There are some limitations or areas of improvement to this work. Our manually constructed demonstrations are fixed for each query class. Future research may explore adaptive and automated methods for generating demonstrations at finer granularities, which can further enhance the performance of our approach. Additionally, as of the time of writing this paper, our proposed approach, characterized by its decomposed and step-by-step structure, incurs a cost of approximately $0.5 and exhibits a latency of approximately 60 seconds when responding to a natural language question from the Spider dataset using GPT-4. We anticipate that as LLMs continue to advance, these costs and latencies should decrease, but reducing the cost is another possible direction.