Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison
Authors: Shuo Sun
Yuchen Zhang
Jiahuan Yan
Yuze GAO
Donovan Ong
Bin Chen
Jian Su
Venue: EMNLP 2023 Findings (2023)
Keywords: Text-to-SQL
large language model

Abstract: The success of ChatGPT has ignited an AI race, with researchers striving to develop new large language models (LLMs) that can match or surpass the language understanding and generation abilities of commercial ones.  In recent times, a number of models have emerged, claiming performance near that of GPT-3.5 or GPT-4 through various instruction-tuning methods.  As practitioners of Text-to-SQL parsing, we are grateful for their valuable contributions to open-source research.  However, it is important to approach these claims with a sense of scrutiny and ascertain the actual effectiveness of these models. Therefore, we pit six popular large language models against each other, systematically evaluating their Text-to-SQL parsing capability on nine benchmark datasets with five different prompting strategies, covering both zero-shot and few-shot scenarios. Regrettably, the open-sourced models fell significantly short of the performance achieved by closed-source models like GPT-3.5, highlighting the need for further work to bridge the performance gap between these models.

Battle of the Large Language Models: Dolly vs LLaMA vs Vieuna vs Guanaco vs Bard vs ChatGPT A Text-to-SQL Parsing Comparison
Shuo Sun', Yuchen Zhang 12, Jiahuan
Jiahuan Yan', Yuze Gao', Donovan Ong"*, Bin Chen', Jian Sul 'Inttittte for Infocomm Research (I²R), A*STAR, Singapore
*CNR CREATE LTD, Singapore, 'National University of Singapore "Nanyang Technology University
'(Sun_Shuo, Zhang Yuchen, Gao_ _uz_e Donovan_Onnn bchen, sujian] @ri2. aaaaaeir.aasaarrrr.......a.aa.aa a-star. edu "jiahuan_22e0) nus. edu, "s210060e nwt. edu. sg
Abstract
of these models to push the boundaries of Text-to- SQL capabilities.
The success of ChatGPT has ignited an Al race, with researchers striving to develop new large language models (LLMs) that can match or sur- pass the language understanding and generation abilities of commercial ones. In recent times, a number of models have emerged, claiming performance near that of GPT-3.5 or GPT-4 through various instruction-tuning methods. As practitioners of Text-to-SQL parsing, grateful for their valuable contributions to open- source research. However, it important to approach these claims with sense of scrutiny and ascertain the actual effectiveness of these models. Therefore, we pit six popular large language models against each other, system- atically evaluating their Text-to-SQL parsing capability on nine benchmark datasets with five different prompting strategies, covering both zero-shot few-shot scenarios. Regrettably, the open-soe sourced models fell significantly short of the performance achieved by closed-source models like GPT-3.5, highlighting the need further work to bridge the performance gap be- tween these models.
Recently, breakthroughs in decoder-based large language models (Brown et al., 2020b; Touvron et al., 2023) have further revolutionized the field of NLP. A prominent trend is the pursuit of training in- creasingly larger language models, encompassing billions of parameters, and utilizing vast amounts of textual data. Subsequently, these models are fine-tuned using iastruction-basee techniques, en- abling them to better adhere to human-generatee text prompts.
Among the prominent applications of decoder LLMs is ChatGPT, which is built upon OpenAl's GPT-3.5 and GPT-4 models. ChatGPT has demon- strated exceptional capabilities in zero-shot and few-shot scenarios, as evidenced by various Text- to-SQL evaluation studies (Rajkumar et al., 2022; Liu et al., 2023). Regrettably, the success of Chat- GPT has ignited an Al race, leading industry re- search labs to cease public disclosure of their model parameters and training methodologies.
Therefore, researchers have been actively pur- suing the development of new language models that can potentially rival the capabilities of Chat- GPT. These models include Dolly, which builds upon the Pythia models (Biderman et al., 2023), as well Vicuna (Chiang et al., 2023) and Gua- (Dettmers et al., 2023), based on the LLaMA models (Touvron et al., 2023). Some of them have garnered attention by claiming achieve perfor- mance levels surpassing 90% of GPT-4 through fine-tuning techniques.
1 Introduction
Text-to-SSe -OLL parsing automatically converts user input questions into SQL statements, enabling the retrieval of relevant information from databases. By enabling users to express their objectives in natural language, Text-to-SQL systems could min- imize the technical obstacles for non-expert users interact with relational databases and enhance productivity.
The introduction large pre-trained language models like BERT (Devlin et al., 2019) and T5 (Raffel et al.. 2020) has further improved the per- formance of Text-to-SQL systems. Researchers have been leveraging the impressive understanding
As practitioners of Text-to-SQL, ful for the contributions made by these models. However, remain uncertain about whether these open-source models truly deliver the level of qual- ity they claim to achieve. To address this concern, this paper presents a comprehensive evaluation
grate-
Work done when Donovan was working at FR

---

six language models: Dolly, LLaMA, Vicuna, Gua- naco, Bard, and ChatGPT, directly comparing their performance on nine benchmark datasets, utilizing five distinct prompting strategies.
conversations collected from ShareGPT which claims achieve 90% ChatGPT quality based on an automated evaluation with GPT-4. We also ex- periment with the 7B version.
Our main findings are:
Guanaco (Dettmers et al.. 2023) is a family of large language models claiming to achieve 99.3% of ChatGPT performance with only 24 hours of fine-tuning on GPU. Similar to Vicuna, the Guanaco model was instruction-tunee on the LLaMA models. We conduct our evaluations on the 33B version.
1. Open-source models demonstrate notably in- ferior performance compared to closed- source models across a majority of Text-to- SQL datasets.
2. While LLMs demonstrate proficiency in gen- erating syntactically valid SQL statements, they often struggle to produce semantically accurate queries.
Bard is a conversational chatbot released by Google as an answer to OpenAl's chatGPT. is ini- tially powered by LaMDA (Thoppilan et al., 2022) and later transitioned to PaLM 2 (Anil et al., 2023). As the technical details of Bard are not publicly released, evaluate its performance on Text-to- SQL datasets as a black box model. We denote the version powered by LaMDA as Bard-L. and the version powered by PaLM 2 as Bard-P2.
3. LLMs prove to be highly sensitive to the ex- amples utilized for few-shot learning.
To facilitate further research in the field of Text-to-SQL parsing, we are making all raw and post-processed outputs from the large lan- guage models publicly at https: /tt:://ittttt thub.hu..c.. com/ Thangg/ichenV//eeeeeval-xxxx2
GPT-3.5 (Brown et al., 2020a) is OpenAl's most cost-effective large language model optimized for chat-based applications at the point of writing this paper. It has 175 billion parameters and powers the popular ChatGPT chatbot. We conduct all eval- uations on the "GPT-3.5tturbo-030 variant of GPT-3.5 through openAl's API.
2 Experiment Setup
2.1 The Large Language Model Contestants We have carefully selected contestants from five families of large language models to provide a com- prehensive representation of the current landscape in the field:
2.2 Prompting Strategies
We explore five commonly used prompting strate- gies:
Dolly' is 12 billion parameter language model, claimed to be the first publicly available instruction- tuned LLM licensed for both academic and com- mercial applications. It is based on Pythia (Bider- man et al., 2023) and has undergone fine-tuning using an instruction dataset created by Databricks employees. Additionally, we have experimented with smaller variants of Dolly, including the 3B and 7B versions.
The Informal Schema (IS) strategy provides description of tables and their associated columns in natural language. In this approach, the schema information expressed in less formal manner. contrast, the API Does (AD) strategy, as outlined in the evaluation conducted by Rajkumar et al. (2022), follows the default SQL translate prompt provided in OpenAl's s documentation". This prompt adheres to slightly more formal definition of the database schema. The Select 3 strategy includes three ex- ample rows for each table in the database. This additional information aims to provide concrete examples of the data contained within each ta- ble, supplementing the schema description. We also investigate the effectiveness of 1-Shot Learn- ing (1SL) and 5-Shot Learning (5SL) strategies. where we provide one and five golden examples
LLaMA (Touvron et al., 2023) is collection of large language models ranging from 7 billion 65 billion parameters. These models have been trained exclusively on publicly available text cor- pora. Unlike the other models, LLaMA models are not instruction fine-tuned. To ensure the natural continuation of prompts, we append the keyword "SELECT" at the end of the prompts when inter- acting with LLaMA models.
Vicuna (Chiang al., 2023) is a 13 billion pa- rameter LLaMA model fine-tuned user-shared
hnttps://shoregpt.co http::://bbrr.goglle. com/
'https://platform. www.m/ppeeaacic..mmmmmmm wttpa/ai...mm///aaaaaaaaeaa default-sql-translan

---

in the prompts, respectively. Examples of various prompting strategies can be found in the appendix.
'he model may inadvertently learn to memorize the specific SQL queries rather than understanding the underlying semantic parsing. Therefore, we employ the query-based data splits described Finegan-Dollak et al. (2018) where SQL queries with similar structures are assigned to the same splits. As result, some of our reported results are lower than those documented in previous work such as Rajkumar et al. (2022).
2.3
Benchmark Datasets
Dataset
Number of examples
Academic
196
ATTS
347
GeoQuery
182
We also want to highlight that unlike Suhr et al. (2020); Lan al. (2023) who use the filtered combi- nations of train and dev splits of GeoQuery, Scholar and Advising and the filtered dev split of ATIS as their evaluations sets, we adhere to the evaluation splits in Finegan-Dollak et al. (2018). The reason behind this decision was our intention to sample examples from the train sets for the purpose of conducting one-shot and five-shot prompting ex- periments.
Yelp
128
IMDB
131
Restaurants
378
Scholar
315
Advising
1832
Spider
1034
Table 1: Number of examples used for evaluation of various Text-to-SLL datasets.
We evaluate the prompting strategics on nine Text-to-SQL datasets: Academic (Li and Jagadish, 2014), ATIS (Price, 1990; Dahl et al., 1994), Geo- Query (Zelle and Mooney, 1996), Yelp and IMDB (Yaghmazadeh et al., 2017), Restaurants (Tang and Mooney, 2000; Popescu al., 2004), Scholar (Iyer al., 2017), Advising (Finegan- Dollak et al., 2018) and Spider (Yu et al., 2018). Note that for the first eight datasets, we employ the standardized and improved versions released by Finegan-Dollae et al. (2018).
2.4 Evaluation Metries
The primary evaluation metric employed this paper is the execution accuracy (EX), which mea- sures the percentage of generated SQL queries that preciscly align with the outputs of the gold SQL queries. Additionally, for the Spider dataset, we also calculate the test suite accuracy (TS), which serves the official evaluation metric for this dataset. TS provides an upper-bound estimation of semantic accuracy by assessing the execution accuracy of predicted queries on a set of distilled randomly generated databases (Zhong et al., 2020). Similar to Liu et al. (2023). we refrain from uti- lizing the exact match accuracy (Yu et al., 2018) metric SQL query can often be expressed in multiple equivalent ways to achieve the same ob- jective. Consequently, exact match accuracy may inadvertently penalize large language models that generate SQL queries that differ in style from the gold data.
Trinegan-Doolak
Following Zhong et al. (2020), we evaluate the performance of large language models on the test sets of the datasets if such sets were defined. In cases where test sets were not provided, we evalu- ated the models on the entire datasets. For the Spi- der dataset, we evaluate the models on the develop- ment set since the test set is not publicly available at the point of writing this paper. To maintain con- sistency with Zhong et al. (2020), we collectively refer to the first eight datasets as the "classical datasets".
2.5 Evaluation Details
Important details on data splits Classical Text- to-SQL datasets such as GeoQuery, ATIS and Scholar often utilize a nuestion-bbsed data split strategy where matching (text, SQL) pairs are as- signed to the same split. However, one potential issue with this split strategy that the same SQL query could appear in both the training and test- ing set. This duplication of SQL. queries across the training and testing sets can introduce a bias and potentially inflate the model's performance.
We utilize several models our study, including three variants of Dolly (v2-3b, v2-7b, and v2-12b), two variants of Vicuna (7B and 13B), one variant of Guanaco (33B), and four variants of LLaMA (7B, 13B, 30B, and 65B). To ensure consistency, aim to adhere closely the default hyperparam- eters of each model. We set top-p sampling rate of 0.92 and temperature of 0.8, for Dolly, a tem- perature of 0.8 for Vicuna and Guanaco and top-p sampling rate of 0.95 and a temperature of 0.8 for

---

LLaMA. During the evaluation, we conduct our ex- periments on server equipped with eight NVIDIA RTX A6000 GPUs. For Bard, we have developed a script that directly extracts evaluation outputs from its web user interface. For GPT3.5, we leverage version through OpenAl's API and adhere to the default hyperparameters of a temperature of 1.0 and top-p sampling rate of 1.0.
deteriorates the performance of GPT-3.5.
Few-shot learning with random examples of- fers limited performance gains: The majority of results obtained from ISL and 5SL tend to under- perform or, at best, achieve comparable results to other prompting strategies. However, there are a couple of exceptions to this trend. One exception is the Dolly model, which shows improved perfor- mance with the ISL prompting strategy compared to other prompting strategies in the 12B variant. This result appears to be anomalous because similar gains in performance are not observed in other 1SL and 5SL results. The other exception is the LLaMA model, where the few-shot prompting strategies outperform some of the zero-shot strategies. For example, the 30B LLaMA model achieves 22.4% EX and 19.9% TS accuracy with only 5 given ex- amples, which is close to the performance of the Guanaco model (24.4% EX and 19.0% TS).
the "gpt-3.5-turbo-0300
3 Evaluation Results
3.1 Spider Dataset
The accuracy of execution (EX) and the accuracy of the test suite (TS) on various combinations of prompting strategies and models are presented in Table 2. Our main findings are:
Closed-source models exhibit superior perfor- mance compared to open-source models: GPT- 3.5 is the leading model, surpassing the second- place Bard model by 17.8% in execution accuracy (EX) and by 14.1% 1 in test suite accuracy (TS). However, GPT-3.5 still falls behind state-of-thes thefthe-artt Text-to-SQL models like the one proposed by Li al. (2023a) by margin of least 13% in abso- lute terms. Bard-P2 demonstrates some enhanced performance over Bard-L when employing the IS, S3, and 5SL prompting strategies, but suffers from significant performance degradations when using the AD and ISL prompting strategies.
3.2 Classical Datasets
Since there are no training sets for Academic, Restaurants, IMDB and Yelp, we sample exam- ples for ISL and 5SL from the evaluation sets of other classical datasets. We highlight some of our key findings based on the results in Table 3;
LIMs show lacklustre performance on most of the classical datasets: In particular, there a noticeable disparity in the results obtained from the Academic and Restaurants datasets when compared to the baseline performance reported in previous research. The highest achieved accuracies on these datasets merely 2.9% and 2.4% respectively, which is significantly lower than the baseline sults of 34.0% and 45.2% observed in other studies utilizing traditional seq2seq models with LSTM or BERT (Devlin et al., 2019). Furthermore, even with instruction tuning, Vicuna, Guanaco and Dolly face considerable challenges on the classical datasets. They often yield nearly zero execution accuracies across various prompt strategies and datasets com- binations.
Open-source models struggle with the Spider dataset: Despite positive correlation between the number of parameters and model performance, open-source models face challenges in achieving high accuracies on the Spider dataset. For example, although Vicuna 7B and 13B have demonstrated improvements over the raw pre-trained LLaMA 7B and 13B models, there still exists signifi- cant gap in performance when compared to Bard and GPT-3.5. Furthermore, the Dolly models also demonstrate underperformance when compared to LLaMA's 13B version across different prompting strategies.
The performance of LLMs are highly sensi- tive to the styles of prompts: Our empirical find- ings confirm that there is no universal prompting strategy that works well across all models. While the IS prompting strategy proves effective for GPT- 3.5, Bard, Vicuna, and Guanaco, yields subopti- mal accuracies for Dolly and LLaMA. Surprisingly, LLaMA achieves its optimal results when employ- ing the S3 prompts, which in contrast, significantly
The effectiveness of few-shot learning varies across different models: In contrast to the find- ings from the Spider dataset, we observe some per- formance improvements LLaMA and GPT-3.5 with ISL and 5SL. For example, the performance of GPT-3.5 on the GeoQuery dataset improves from 15. 15.4% to 42.3% with ISI,, while the performance of LLaMA on the same dataset also significantly improves from 12. 1 1 to 15.4% with 5SL. However,

---

Prompting Strategies
IS
AD
S3
1SL
5SL
Models
#p
EX
TS
EX
TS
EX
TS
EX
TS
EX
TS
3B
8.7
5.5
2.2
1.5
0.2
0.2
1.4
0.8
0.6
0.2
Dolly
7B
9.2
6.5
0.5
0.3
0.6
0.4
1.7
1.3
0.4
0.3
12B
9.5
7.4
0.1
0.1
0.8
0.6
11.8
9.3
5.3
3.7
7B
4.1
2.1
2.9
2.3
1 11.3
7.4
7.5
6.0
9.0
8.0
13B
8.7
4.8
6.1
4.4
16.2
12.8
13.5
11.4
14.4
13.2
LLaMA
30B
4.7
3.3
10.4
7.5
18.5
13.9
18.8
15.5
22.4
19.9
65B
12.2
9.1
14.0
10.5
29.5
23,9
23.7
19.2
23.8
20.1
7B
25.8
19.6
17.6
13.8
18.2
13.8
16.7
13.2
5.0
3.7
Vicuna
13B
34.8
26.5
21.2
16.8
9.5
6.5
19.1
14.5
18.8
15.8
Guanaco
33B
24.4
19.0
14.5
10.6
15,8
12.9
10.3
7.9
2.0
1.6
Bard-L
53.6
46.6
52.5
45.1
53.1
45.6
50.5
43.9
51.8
45.0
UNK
Bard-P2
60.2
52.3
48.7
41.8
54.6
46.1
47.8
41.4
53.6
46.9
GPT-3.5
175B
70.9
59.4
67.2
57.9
31.1
27.0
67.5
58.2
70.4
59.7
Table 2: Execution accuracy (EX) and test suite accuracy (TS) results for various large language models and prompting strategies (Informal Schema (IS), API Does (AD), Select (S3), 1-shot learning (ISL) and 5-shot learning (5SL)) on the Spider development set. The number of parameters (#p) for Bard is unknown (UNK), Highlighted bold are the best results in each LLM family.
we do not see similar performance improvements with ISL 5SL for Dolly, Vicuna and Bard. Appending database example rows is ineffec- tive: Just like the outcomes observed with the spi- der dataset, the S3 prompting strategies yield sub- par results when applied to the classical datasets across different models. Therefore, it is evident that the S3 prompting strategy may not be effective in the context of Text-to-SQL.
els, excluding Dolly, consistently generate valid SQL responses over 90% of the time with IS, ISL, and 5SL prompt strategies. Interestingly, LLaMA also demonstrates the ability generate valid SQL statements, even though was not specifically fine-tuned on instruction datasets. For the clas- sical datasets, Bard-P2 and GPT-3.5 are still ca- pable of generating valid SQLs in the 80-100% range. However, the open-source models such as Vicuna and Dolly encounter challenges in achiev- ing valid SQL percentage above 75%. What's particularly noteworthy is the divergent trends ob- served in LLaMA and Guanaco. LLaMA gener- ates more valid SQLs through few-shot learning, whereas Guanaco's performance declines as the number of examples increases.
4 Discussions
4.1 Are LLMs generating valid SQLs?
One potential explanation for the underwhelm- ing performance of large language models lies in their inability to grasp the intention behind prompts aimed generating SQL statements. For instance, in response to the question "re- turn me the homepage of PVLDB," Guanaco directly provided the answer "The website is https//wwww.vldb.orggmoe
Furthermore, noticed that the AD and S3 prompting strategies are generally suboptimal, they lead to significant decreases in the number of valid SQL responses across dltasetss for many large language models. GPT-3.5 is particularly susceptible to the S3 prompting strategy, resulting in sharp decline in the percentage of valid SQLs generated in both the spider and classical datasets.
When faced with many S3 prompts, GPT-3.5 fails to generate valid responses. To assess the extent of such instances, we plotted the proportion of valid SQL statements generated using different prompt strategies for var- ious large language models Figure la and lb. For spider dataset, we discovered that many mod-
Lastly, it is crucial to emphasize that although these language models can produce valid SQL re-

---

Model
#P
PS
Acad
ATIS
Adv
Geo
IMDB
Rest
Sch
Yelp
AVG
IS
0.0
0.0
0.0
8.2
3.8
0.0
0.0
3.1
1.9
AD
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Dolly 2.0
12B
S3
0.0
0.0
0.0
0.0
1.5
0.0
0.0
0.8
0.3
ISL
0.5
0.0
0.0
0.0
0.0
0.0
0.0
2.3
0.4
5SL
0.5
0.0
0.0
0.0
0.8
0.0
0.0
0.8
0.3
IS
0.0
0.3
0.0
0.5
0.0
0.0
0.0
0.0
0.1
AD
0.0
0.0
0.0
12.1
3.1
0.0
0.3
1.6
2.1
LLaMA
30B
S3
1.0
0.3
0.0
2.7
0.0
0.0
0.3
1.6
0.7
ISL
0.0
0.0
0.0
10.4
0.8
0.0
0.3
0.0
1.4
5SL
0.5
0.6
0.1
15.4
0.0
0.0
2.5
1.6
2.6
IS
2.0
0.0
0.0
0.5
6.1
0.0
0.3
2.3
1.4
AD
0.0
0.0
0.0
1.6
4.6
0.0
0.3
1.6
1.0
Vicuna
13B
S3
1.0
0.0
0.0
4.9
3.1
0.0
1.0
2.3
1.5
1SL
0.0
0.3
0.1
4.9
0.0
0.0
0.0
0.8
0.8
5SL
0.0
0.3
2.7
0.0
0.0
0.3
0.8
0.5
IS
0.5
0.9
0.0
1.1
1.5
0.0
0.0
2.3
0.8
AD
0.6
0.0
2.7
0.8
0.0
0.0
0.8
0.9
Guanaco
33B
S3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1SL
0.0
0.0
0.0
0.5
0.0
0.0
0.3
0.0
0.1
5SL
0.0
0.0
0.0
0.5
0.0
0.0
0.0
0.0
0.1
IS
6.6
0.0
0.0
8.2
17.6
0.0
0.3
3.9
4.6
AD
5.6
0.3
0.0
7.7
11.5
0.0
0.3
3.1
3.6
Bard-P2
UNK
S3
3.1
1.2
0.0
4.4
7.6
0.0
0.3
6.2
2.9
ISL
3.1
0.9
0.0
13.2
12.2
0.0
1.0
3.9
4.3
5SL
1.0
0.9
0.0
3.8
10.7
0.0
0.0
2.3
2.3
IS
13.8
1.4
0.0
15.4
17.6
2.4
2.2
4.7
7.2
AD
10.7
0.9
0.0
11.5
16.0
0.8
1.0
2.3
5.4
GPT-3.5
175B
S3
1.5
0.0
0.0
3.3
5.3
0.0
0.0
1.6
1.5
1SL
11.2
2.6
0.3
42.3
18.3
2.4
6.3
6.2
11.2
5SL
6.1
0.3
39.6
16.8
1.6
3.5
10.2
10.1
text2sql-data
75.0
34.0
8.0
49.0
24.0
33.0
6.0
32.0
32.6
XSP
12.1
33.3
45.2
49.2
Unite
41.1
Table 3: EX and TS results for various LLMS and prompting strategies classical datasets: Academic, ATIS, Advising. GeoQuery, IMDB, Restaurant, Scholar and Yelp. We also include baseline results in Finegan-Dollak et al. (2018) (text2sll-data), XSP Suhr et al. (2020) (XSP) and Lan al. (2023) (Unite) possible. Highlighted bold are the best results in each LLM family.
sponses, these SQLs are often semantically inac- curate and fail to adequately address the input text questions. As a consequence, the execution accu- racies across the majority of datasets are notably low.
4.2 How does sample selection affect the performance 1SL and 5SL?
Based on the results presented in Table 2 and Table 3, it becomes evident that the inclusion of random examples from the training in prompts does not significantly enhance the performance of different models. The only exceptions are LLaMA and GPT- 3.5, which demonstrate noticeable improvements

---

SELECT PUBLICATIONO1 las ABSTRACT FROM PUBLICATION AS IALLICATTONNaass WHERE PUBLICATIONal TITLE "Making database systems usable";
0.9
Proportion of valid SQL
0.8
0.7
Figure 2: Example of canonicalized SQL statement.
0.6
0.5
SQLs with ISL and 5SL, we examine the propor- tion of generated SQL statements that contain the term "alias" in Table 4. Our findings reveal that the change in generated SQL style only evident when employing the ISL and 5SL prompting strategies. Notably, LLaMA stands out among all the models, as it consistently appends the term "alias" to over 86% of the generated SQL. statements. Interest- ingly, Bard is less sensitive to the canonicalized SQL style, with style change observed in only 16.0% of all generated SQLs. On the other hand, GPT-3.5 demonstrates higher sensitivity, with more than 50% of the generated SQLs being affected. Based on this observation, we hypothesize that this disparity in sensitivity could be a contribut- ing factor to the greater success the ISL and 5SL prompting strategies employed by LLaMA and GPT-3.5.
0.4
0.3
Delly (120)
0.2
Vican (130)
Guarzoo (310)
0.1
Bud
GPT
AD
S3
ISL
5SL
Prompt Strategy
(a) Spider dataset
0.9
Propertion of valid SQL
0.8
0.7
0.6
0.5
0.4
Guurazo (33B)
0.3
0.2
0.1
IS
AD
S3
ISL
5SL
Prompt Strategy
Model
AD
1S
S3
1SL
SSL
(b) Classical datasets
Dolly (12B)
0.1
0.0
0.0
37.8
38.0
Figure The proportion of valid SQL generated for different prompting strategies across multiple models. Figure (a) presents the results on Spider, while Figure (b) presents average results based on the classical datasets.
LLaMA (30B)
0.0
0.1
0.1
86.1
92.0
Vicuna (13B)
0.0
0.1
0.1
37.8
38.5
Guanaco (33B)
0.0
0.1
0.2
1.8
1.8
Bard-P2
0.0
0.0
0.0
16.0
28.3
GPT-3.5
0.0
0.0
0.0
51.6
58.3
across most classical datasets when utilizing ISL and 5SL. prompting strategies. The improvements in LLaMA's performance with ISL or 5SL prompt- ing strategies can be partially attributed to the fact that exposing LLaMA to more examples substan- tially enhances its ability to generate valid SQL, as depicted in Figure lb.
Table 4: Percentage of generated SQLs containing the word "alias" for classical datasets.
Impact of sampling from different sources on Performance
We conclude this section by providing brief discussion on experiments involving the sampling of examples from sources other than the training sets. Table presents the ISL and 5SL results ob- tained when samples are taken from two different sources: 1) the Spider train set, and 2) the evalua- tion sets. In the second case, we take precautions to avoid any potential answer leakage, by filtering out all examples that have the same SQL answer as the question of interest. We find that using examples from the Spider dataset not only fails to yield any benefits but also leads to a decline the perfor-
LLMs adapt to the canonicalized SQL style: An- other noteworthy observation is that when large language models presented with examples from classical datasets, they start generate SQL in style similar to canonicalized format described Finegan-Dollat et al. (2018), as shown in Figure 2, where tables alias follow a standardized conven- tion of <TABLE NAME>alias<NN
Sensitivity LLMss to style change: To assess the extent to which Language Models (LLMs) fol- low the canonicalized SQL style while generating

---

mance of the models, performing worse than the zero-shot methods. On the other hand, when we include examples from the evaluation sets, we ob- serve improvements in the evaluation results. Upon closer examination of the prompts, we discovered instances where the few-shot examples were syn- tactically similar to the expected SQL responses, differing primarily in terms of tables, columns, and values. This finding highlights the sensitivity of LLMs to the examples provided in the prompt. We hypothesize that LLMs may generate more accu- rate SQL statements if we feed them with examples that are syntactically close to the expected SQL re- sponse.
5 Related Work
Recently, decoder-based large language mod- els have contributed tremendously the code- generation tasks (Li et al., 2023b; Fu et al., 2023; Darm et al., 2023). These models leverage unsuper- vised auto-regressive learning on large-scale text data, allowing them to capture rich semantic re- lationships and probability distributions of words, Despite their remarkable performance with just or few-shot examples in context, recent research suggests that they still face challenges on the Text- to-SQL task, which involves complex reasoning (Liu et al., 2023).
There are several works which focus improv- ing the text-to-SQL parsing capabilities of large language models through enhanced prompt designs. In one study conducted by Nan al. (2023), the au- thors emphasize the significance of carefully select- ing examples for in-context learning. They demon- strate that incorporating syntactic structures from example queries can greatly enhance the few-shot capabilities of large language models. Chang and Fosler-Lussier (2023) conducted comprehensive study that explores the impact of prompt length on the performance of text-to-SQL models. Ad- ditionally, they examine the sensitivities of repre- sentations of database knowledge across various domains. Guo et al. (2023) propose a case-based reasoning framework that adjusts the inputs for GPT-3.5 in cross cross-domain settings by adaptively re- trieving case prompts. Rai et al. (2023) improve the generalization capabilities of large language models with boundary- techniques that pre- process prompts at both token-level and sequence- level of schema and SQL.
Model
Train
Spider
Eval
Dolly (12B)
1.5/0.5
0.4/0.4
0.8/0.3
LLaMA (30B)
1,4/2.6
1.0/1.3
8.2/17.8
Vicuna (13B)
0.8/0.5
0.7/0.4
2.2/4.9
Guanaco (33B)
0.1/0.1
0.0/0.0
0.1/0.0
Bard-P2
4.3/2.3
2.9/2.9
7.1/15.4
GPT-3.5
11.2/10.1
6.6/7.6
16.6/28.2
Table 5: Mean 1SL/5SL EX results when sampling from the train sets, Spider train set and evaluation sets.
4.3 Are we truly evaluating the Text-to-SQL. datasets in zero-shot or few-shot manner?
We have identified several potential sources of data contamination (Elangovan et al., 2021: Lewis et al.. 2021: Magar and Schwartz, 2022) that raise con- cerns about the true nature of zero-shot or few-shot evaluations of Text-to-SQL. datasets. These sources include the availability of both the Spider dataset and classical datasets on GitHub repositories, as well as the presence of the Spider dataset on plat- forms like Huggingface datasets Furthermore, the TeXI-to-SQL datasets may also be included in instruction-tuning dataset collections such as FLAN (Wei et al.). We end the paper with a ques- tion for researchers to contemplate: Are we gen- uinely conducting zero-shot or few-shot evalua- tions of large language models when they have already been exposed to our evaluation data?
Concurrently, some studies have also explored the potential benefits of complex, multi-step rea- soning in improving the performance of large lan- guage models on text-to-SQL parsing. Tai et al. (2023) show that Least-to-Men prompting (Zhou et al.. 2023) might be unnecessary and directly ap- plying chain-of-thought (CoT) prompts (Wei et al., 2022) could lead to error propagation. Liu and Tan (2023) introduce divide and prompt paradigm for the Text-to-SQL. task, which involves dividing the task into multiple subtasks and applying the CoT approach each subtask. In another study by Pourreza and Rafiii (2023), self-correction mod- ule is employed in a zero-shot setting to achieve a new state-of- the-art result on the Spider leader- board. This module feeds the solution of each
'https: httpa///a.....t.taattttsss/pppaa

---

sub-problem back to the large language model, en- abling it to construct a better overall solution.
Acknowledgments
This research is partially supported by the pro- gramme DesCartes funded by the National Re- search Foundation, Prime Minister's Office, Singa- pore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
6 Conclusions and Future Work
This paper systematically evaluates the Text-to- SQL parsing capabilities of six popular large lan- guage models across nine benchmark datasets, us- ing five distinct prompt strategies. Our findings indicate that the open-sourced models fall signif- icantly short in performance when compared to closed-source models. However, it is worth not- ing that even GPT-3.5 performs worse than smaller baseline models on several classical datasets. we are making outputs available for further anal- ysis and to facilitate future research endeavors. There are several research topics we want to ex- plore in the future. Firstly, we plan to investigate the fine-tuning of these large language models on Text-to-SQL. datasets using limited GPU resources using techniques such as low-rank adaptation Hu et al. (2021). Second, we want to explore methods that can dynamically select examples for in-context learning. Third, we would also explore applying model compression techniques recent Text-to- SQL models (Sun al., 2023). Lastly, we are interested in examining the feasibility and limita- tions of employing these large language models on multi-turn Text-to-SQL. datasets, such as the SPARC (Yu et al., 2019).
References
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent Shafey, Yanping Huang, Kathy Meier-Hellstern, Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du Ethan Dyer, Vlad Feinherg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Gar- cia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur- Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui. Jeremy Hur- witz, Michael Isard, Abe Ittycheriah, Matthew Jagiel- ski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Ben- jamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra. Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nys- trom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley. Alex Castro Ros, Au- rko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So. Daniel Sohn, Simon Tokumine, Dasha Valter. Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wiet- ing, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report.
Limitations
First and foremost, we acknowledge that the scope of this research is limited to six large language models and these models do not encompass the en- tire research landscape. There have been new and exciting entries to the family, such as the Falcon model. 6 Second, appending five examples to the database schema of some classical datasets might exceed the 2048 token limits of the open-source models in some cases, leading to truncations that might penalize these models with shorter context window. Lastly, some models generate not just SQL statements but also supplementary informa- tion, including explanations. To ensure accuracy, we have developed regular expression patterns that aim to extract only the SQL statements to the best acknowledge
Stella Biderman, Hailey Schoelkopf, Quentin Anthony. Herbie Bradley. Kyle O' Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
of our abilities. Nevertheless, that our rules may not be entirely foolproof and could potentially introduce erroncous SQL in cer- tain cases.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
"https://falccas tii.

---

Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Zhang, and Dragomir Radev. 2018. Improving text- to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramu- ditha Perera, Henghui Zhu, Yuhao Zhang, Alexan- der Hanbo Li, William Yang Wang. Zhiguo Wang. Vittorio Castelli, Patrick Ng. Dan Roth, and Bing Xi- ang. 2023. Generate then select: Open-ended visual question answering guided by world knowledge.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Chunxi Guo, Zhiliang Tian, Jintao Tang, Pancheng Wang, Zhihua Wen, Kang Yang, and Ting Wang- 2023. A case-based reasoning framework for adap- tive prompting in cross cross-domain text-to-sql.
Shuaichen Chang and Eric Fosler-Lussier. 2023. How prompt lls for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhn Zhu,, Yvanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Srinivasan Iyer. loannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learn- ing a neural semantic parser from user feedback.
Wuwei Lan, Zhiguo Wang. Anuj Chauhan, Henghui Zhu, Alexander Li, Jiang Guo, Sheng Zhang. Chung- Wei Hang. Joseph Lilien, Yiqun Hu, Lin Pan, Ming- wen Dong, Jun Wang, Jiarong Jiang, Stephen Ash. Vittorio Castelli, Patrick Ng. and Bing Xiang. 2023. Unite: A unified benchmark for text-to-sql evalua- tion.
Deborah A. Dahl. Madeleine Bates, Michael Brown, William Fisher, Kate Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Tech- nology: Proceedings of a Workshop held at Plains- boro, New Jersey, March 8-11, 1994.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open- domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1000-1008, Online. Association for Computational Linguistics
Paul Darm, Antonio Valerio Miceli-Bande BeriieBarone, Shay B. Cohen, and Annalisa Riccardi. 2023. Knowledge base question answering for space debris queries.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Ilms.
Fei Li and Hosagrahar V Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.
Peng Li, Tianxiang Sun, Qiong Tang. Hang Yan, Yuan- bin Wu, Xuanjing Huang, and Xipeng Qiu. 2023b. Codeie: Large code generation models are better few- shot information extractors.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization Quantify- ing data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1325-1335, Online. Association for Computational Linguistics.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehensive evaluation of chatgpt's zero- shot text-to-sql capability.
Catherine Finegan-Doie Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui
Xiping Liu and Zhao Tan. 2023. Divide and prompt:
Chain of thought prompting for text-to-sql.

---

Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pa- pers), pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
parsing. In 2000 Joint SIGDAT Conference on Em- pirical Methods in Natural Language Processing and Very Large Corpora, pages 133-141.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker. Yu Du, YaGuang Li. Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali. Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu. Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstems Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog appli- cations.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing few-shot text-to- sql capabilities of large language models: study on prompt design strategies.
Ana-Maria Popescu, Alex Armanasu, Oren Etzioni, David Ko, and Alexander Yates. 2004. Modern nat- ural language interfaces to databases: Composing statistical parsing with semantic tractability. In COL-
ING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 141- 147, Geneva, Switzerland, COLING.
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text- to-sql with self-correction.
Patti J. Price. 1990. Evaluation of spoken language systems: the atis domain. In Human Language Tech- nology- - The Baltic Perspectiv.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Daking Rai, Bailin Wang, Yilun Zhou, and Ziyu Yao. 2023. Improving generalization in language model- based text-to-sql semantic parsing: Two simple se- mantic boundary-based techniques.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1990.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabilities of large language models.
Alane Suhr, Ming-Wat Wei Chang, Peter Shaw, and Ken- ton Lee. 2020. Exploring unexplored generalization challenges for cross- database semantic parsing. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8372- 8388, Online. Association for Computational Lin- guistics.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Type- and content-driven syn- thesis of sql queries from natural language.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labell dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Shuo Sun, Yuze Gao, Yuchen Zhang. Jian Su, Bin Chen, Yingzhan Lin, and Shuqi Sun. 2023. An exploratory study on model compression for text-to-sql. In Find- ings of the Association for Computational Linguistics: ACL 202.3 pages 1647--11544.
Chang-You You Tai. Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. 2023. Exploring chain-of- of-ttoughtt style prompting for text-to-sql.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li. Heyang Er, Irene Li, Bo Pang. Tao Chen, Emily Ji. Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vin- cent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev, 2019. SParC: Cross-domain se- mantic parsing in context. In Proceedings of the
Lappoon R Tang and Raymond Mooney. 2000. Auto- mated construction of database interfaces: Intergrat- ing statistical and relational learning for semantic

---

57th Annual Meeting of the Association for Computa- tional Linguistics, pages 4511-4523, Florence, Italy. Association for Computational Linguistics.
John M. Zelle and Raymond J. Mooney. 1996. Learn- ing parse database queries using inductive logic programming. In AAAI/IAA,, Vol. 2.
Ruiqi Zhong. Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-sql with distilled test suites. arXiv preprint arXiv:2010.02840.
Denny Zhou, Nathanael Schärli, Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le. and Ed Chi. 2023. Least-to-most prompting enables complex rea- soning in large language models.
A Appendix

---

Prompt Strotgyy AD
Prompt Strotey
SOLne with properties:
dotabase 'Highschoof' has tables named Highschooler, Friend 'Ukes:
Highachooler table has columne: 10"; "Namme (grade..
HighechooleriB,, neme, gradel
Friend tuble has columri: Studere. 1D/friend ID:
Frlandistudent_ J0, friend X4)
Ukes table columns: Studenn_II), !iked_I0:
Ulkesjtueent_ LD, Ilee__DI
Gave SCU query: Hyle's No need explanation.
SQL query: Kyle's
SELECT
Examples of prompt strategies IS in the Spider datasct
Examples of prompt strategies AD the Spider dataset
Frompt Strriagy Seacct
Prongt Strategy
"iggocheal mable named Nightthedle"" Triewd, 'Uooe Highedhoster balee has molure 10, "Manel leadde Frived callanee saudewr thrrt
soLECT 1MM consert LIMIT
Tall:.cnaant
conoer iD coroet, Theme Seskkmm
Student
Super snonompp Sine cheechholo (2014)
Transktte SOL
B. Werre Watit Weeding Loee 23199
How nome high schookes are there 301 count)") from rightchoolrr
Prompt Stretagy SRL
Aaanale amanfrrn hoon-ton SRIECT FROM riiger /MIT
"Eghsckeo" sakles named Nighocooerr' Triend; 'Wles.
Mike unger
Highedhosker biee has odlumn 10, "Mame" 'eaad"
Singe_10 Country
Seng_relssos_sce Age buymaee O. misbaardi Saetee Ststen: bergenzud, 2008;
Friond celenre Student (irloo
Staden, Wed..1.
(b, Suetin Brown, "toree Hey de: 2012) 20, (7)
Transkitt SCL
How mome high se schookos are therc! or SEUOCT court from Highshooler and gradi inghtehce
Using
following proesprovieslllee
nams, greza mighchoader
SELECT
Whet is grade whonlers SELFCT roste lighthooiiee
Courm
shoole? SELECT goode FROM Highsthooler
to need explrration
(c) Examples of prompt strategies Select in the Spider dataset
(d) Examples of prompt strategies 1SL/5SL in the Spider dataset
Figure Examples our prompt strategies in the Spider dataset
Model
#P
PS
Acad
ATIS
Ady
Geo
IMDB
Rest
Sch
Yelp
AVG
IS
0.5
0.3
0.0
0.5
0.0
0.0
0.0
1.6
0.4
AD
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
3
S3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
ISL
0.0
0.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
5SL
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
IS
0.5
0.6
0.0
0.5
2.3
0.0
0.0
1.6
0.7
AD
0.0
0.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Dolly
13B
S3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
ISL
0.0
0.3
0.0
0.0
0.0
0.0
0.0
1.6
0.2
5SL
0.0
0.3
0.0
1.6
0.0
0.0
0.0
0.0
0.2
IS
0.0
0.0
0.0
8.2
3.8
0.0
0.0
3.1
1.9
AD
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
12B
$3
0.0
0.0
0.0
0.0
1.5
0.0
0.0
0.8
0.3
ISL
0.5
0.6
0.0
8.2
0.0
0.0
0.3
2.3
1.5
5SL
0.5
0.3
0.0
1.1
0.8
0.0
0.3
0.8
0.5
Table EX and TS results for Dolly models on classical datasets: Academic, ATIS, Advising, GeoQuery, IMDB, Restaurant, Scholar and Yelp.

---

Model
#P
PS
Acad
ATTS
Adv
Geo
IMDB
Rest
Sch
Yelp
AVG
IS
2.0
0.6
0.0
5.5
3.8
0.0
0.3
2.3
AD
0.0
0.0
0.0
2.2
0.0
0.0
0.3
1.6
0.5
7B
S3
1.0
0.3
0.0
1.6
1.5
0.0
0.3
0.8
0.7
ISL
0.0
0.6
0.0
0.0
0.8
0.0
0.0
1.6
0.4
5SL
0.5
0.0
0.0
0.0
1.5
0.0
0.0
0.0
0.3
Vicuna
IS
2.0
0.0
0.0
0.5
6.1
0.0
0.3
2.3
1.4
AD
0.0
0.0
0.0
1.6
4.6
0.0
0.3
1.6
1.0
13B
S3
1.0
0.0
0.0
4.9
3.1
0.0
1.0
2.3
1.5
ISL
0.0
0.3
0.1
4.9
0.0
0.0
0.0
0.8
0.8
5SL
0.0
0.3
0.1
2.7
0.0
0.0
0.3
0.8
0.5
Table 7: EX and TS results for Vicuna models on classical datasets: Academic, ATIS, Advising, GeoQuery, IMDB, Restaurant, Scholar and Yelp.
Model
#P
PS
Acad
ATIS
Adv
Geo
IMDB
Rest
Sch
Yelp
AVG
IS
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
AD
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0,0
7B
S3
0.0
0.0
0.0
3.3
0.0
0.0
0.3
0.8
0.5
ISL
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
5SL.
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
IS
0.0
0.0
0.0
0.0
0.8
0.0
0.0
0.8
0.2
AD
0.0
0.0
0.0
1.6
0.8
0.0
0.3
0.8
0.4
13B
S3
1.0
0.6
0.0
13.2
3.1
0.0
0.3
1,6
2.5
ISI.
1.0
0.0
0.0
0.0
0.0
0.0
0.0
1.6
0.3
LLaMA
5SL
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.8
0.1
0.0
0.3
0.0
0.5
0.0
0.0
0.0
0.0
0.1
AD
0.0
0.0
0.0
12.1
3.1
0.0
0.3
1.6
2.1
30B
S3
1.0
0.3
0.0
2.7
0.0
0.0
0.3
1.6
0.7
ISL
0.0
0.0
0.0
10.4
0.8
0.0
0.3
0.0
1.4
5SL
0.5
0.6
0.1
15.4
0.0
0.0
2.5
1.6
2.6
0.5
0.0
0.0
1.6
0.8
0.0
0.0
2.3
0.7
AD
1.0
0.0
0.0
0.5
2.3
0.0
0.0
0.0
0.5
65B
S3
0.5
0.0
0.0
6.0
2.3
0.0
0.3
0.8
1.2
1SL
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
5SL
0.0
0.3
0.1
11.0
0.0
0.0
2.5
0.0
1.7
Table 8: EX and TS results for LLaMA models on classical datasets: Academic, ATIS, Advising, GeoQuery, IMDB, Restaurant, Scholar and Yelp.


Reviews:
[{"review_id": "pG28xD3rtw", "reviewer": "Reviewer_tPZP", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the reviewer for taking the time to read through our manuscript and would like to address some of the reasons for reject: \n\n“Lack of innovation”: We want to politely point out that this is an empirical paper which directly addresses one of the discussion topics in the theme track: “How reliably do the current generation of LLMs perform on NLP tasks and applications?” We argue that this paper is important to Text-to-SQL practitioners as it systematically examines how the current generation of prompt-based decoder LLMs perform on Text-to-SQL and provides insights on future research directions in this subfield of NLP. \n\nWe also want to politely argue that the reasons for rejection stated in “Lack of Clarity in methodology”, “comparison with State-of-the-Art Models” and “challenges with open-source models” can be easily fixed with some minor revisions to our manuscript. We would actively revise our paper to address these issues before the camera-ready deadline.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper offers a detailed and systematic comparison of various language models in the context of Text-to-SQL parsing. \n\nComprehensive Evaluation of Large Language Models (LLMs): The paper systematically evaluates six popular large language models on their Text-to-SQL parsing capabilities across nine benchmark datasets and five different prompting strategies. This extensive comparison provides valuable insights into the performance of both open-source and closed-source models.\n\nExploration of Prompting Strategies: The paper investigates various prompting strategies, including zero-shot and few-shot scenarios, to understand their impact on different models. This exploration can provide insights into how different models respond to various prompting techniques, which is a relatively new area of study.\n\nOpen source models vs closed source models : The significant performance gap between open-source and closed-source models like GPT-3.5 highlights the need for further research and development in the open-source community. This gap might limit the accessibility and applicability of state-of-the-art models for researchers and practitioners without access to closed-source technologies.\n\nSensitivity Analysis of Few-Shot Learning: The paper's analysis of how LLMs are highly sensitive to the examples utilized for few-shot learning is an interesting observation that could lead to further research in the area of few-shot learning techniques.\n\nPotential Impact on Text-to-SQL Parsing Research: By making all raw and post-processed outputs publicly available, the paper contributes to the broader research community, facilitating further work in the field of Text-to-SQL parsing.\n", "reasons_to_reject": "Lack of innovation : The paper appears to focus solely on text-to-SQL evaluation of existing models without introducing any new techniques or insights.\n\nLack of Clarity in Methodology: The paper might benefit from a more detailed explanation of the methodology, including the selection criteria for models, the rationale behind the prompting strategies, and the specific challenges faced by the models in different scenarios.\n\nComparison with State-of-the-Art Models: The paper should include a more comprehensive comparison with state-of-the-art models in Text-to-SQL parsing to provide a clearer context for the performance of the evaluated LLMs. \n\nChallenges with Open-Source Models: The paper notes that open-source models like Vicuna and Dolly face challenges in achieving valid SQL percentage above 75% on classical datasets. But the paper does not provide a thorough analysis of why these models underperform.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Title:\nBattle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison\n\nMain contributions:\n- Comparison of open source models and closed source models on various benchmarks for text-to-sql parsing.\n- Sensitivity to prompt strategies: The effectiveness of different prompting strategies which varies for various models, with no one strategy working for all models.\n- Inconsistent Performance Across Models and Datasets: The large language models (LLMs) evaluated in the paper show inconsistent performance across different datasets and prompting strategies. This inconsistency might raise concerns about the generalizability and robustness of the models.\n- Few shot learning limitations : Few-shot learning with random examples offered limited performance gains in most cases.\n\nModels Evaluated\nDolly: A 12 billion parameter model, instruction-tuned and licensed for both academic and commercial applications.\nLLaMA: A collection of models ranging from 7 billion to 65 billion parameters, trained exclusively on publicly available text corpora.\nVicuna: A 13 billion parameter LLaMA model fine-tuned on user-shared conversations.\nGuanaco: A family of models claiming to achieve 99.3% of ChatGPT performance with only 24 hours of fine-tuning.\nBard: A conversational chatbot by Google\nGPT-3.5: OpenAI's 175 billion parameter model optimized for chat-based applications.\n\nEvaluation Metrics\nThe primary evaluation metric is execution accuracy (EX), measuring the alignment of generated SQL queries with gold SQL queries. Test suite accuracy (TS) is also used for the Spider dataset.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7jGhIS0qff", "reviewer": "Reviewer_A359", "summary": "", "questions": "Question A: Are there any different behaviors among the predictions of those LLMs? Is there any tendency for some LLMs to make certain wrong predictions?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for reading our manuscript.\n\nWe want to politely point out that the goal of this paper is to systematically evaluate representative open-sourced and close-sourced large language models by applying the same set of commonly used zero-shot and few-shot prompting strategies. We argue that it is our controlled experiments which provide important empirical insights, revealing that most of these LLMs are not omnipotent.  The results are important as they can steer the Text-to-SQL community towards potential direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides references for the text-to-SQL parsing performance of several large language models.", "reasons_to_reject": "The paper reports LLM results for text-to-SQL parsing, but it lacks deeper analysis and insightful findings. In addition, most results look very bad which may indicate additional finetuning or some other strategies like Chain-of-thought is needed. For some LLMs, it may be not suitable to directly do in-context learning for text-to-SQL parsing.", "questions_to_authors": "Question A: Are there any different behaviors among the predictions of those LLMs? Is there any tendency for some LLMs to make certain wrong predictions?", "paper_topic_and_main_contributions": "The author tries to do a comprehensive study for text-to-SQL parsing in terms of different large language models, different prompting strategies and different text-to-SQL datasets. However, the paper lacks more concrete error analysis and deeper reasons which cause different performances among those models. And the novelty of this paper is also limited.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LAeOuECUYj", "reviewer": "Reviewer_9kjy", "summary": "", "questions": "A: Regarding the final acknowledgement in the \"Limitations\" section about SQL extraction using regular expressions, is that just a potential problem or an observed one? Has the potential impact on the results been quantified at all?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like the reviewer for your recognition and affirmation of our work. \n\nWe would like to use this opportunity to address your question “the final acknowledgement in the \"Limitations\" section about SQL extraction using regular expressions, is that just a potential problem or an observed one? Has the potential impact on the results been quantified at all?”\n\nResponse: Figure 1 shows the proportion of valid SQLs we extracted for each prompt strategy/model combination.  We did carefully review the non-sql responses and made sure our regular expressions did not miss out any valid SQL generations. We did not observe any erroneous extraction during our manual review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I find the results interesting as they contradict certain existing assumptions in the community regarding the current performance of open-source LLMs in comparison to closed-source ones.\n\nEven if the results are quickly (or already) not entirely representative of the landscape, I believe it could still be a useful resource in the future due to the quantity of the experimental results on a specific task (Text-to-SQL) and the relevant insights that the authors derive from them.", "reasons_to_reject": "Due to the nature of the subfield, such papers often quickly become obsolete. In this case, open-source models that could change the conclusions are already available. Falcon was mentioned in the paper itself, and there are claims that the very recently released LLaMA 2 beats GPT-3.5 in various tasks. Perhaps the latter could be added to the paper, assuming the procedure used for testing the original LLaMA is applicable to the newer version.", "questions_to_authors": "A: Regarding the final acknowledgement in the \"Limitations\" section about SQL extraction using regular expressions, is that just a potential problem or an observed one? Has the potential impact on the results been quantified at all?", "paper_topic_and_main_contributions": "The authors cover much of the current landscape when it comes to the Text-to-SQL parsing capability of various large language models, conducting original experiments and demonstrating that, at least in this specific use case, open-source models still produce significantly inferior results when compared to closed-source alternatives.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Satisfactory list of references.", "Typos_Grammar_Style_and_Presentation_Improvements": "Presentation and language satisfactory.\n\n099. five-->six", "Justification_for_Ethical_Concerns": "I do not believe that it is necessary for this paper to go through this process."}, "extra_text": {}}, {"review_id": "kr9RKXAIqp", "reviewer": "Reviewer_AYeh", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for taking the time to read through our manuscript and would like to address some your concerns: \n\n“At least 25 (out of 44 – more than 50%) of the references have no peer-review venue indication or are based on arXiv. This is not good.”: \nWe would like to politely point out that the field of LLM develops rapidly and it turns out most of the recent and important work on Text-to-SQL and LLMs are published on arXiv. In this context, we argue it may be unjust to penalize us based on a trend within the NLP landscape.\n\n“The paper’s novelty is very limited, if any. It only compares different LLMs along one single task: text-to-SQL (though over multiple datasets).” and “Although the objective of the paper is stated, it is not phrased as a research question or in a clear scientific motivation for investigation.”:\nAlthough this paper focuses on Text-to-SQL, we politely argue that this paper can still make a big contribution to this subfield of NLP. We would like to point out that this paper adequately addresses the theme track’s discussion topic on “How reliably do the current generation of LLMs perform on NLP tasks and applications?”. \n\n“The paper does not explain or hypothesize why the anomaly described on line 315 occurs”:\nWe believe this can be easily fixed with minor revisions to our manuscript and would like to thank the reviewer for the suggestion.\n\n“It is difficult to know from Table 2 which of the results are EX and which are TS.”:\nTS metric is currently only applicable to the Spider dataset, and Table 2 represents the performance EX scores. We would fix the table caption and apologize for any confusion caused by the unclear description.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper appears to be generally well-written.\n\nThe authors conduct relatively extensive experiments/analysis. \n\nThe authors provide some of the hyperparameters used, which will aid reproducibility.\n", "reasons_to_reject": "At least 25 (out of 44 – more than 50%) of the references have no peer-review venue indication or are based on arXiv. This is not good.\n\nThe paper’s novelty is very limited, if any. It only compares different LLMs along one single task: text-to-SQL (though over multiple datasets).\n\nAlthough the objective of the paper is stated, it is not phrased as a research question or in a clear scientific motivation for investigation.\n\nThe paper uses only one metric (EX) for evaluation of all models/experiments (except TS for the Spider dataset), thereby limiting the robustness of the results.\n\nThe paper does not explain or hypothesize why the anomaly described on line 315 occurs.\n\nIt is difficult to know from Table 2 which of the results are EX and which are TS.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors test six popular large language models, evaluating their Text-to-SQL parsing capability on nine benchmark datasets with five different prompting strategies, covering both zero-shot and few-shot scenarios and found open-source models to be lagging.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The Introduction explains what text-to-SQL parsing is but without any reference.\n\nThe paper can benefit from grammar-check and proof-reading for improvements. For example, line 96.\n\nUnnecessary repetition of some points made in the Section 2.1 about LLMs are repeated in 2.5. Foe example, about the version of ChatGPT used.\n"}}]

Author rebuttal:

Meta-review: This paper evaluates the text-to-SQL parsing ability of various large language models (LLMs), both open-source and closed-source. The paper compares different LLMs, such as Dolly, LLaMA, Vicuna, Guanaco, Bard, and GPT-3.5, on several text-to-SQL benchmarks, using different prompting strategies. The paper measures the execution accuracy (EX) and test suite accuracy (TS) of the generated SQL queries. The paper finds that open-source models are still far behind closed-source models in text-to-SQL parsing, and that there is no single prompting strategy that works well for all models. The paper also observes that few-shot learning with random examples has limited impact on the performance. The paper provides a comprehensive analysis of the current state-of-the-art and the challenges of text-to-SQL parsing with LLMs.
Decision: Accept-Findings