DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models

## Abstract

Abstract Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub , an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub , along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub .

## 1 Introduction

The task of text-to-SQL, which converts natural utterances into SQL queries, has emerged as a popular topic in both natural language processing and database (Yu et al., ; Deng et al., ). It effectively narrows the gap between non-expert users and database systems, significantly enhancing data processing efficiency. Essentially, text-to-SQL can be characterized as a sequence-to-sequence modeling task (Sutskever et al., ), where the database schema and the natural language question are transformed into a sequential input, while the desired SQL query serves as the sequential output target. Early works focus on fine-tuning domain-specific Transformer models and developing decoding techniques specifically for the task, leveraging SQL syntax, semantics, and the complex interplay between questions and databases (Scholak et al., ; Qi et al., ).

While recently large language models (LLMs) such as ChatGPT (Brown et al., ) and GPT-4 (OpenAI, ) have showcased their remarkable capabilities in engaging in human-like communication and understanding complex queries, LLMs have emerged as a new paradigm for text-to-SQL (Liu et al., ; Trummer, ). Notably, since 2023, the majority of top-performing solutions on the Spider leaderboard (Yale, ) have been methods based on LLMs.

The most recent advancement in this area involves employing LLMs for generating accurate SQL queries through in-context learning (ICL) techniques, notably zero-shot and few-shot prompting  (OpenAI, ; Dong et al., ; Pourreza and Rafiei, ). Beyond the inherent challenge of ambiguity and complexity, the laborious efforts for annotating SQL query-response exemplars by domain experts hinder the process of scaling-up data hungry LLMs for text-to-SQL applications. Meanwhile, another prominent approach is fine-tuning LLMs using additional task-specific training data to enhance their efficacy for text-to-SQL tasks  Li et al. (); Sun et al. (). The remarkable performances achieved in these works indicate the immense potential of fine-tuning. However, compared to prompting approaches, fine-tuning approaches have been relatively under-explored, partially attributed to the prohibitively high computational cost. Recent systematic studies  (Gao et al., ; Zhang et al., ) still mainly highlight the ICL abilities of LLMs and their accuracy in generating SQL queries in relevant tasks.

Up until now, there still has not been a universally acknowledged open benchmark for tuning approaches, which impedes researchers and practitioners from
comparing methods and reproducing results, potentially slowing down advancement in this field. As a first step towards addressing these challenges, in this work, we present a holistic framework, namely DB-GPT-Hub,.
Apart from existing works that mostly focus on few-shot prompting
strategies or tuning relatively smaller LLMs, our work focuses on tuning larger LLMs. In all, DB-GPT-Hub consolidates essential research assets (e.g., data, model services, evaluation methods, documentation) with following distinct merits:

- •
Standardization. We establish a standardized pipeline in an open-source codebase, with unified experimental settings and containerized environments, to enable transparent and consistent comparisons of LLM models after text-to-SQL tasks tuning.
- •
Comprehensiveness. We conduct extensive benchmarking that covers a range of medium to large-sized, fine-tuned LLMs across various experimental scenarios and explore their relative performance compared to prompting methods. Our work comprises one of the most pragmatic and expansive sets of benchmark suites available.
- •
Extensibility. As a rapidly evolving field, novel LLM-based methods constantly emerge, and the best practice continuously evolves. Following our documentation and protocols, one could effortlessly incorporate novel ingredients into our codebase: new datasets, new modules, new models (or model services), and new evaluation programs. Moreover, our framework offers easy compatibility with various prompting techniques. The high extensibility will eventually benefit the research area of text-to-SQL.

## 2 Background and Problem Formulation

### 2.1 A Generalized Setup

The input of text-to-SQL task is a natural language question $q$ and the database information $\mathcal{D}$. The output is the SQL $s$ corresponding to the question. The database $\mathcal{D}=\{S,K_{p},K_{f}\}$ includes database schema $S$, primary keys $K_{p}$ and foreign keys $K_{f}$, where $S$ usually contains multiple tables $T_{k}:S=\{T_{1},T_{2},...T_{s}...\}$. Each table $T_{k}$ has table name $N_{k}$,
column names $c_{j}$ and column data types $t_{j}$. Therefore, $T_{k}=\{N_{k},(c_{k1},t_{k1}),(c_{k2},t_{k2})...\}$. Consider the queries may come from various database domains, we formulate the data into a set of triples $\mathcal{M}=\{(q_{i},s_{i},\mathcal{D}_{i})\}$, with $i$ denoting the index of the query, the output and source database.

### 2.2 Prompt-based and Fine-tuning Settings

Based on how LLMs are used for text-to-SQL generations, the problem settings can be categorized into two scenarios: zero-shot/few-shot prompting and fine-tuning.

#### Zero-shot / Few-shot Prompting.

In zero-shot scenarios, no exemplar is provided while in few-shot a few input–output exemplars are provided to prompt LLMs. Formally, given a pretrained LLM parameterized by $\theta$, the question $q_{i}$, and $k$ exemplars ($k\geq 0$), the objective is maximize the probability of generating the correct SQL $s_{i}$ from the LLM:

$$ $\displaystyle\max_{s_{i}}\mathbb{P}_{LLM_{\theta}}(s_{i}|\sigma(q_{i},\mathcal {M})),\quad|\mathcal{M}|=k$ (1) $$

where ${\Theta}$ and $\sigma(q_{i},\mathcal{M})$ (^1^11$\sigma(q_{i},\mathcal{M})$ technically denotes the information set generated by $q_{i}$ and $\mathcal{M}$.) denotes a representation space of the target question $q_{i}$ by incorporating relevant information from exemplars.

#### Fine-tuning.

The fine-tuing process involves
adapting the pretrained $LLM_{\theta}$ to generate
SQL from the input sequences by tuning the model with text-to-SQL datasets, which contain a collection of serialized inputs $q_{i}$ and corresponding SQL outputs $s_{i}$ pairs. The object of fine-tuning is minimize the empirical loss:

$$ $\displaystyle\min_{\theta}\mathcal{L}(\widehat{s}_{i}(LLM_{\theta}),s_{i}| \sigma(q_{i})),$ (2) $$

where $\mathcal{L}$ is the loss function to measure the difference between the generated SQL and the groundtruth.

Despite the significant advances achieved with few-shot prompting of LLMs, it remains a formidable challenge
for a pretrained LLM to rely solely on its parametric knowledge and prompting to accurately process highly complex SQL queries.

#### Parameter-Efficient Fine-tuning.

Medium to large-sized models with billions of parameters, are prohibitively expensive to fine-tune in order to adapt them to particular tasks or domains. Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of large pretrained models to various downstream applications by only fine-tuning a small number of (extra) model parameters instead of all the model’s parameters. Two mostly commonly used techniques are LoRA (Hu et al., ), which proposes to freeze pretrained model weights and inject trainable layers (rank-decomposition matrices) in each transformer block, and its quantized version QLoRA (Dettmers et al., ). Throughout the benchmark, we use these two strategies consistently to tune the LLMs. See Section 3 and Section 4 for details of tuning benchmark design and experimental results.

## 3 Benchmark Design and Resources

### 3.1 Setup

#### Datasets.

We conduct experiments mainly on the following 2 well recognized public datasets:

- •
Spider (Yu et al., ). Spider is a large-scale cross-domain dataset consisting of 10,181 natural language queries, 5,693 unique complex SQL queries across 200 databases, covering 138 domains. The standard protocol for
this dataset divides it into 8,659 training examples and a holdout of
2,147 test examples across 34 databases. SQL queries are categorized into
four difficulty levels, i.e., easy, medium, hard and extra hard.
- •
BIRD (Li et al., ). It comprises an extensive dataset with 12,751 unique question-SQL pairs, encompassing 95 large databases. SQL queries are categorized into
three difficulty levels, i.e., simple, moderate and challenge. Notably, the SQL queries in
the BIRD dataset tend to be more intricate than those in the Spider dataset.

Moreover, our codebase universally supports tuning a wide range of popular dataset, such as WikiSQL (Zhong et al., ), CoSQL (Yu et al., ), Chase (Guo et al., ) (see section A.1 for the detailed statistics of each dataset.) and due to the page limit, we continually post updated experimental results on the project site(^2^22[https://github.com/eosphoros-ai/DB-GPT-Hub/blob/main/docs/eval_llm_result.md](https://github.com/eosphoros-ai/DB-GPT-Hub/blob/main/docs/eval_llm_result.md)).

#### Query-response Construction.

We construct query-response pairs from the datasets so that LLMs can be tuned with (Gao et al., ; Xue et al., ). Following Gao et al. (), we formulate the pairs using the widely-used Text Representation Prompt (Nan et al., ) (TRP) format for train, development and test split for all the datasets throughout the experiments.

Shown in LABEL:lst:trp, TRP represents both
schema and query in natural language. In addition, it adds instructions at the very beginning of
the prompt to guide LLMs. See LABEL:lst:full_trp and LABEL:lst:full_trp_bird in section A.4 for full examples.

Figure: Listing 1: Query-response Pairs in TRP Format on Spider Dataset.

Figure: Figure 1: An open benchmarking pipeline using DB-GPT-Hub.
Refer to caption: x1.png

#### Metrics.

We use two commonly used metrics, exact-set-match accuracy (EM), execution accuracy (EX) to evaluate the performance of all models.
EM measures the matched SQL keywords between the predicted SQL query and its corresponding
ground truth while EX compares the execution output of the predicted SQL query with that
of the ground truth SQL query on some database instances. EX provides a more precise estimate of the model’s performance since there may be multiple valid SQL queries for a given question. For both metrics, the higher is considered the better. We mainly use EX to evaluate the accuracy of SQLs in the paper. See section A.2 for details.

#### Base LLMs.

We benchmark a range of medium to large-sized LLM variants from 4 prominent LLM families: GLM (Zeng et al., ), Qwen (Bai et al., ), Baichuan (Baichuan, ) and Llama (Touvron et al., ).

- •
ChatGLM3-6B, the up-to-date open version of ChatGLM, an open bilingual language model based on GLM framework.
- •
Qwen-7B/14B/72B-Chat, a series of aligned models of Qwen.
- •
Baichuan2-7B/13B-Chat, the up-to-date collection of aligned models of Baichuan.
- •
LLaMA2-7B/13B/70B-Chat(^3^33Due to the page limitation, we have omitted the suffix “-Chat” from the names of LLMs in the tables throughout the following sections. For instance, “Qwen-7B” should be read as “Qwen-7B-Chat” model.), the up-to-date aligned version of LLaMA.
- •
CodeLLaMA-7B/13B/70B-Instruct, an aligned version of LLaMA-2-13B, tuned with code data.

To ensure a fair comparison, we use the same maximal context length 2048 for all the LLMs. During the evaluation, we leave 512 tokens for response generation. We set the argument temperature as 0 to eliminate the influence of randomness.

#### Tuning Methods.

As the scale of the dataset is notably smaller than that of LLMs, we apply the PEFT strategies –LoRA and QLoRA – to tune the LLMs, respectively. For medium-sized models (7B/13B), we adopt 1 Nvidia A100 Tensor Core GPU to run training and inference. For large-sized models (70B), we adopt 8 A100s.

#### Benchmark Pipeline.

Figure 1 presents the open benchmarking pipeline implemented in DB-GPT-Hub. This pipeline will facilitate future research in this area and help promote reproducible work.

### 3.2 Codebase

To facilitate the innovation of the community, our DB-GPT-Hub contains a well-modularized, easy-to-extend codebase for standardization of implementation, evaluation, and ablation of text-to-SQL methods.

#### Software Architecture.

Figure 1 presents the pipeline and architecture of our codebase. Pipelines are decomposed into following parts:

- •
Dataset Construction. Raw text-to-SQL data is processed into a suitable format (e.g., TRF shown in LABEL:lst:trp ) to tune LLMs. This includes integrating the schema and database description into a prompt as an instruction, along with various question representations to boost performance during training and evaluation. Additionally, we will select different few-shot strategies, such as example selection and organization, to construct the evaluation dataset Gao et al. ().
- •
Training. Our codebase supports the fine-tuning of open-source LLMs with PEFT strategies. We support most of the public architecture with small to large-sized model scales, such as Qwen, Llama, Baichuan, and ChatGLM.
- •
Prediction. Our codebase supports SQL query inference for open-source LLMs with its fine-tuned version and closed-source LLMs as well. We support the few-shot and zero-shot method to generate SQLs for specific scenarios.
- •
Evaluation. Our repository holds different metrics(EX, EM, valid efficiency score(VES)) to evaluate the performance of generated SQL from different perspectives.

#### Implementations.

The codebase is built with the PyTorch framework (Paszke et al., ), upon the open source project DB-GPT (Xue et al., , ). We release the code with Apache License 2.0 and we are committed to actively maintain the repository.

## 4 Experiments

In this section, with the utility of DB-GPT-Hub, we formally evaluate the text-to-SQL process to
determine the performance differences among various LLMs and explore the effect of training paradigms that influence tuning performance of LLMs.

### 4.1 Main Results

**Table 1: Evaluations on Spider: EX of base models vs fine-tuned models on each split of complexity and overall dataset. “L” and “QL” denote “LORA” and “QLoRA” tuing methods, respectively.**
| Model | Easy | Medium | Hard | Extra | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | L/QL | Base | L/QL | Base | L/QL | Base | L/QL | Base | L/QL |
| Llama2-7B | $0.000$ | $0.887/0.847$ | $0.000$ | $0.641/0.623$ | $0.000$ | $0.489/0.466$ | $0.000$ | $0.331/0.361$ | $0.000$ | $0.626/0.608$ |
| Llama2-13B | $0.000$ | $0.907/0.911$ | $0.000$ | $0.729/0.700$ | $0.000$ | $0.552/0.552$ | $0.000$ | $0.343/0.319$ | $0.000$ | $0.680/0.664$ |
| Llama2-70B | $0.411$ | $0.915/-$ | $0.229$ | $0.732/-$ | $0.190$ | $0.560/-$ | $0.072$ | $0.392/-$ | $0.241$ | $0.687/-$ |
| CodeLlama-7B | $0.214$ | $0.923/0.911$ | $0.177$ | $0.756/$0.751 | $0.092$ | $0.586/0.598$ | $0.036$ | $0.349/0.331$ | $0.149$ | $0.702/0.696$ |
| CodeLlama-13B | $0.698$ | $0.940/$0.940 | $0.600$ | $0.789/0.744$ | $0.408$ | $0.684/$0.626 | $0.271$ | $0.404/0.392$ | $0.529$ | $0.746/$0.727 |
| CodeLlama-70B | 0.722 | 0.962$/-$ | $0.625$ | 0.812$/-$ | $0.443$ | 0.716$/-$ | 0.302 | 0.432$/-$ | $0.567$ | 0.771$/-$ |
| Baichuan2-7B | $0.577$ | $0.871/0.891$ | $0.352$ | $0.630/0.637$ | $0.201$ | $0.448/0.489$ | $0.066$ | $0.295/0.331$ | $0.335$ | $0.603/0.624$ |
| Baichuan2-13B | $0.581$ | $0.903/0.895$ | $0.413$ | $0.702/0.675$ | $0.264$ | $0.569/0.580$ | $0.187$ | $0.392/0.343$ | $0.392$ | $0.678/0.659$ |
| Qwen-7B | $0.395$ | $0.855/0.911$ | $0.256$ | $0.688/0.675$ | $0.138$ | $0.575/0.575$ | $0.042$ | $0.331/0.343$ | $0.235$ | $0.652/0.662$ |
| Qwen-14B | 0.871 | $0.895/0.919$ | $0.632$ | $0.702/0.744$ | $0.368$ | $0.552/0.598$ | $0.181$ | $0.367/0.458$ | $0.573$ | $0.663/0.701$ |
| Qwen-72B | $0.831$ | $0.927/-$ | 0.635 | $0.756/-$ | 0.489 | $0.621/-$ | $0.277$ | $0.367/-$ | 0.600 | $0.712/-$ |
| ChatGLM3-6B | $0.000$ | $0.855/0.843$ | $0.000$ | $0.605/0.603$ | $0.000$ | $0.477/0.506$ | $0.000$ | $0.271/0.211$ | $0.000$ | $0.590/0.581$ |

**Table 2: Evaluations on BIRD: EX of base models vs fine-tuned models on each split of complexity and overall dataset. “L” and “QL” denote “LORA” and “QLoRA” tuing methods, respectively.**
| Model | Simple | Moderate | Challenge | Overall |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | L/QL | Base | L/QL | Base | L/QL | Base | L/QL |
| Llama2-7B | $0.000$ | $0.214/0.211$ | $0.000$ | $0.108/0.112$ | $0.000$ | $0.076/0.069$ | $0.000$ | $0.169/0.168$ |
| Llama2-13B | $0.000$ | $0.226/0.217$ | $0.000$ | $0.073/0.086$ | $0.000$ | $0.097/0.069$ | $0.000$ | $0.167/0.163$ |
| Llama2-70B | 0.082 | $0.210/-$ | $0.013$ | $0.138/-$ | 0.014 | $0.126/-$ | $0.055$ | $0.241/-$ |
| CodeLlama-7B | $0.010$ | $0.299/0.076$ | $0.065$ | $0.149/0.146$ | $0.000$ | $0.112/0.128$ | $0.085$ | $0.237/0.223$ |
| CodeLlama-13B | 0.120 | 0.375/0.373 | 0.042 | 0.176/0.179 | 0.042 | 0.141/0.140 | 0.089 | 0.294/0.293 |
| CodeLlama-70B | $0.191$ | 0.423$/-$ | $0.091$ | 0.191$/-$ | 0.063 | 0.159$/-$ | $0.149$ | 0.328$/-$ |
| Baichuan2-7B | $0.051$ | $0.231/0.208$ | $0.024$ | $0.082/0.084$ | $0.000$ | $0.069/0.105$ | $0.038$ | $0.171/0.161$ |
| Baichuan2-13B | $0.048$ | $0.0230/0.182$ | $0.013$ | $0.088/0.067$ | $0.021$ | $0.111/0.069$ | $0.035$ | $0.176/0.136$ |
| Qwen-7B | $0.035$ | $0.235/0.225$ | $0.012$ | $0.073/0.095$ | $0.014$ | $0.083/0.082$ | $0.023$ | $0.171/0.172$ |
| Qwen-14B | $0.188$ | $0.288/0.252$ | $0.049$ | $0.136/0.120$ | $0.028$ | $0.111/0.110$ | $0.131$ | $0.226/0.198$ |
| Qwen-72B | 0.253 | $0.289/-$ | 0.112 | $0.093/-$ | $0.048$ | $0.083/-$ | 0.190 | $0.209/-$ |
| ChatGLM3-6B | $0.000$ | $0.204/0.185$ | $0.000$ | $0.089/0.074$ | $0.000$ | $0.056/0.042$ | $0.000$ | $0.156/0.129$ |

**Table 3: The comparison between LoRA and QLoRA on Spider across different perspectives: EX and EM are the performance metrics; the training time and max GPU memory cost are the resource metrics.**
| Model | EX | EM | Time Cost (hour) | GPU Memory (GB) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LoRA | QLoRA | LoRA | QLora | LoRA | QLoRA | LoRA | QLoRA |
| Llama2-7B | 0.626 | 0.608 | 0.581 | 0.564 | 4.12 | 5.74 | 23.5 | 16.9 |
| Llama2-13B | 0.680 | 0.664 | 0.640 | 0.632 | 7.26 | 8.82 | 34.8 | 29.6 |
| CodeLlama-7B | 0.702 | 0.696 | 0.668 | 0.665 | 4.33 | 6.74 | 23.8 | 16.7 |
| CodeLlama-13B | 0.746 | 0.727 | 0.701 | 0.682 | 7.26 | 8.82 | 34.8 | 29.6 |
| Baichuan2-7B | 0.603 | 0.624 | 0.588 | 0.602 | 3.33 | 7.52 | 20.9 | 11.5 |
| Baichuan2-13B | 0.678 | 0.659 | 0.607 | 0.606 | 8.12 | 15.3 | 34.4 | 17.5 |
| Qwen-7B | 0.652 | 0.662 | 0.610 | 0.621 | 2.57 | 6.45 | 28.9 | 17.1 |
| Qwen-14B | 0.663 | 0.701 | 0.658 | 0.665 | 4.23 | 11.32 | 38.4 | 18.1 |

Table 1 and Table 2 show the evaluation results, measured by EX, on Spider and BIRD datasets, respectively (^4^44For large-sized (70B) models, we found that DeepSpeed optimization is incompatible with QLoRA, so we have left this data blank for the time being.). The results in EM on both datasets can be found in Table 6 and Table 7 in Appendix B.

#### Best Models.

Unsurprisingly, tuned CodeLlama families, whose base models haven been optimized for code generation and infilling, show consistently better performance over other competitors on both datasets. Specifically, we have achieved the following key insights:

- •
As shown in the right-most columns in Table 1 and Table 2, The fine-tuned, small-sized CodeLlama (e.g., CodeLlama-7B-LoRA(^5^55We use the suffix ’-LoRA/QLoRA’ to denote the LoRA/QLoRA PEFT strategies applied to tune LLMs, i.e., ’-LoRA’ means the LLM is tuned with LoRA.)) exhibits comparable, and in some cases even superior, performance to other tuned medium to large-sized open LLMs, such as Qwen-14B/72B-LoRA.
- •
CodeLlama-70B-LoRA is universally optimal.

#### Performance Improvement on Tuning.

Table 1 and Table 2
(also shown in Table 9 in Appendix B) illustrate the improvement of PEFT strategies of LLMs on both datasets, highlighting the LLMs
proficiency to adapt to high-quality text-to-SQL training data. Notably, tuning yields a larger improvement
on Spider compared to BIRD, measured by EX. This suggests that the benefits of tuning become increasingly important in less complex tasks.

Figure: Figure 2: The improvement on tuning with LoRA strategy across subgroups of different complexities.
Refer to caption: x2.png

#### Performance for Different SQL Difficulty Levels.

In Figure 2, using three 7B models for instance, we present the efficacy of tuned LLMs against a spectrum of SQL generation difficulty levels. For all three tuned models, the results highlight that the size of improvement is negatively correlated with the complexities and tuning brings more significant improvement on easy tasks, which reveals the importance of tuning over simpler tasks than difficult ones.

#### LoRA vs QLoRA

We summarize the EX, EM, Time Cost, and GPU memory metrics in Table 3. Firstly, not surprisingly, we see limited differences in generation performance, measured by EX and EM, between models tuned with LoRA and QLoRA. Secondly, consistent with the quantization mechanism, QLoRA takes more time to converge with less GPU memory. For example, compared to Qwen-14B-LoRA, its QLoRA counterpart takes $2\times$ of time with only $~{}50\%$ GPU memory.

To conclude, in circumstances with restricted computational resources, QLoRA is an efficient tuning alternative that can save memory without sacrificing performance.

### 4.2 Analysis I: Fine-tuning vs Few-shot Prompting

In this subsection, we explore the improvements with tuning compared to few-shot prompting.

#### Setup.

We take two model families –Llama2 and Qwen– and conduct our investigations primarily on the Spider dataset. We use
the method DAIL Selection (Gao et al., ), which currently ranks as the second-best open-source model on the Spider leaderboard, to construct the few-shot prompt. It selects exemplars those have good similarity with both queries and responses, better preserving the mapping in between the query-response pairs.

#### Core Insights.

Due to the page limitation, we put the full results in Table 8 in Appendix B. In both zero-shot and few-shot (1/3/5-shot) evaluation scenarios, tuned LLMs demonstrate superior results, highlighting the LLMs proficiency to adapt to high-quality text-to-SQL training data.

#### Effect of Number of Exemplars in Prompting.

In addition to the superior performances of tuned LLMs, Figure 4 reveals that, for strong (large-sized) models, the EX margin of tuned against base model becomes less prominent on few-shot scenarios. For example, the EX of Qwen-72B-LoRA vs Qwen-72B on 3-shot: 68.5 vs 64.8 and on 5-shot: 68.4 vs 65. This is more clearly observed from a different perspective in Figure 3, where the curves for Qwen-13B/72B is flat at low levels.

This fact is possibly because these Qwen-72B already has strong SQL reasoning capabilities, which has barely been discussed in other text-to-SQL benchmarking works.

In all, fine-tuned models exhibit superior SQL reasoning abilities compared to non-tuned models in few-shot generation scenarios; however, the margin of improvement is relatively small for robust models like Qwen-72B.

Figure: (a) 7B
Refer to caption: x3.png

Figure: (a) Llama2
Refer to caption: x6.png

**Table 4: Few-shot Evaluations on Spider: EM and EX of fine-tuned models with the different number of examples in the training corpus.**
| Model | 0-shot | 1-shot | 3-shot | 5-shot |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | EM | EX | EM | EX | EM | EX | EM | EX |
| Qwen-7B | 16.1 | 22.9 | 27.4 | 34.0 | 27.6 | 33.9 | 25.9 | 33.8 |
| Qwen-7B-LoRA (0-shot) | 61.0 | 65.3 | 58.4 | 61.8 | 57.8 | 62.0 | 57.7 | 61.4 |
| Qwen-7B-LoRA (1-shot) | 61.2 | 64.0 | 61.7 | 64.8 | 60.8 | 63.8 | 61.8 | 64.8 |
| Qwen-7B-LoRA (3-shot) | 61.0 | 62.8 | 62.0 | 62.8 | 60.7 | 62.1 | 60.7 | 62.9 |
| Qwen-7B-LoRA (5-shot) | 60.4 | 62.7 | 62.0 | 64.0 | 61.5 | 63.2 | 60.9 | 63.5 |
| Qwen-7B-LoRA (random-shot) | 61.5 | 63.0 | 62.1 | 64.0 | 62.2 | 63.6 | 61.9 | 63.6 |

#### Effect of Model Size.

From Figure 4, we interpret the few-shot performance w.r.t. the model size for four models (two base models and two tuned models) and observe that:

- •
Larger models consistently achieve better results in few-shot scenarios compared to their smaller-sized counterparts.
- •
For a given few-shot scenario, the performance margin of tuning method over prompting method comes closer when the size of LLMs grows. For example, for 1-shot scenario, the performance improvement on EX of Qwen-LoRA over Qwen is 31.0, 24.1 and 3.5 for 7B, 14B and 72B, respectively.

Recall that the exact figure of few-shot evaluations can be found at Table 8 in Appendix B. Overall, tuning methods continue to outperform prompting methods while the performance gap narrows as the size of the LLMs increases.

### 4.3 Analysis II: Fine-tuning with More Exemplars

In this subsection, we explore the possibility of enhancing the performance of LLMs by adding more contextual examples during fine-tuning.

#### Setup.

We use Qwen-7B as the base model and construct additional three few-shot (1/3/5-shot) training sets to fine-tune the model. Specifically, the 1/3/5-shot training sets consist of query-response pairs with an additional 1/3/5 exemplars.
For a given model, we also evaluate its few-shot performances, same as in section 4.2.

#### Core Insights.

Shown
in Table 4, we primarily conclude with two insights:

- •
In a zero-shot evaluation scenario, tuning with additional exemplars does not yield a significant improvement in performance. See the “0-shot” column. This is possible because the training corpus (more examples) mismatches the evaluation setting (no examples).
- •
In 1/3/5-shot evaluation scenarios, adding more contextual examples contributes to the notable improvement over the counterpart tuned with 0-shot training corpus. It means that the performance loss on few-shot evaluation for zero-shot training is caused by the prompt mismatch of training and evaluation dataset.
- •
The random-shot strategy, which refers to randomly adding 0/1/3/5 examples into the training corpus, achieves the highest EM scores. This finding is consistent with that proposed by (Sun et al., ): diverse training corpus benefits the fine-tuning of LLMs.

## 5 Related Work

### 5.1 LLM-empowered Text-to-SQL Methods

Driven by the considerable success of LLMs, the field of LLM-empowered text-to-SQL has captured the interest of a large amount of researchers both in nature language process and database community recently. The models on LLM-based text-to-SQL can be categorized into supervised fine-tuning based and prompting based methods. Popular fine-tuned text-to-sql models are SQL-PaLM (Sun et al., ), PICARD (Scholak et al., ) and RESDSQL (Li et al., ). In contrast to supervised fine-tuned models, prompting-based models do not require additional fine-tuning on task-specific training data. Instead, they solely rely on the zero-shot and few-shot (Rajkumar et al., ; Liu et al., ) capabilities inherent in LLMs. Within the prompting paradigm, the pivotal factor for query representation lies in the design of the prompt (Wei et al., ; Zhou et al., ; Wang et al., ). In particular, DIN-SQL (Pourreza and Rafiei, ) introduces adaptive prompt strategies via task decomposition to effectively address challenges associated with schema linking. DAIL-SQL (Gao et al., ) proposes a refined prompt selection and organization strategy to improve the performance. In DB-GPT-Hub, we offer scripts to support researchers in fine-tuning LLMs in accordance with the methodologies established in SQL-PaLM. In addition, we also integrate the popular prompt techniques used in DAIL-SQL.

### 5.2 Text-to-SQL Benchmarks

A pivotal factor in the progression of text-to-SQL is the establishment of high-quality benchmarks. Early benchmarks focus on single databases, including ATIS (Dahl et al., ), GeoQuery (Zelle and Mooney, ), Academic (Li and Jagadish, ), Advising (Finegan-Dollak et al., ), and more recent additions such as SEDE (Hazoom et al., ) and MIMICSQL (Wang et al., ). These benchmarks and datasets are often adapted from real-life applications, with many containing domain-specific knowledge that may not generalize effectively to unseen SQL domains. Hence, large-scale cross-domain datasets featuring professional SQL queries, such as Squall (Shi et al., ), Spider (Yu et al., ), Spider-Syn (Gan et al., ), WikiSQL (Zhong et al., ), and SparC (Yu et al., ), have been introduced to facilitate comprehensive method analyses.

In retrospect, we realize two concurrent works (Gao et al., ; Zhang et al., ) which
perform systematical benchmarking on text-to-SQL methods.
Important distinctions of their work from ours include: 1. *comprehensiveness of benchmark settings*: we evaluate both ICL and medium to large-sized fine-tuning methods in an end-to-end manner while Gao et al. () focus on ICL methods and Zhang et al. () assess various sub-tasks of the text-to-SQL process; 2. *open source of the codebase*: we released a well-maintained open repository on Github containing all code and data assets, which, to the best of knowledge, is one of the most popular text-to-SQL benchmark repositories (over 1k stars so far), while neither of them has achieved this.

## 6 Conclusion

In this study, we conduct a systematic benchmarking of the various LLMs within the text-to-SQL pipeline.
Our benchmarking provides a
meticulous perspective on the pipeline, equipping the research community with strategies to improve
the semantic understanding of LLMs.

## 7 Limitations

The large computational resources required for LLM training might not be accessible to all researchers and practitioners, which may limit the reproducibility of our findings.

## Appendix A Experimental Details

### A.1 Dataset Details

Spider (Yu et al., ). It consists of 10,181 questions and 5,693
unique complex SQL queries across 200
databases, covering 138 domains, each containing multiple tables. The standard protocol for
this dataset divides it into 8,659 training examples across 146 databases, 1,034 development
examples across 20 databases, and a holdout of
2,147 test examples across 34 databases. The
databases used in each of these sets are nonoverlapping. SQL queries are categorized into
four difficulty levels, based on the number of SQL
keywords used, the presence of nested subqueries,
and the usage of column selections and aggregations.

BIRD (Li et al., ). This dataset represents a pioneering, cross-domain dataset that examines the impact of extensive database contents on text-to-SQL parsing. BIRD contains over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB. It also covers more than 37 professional domains, such as blockchain, hockey, healthcare and education, etc. BIRD also introduces
external knowledge as an additional resource to
assist models in generating accurate SQL queries.
Specifically four sources of external knowledge
were introduced: numeric reasoning knowledge,
domain knowledge, synonym knowledge, and
value illustration. Notably, the SQL queries in
the BIRD dataset tend to be more intricate than those in the Spider dataset.

WikiSQL (Zhong et al., ). This dataset consists of a corpus of 80,654 natural statement expressions and sql annotations of 24,241 tables. Each query in WikiSQL is limited to the same table and does not contain complex operations such as sorting, grouping. The queries in WikiSQL are limited to the same table and do not include complex operations such as sorting, grouping, subqueries, etc.

CoSQL (Yu et al., ). This dataset is a conversational version of the Spider task. CoSQL consists of 30,000 rounds and 10,000 annotated SQL queries from Wizard-of-Oz’s collection of 3k conversations querying 200 complex databases across 138 domains. Each conversation simulates a realistic DB query scenario in which a staff member explores the database as a user and a SQL expert uses SQL to retrieve answers, clarify ambiguous questions, or otherwise inform.

Chase (Guo et al., ). This data is to date the largest Chinese dataset for the cross-database context-dependent Text-to-SQL problem. It consists of 5,459 question sequences (17,940 questions) over 280 databases. Each question in Chase has rich semantic annotations, including its SQL query, contextual dependency, and schema linking.

### A.2 Metrics Details

We clarify the properties of the two metrics in details.

Exact-set match accuracy (EM). EM treats each clause as a set and compares the prediction for each
clause to its corresponding clause in the reference query. A predicted SQL query is considered
correct only if all of its components match the ground truth. EM does not take values into
account.

Execution accuracy (EX). EX compares the execution output of the predicted SQL query
with that of the ground truth SQL query on some database instances. Execution accuracy provides a
more precise estimate of the performance of the method as there may be multiple valid SQL queries for a given question while EM only evaluates the predicted SQL against one of them.

### A.3 Implementation Details

All models are implemented using the PyTorch framework (Paszke et al., ). For parameter scale with 7B and 13B models, we adopt 1 Nvidia A100 Tensor Core GPU to run training. For the parameter scale of 70B model, we adopt 8*A100 to run training and inference.

#### Fine-tuning hyperparameters setting

The hyperparameters of the training are shown in Table 5.

**Table 5: Parameter setting of fine tuning for different model scale**
| Parameter | 7B | 13B | 70B |
| --- | --- | --- | --- |
| GPUs | 1*A100 | 1*A100 | 8*A100 |
| max source length | 2048 | 2048 | 2048 |
| max target length | 512 | 512 | 512 |
| fine-tuning type | lora | lora | lora |
| lora rank | 64 | 64 | 64 |
| lora alpha | 32 | 32 | 32 |
| lr | 0.0002 | 0.0002 | 0.0002 |
| epoch | 8 | 8 | 8 |

### A.4 Few Shot Prompting

Figure: Listing 2: Full Examples of Text Representation Prompt on Spider Dataset.

Figure: Listing 3: Full Examples of Text Representation Prompt on BIRD Dataset.

## Appendix B More Experiment Result

### B.1 EM metrics of Spider Dataset

The EM metric of BIRD dataset are show in Table 6.

**Table 6: Evaluations on Spider: EM of base models vs fine-tuned models on each split of complexity and overall dataset. “L” and “QL” denote “LORA” and “QLoRA” tuing methods, respectively.**
| Model | Easy | Medium | Hard | Extra | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | L/QL | Base | L/QL | Base | L/QL | Base | L/QL | Base | L/QL |
| Llama2-7B | $0.000$ | $0.827/0.810$ | $0.000$ | $0.614/0.574$ | $0.000$ | $0.408/0.443$ | $0.000$ | $0.307/0.295$ | $0.000$ | $0.581/0.564$ |
| Llama2-13B | $0.000$ | $0.867/0.835$ | $0.000$ | $0.670/0.670$ | $0.000$ | $0.483/0.517$ | $0.000$ | $0.386/0.349$ | $0.000$ | $0.640/0.632$ |
| Llama2-70B | $0.327$ | $0.847/-$ | $0.112$ | $0.679/-$ | $0.075$ | $0.454/-$ | $0.018$ | $0.382/-$ | $0.142$ | $0.635/-$ |
| CodeLlama-7B | $0.174$ | $0.883/0.871$ | $0.127$ | $0.736/$0.721 | $0.063$ | $0.523/0.553$ | $0.012$ | $0.309/0.291$ | $0.121$ | $0.643/0.628$ |
| CodeLlama-13B | $0.617$ | $0.910/$0.910 | $0.545$ | $0.727/0.688$ | $0.377$ | $0.624/$0.556 | $0.224$ | $0.365/0.382$ | $0.487$ | $0.706/$0.682 |
| CodeLlama-70B | 0.688 | 0.928$/-$ | 0.582 | 0.723$/-$ | 0.400 | 0.655$/-$ | 0.278 | 0.366$/-$ | 0.527 | 0.713$/-$ |
| Baichuan2-7B | $0.326$ | $0.832/0.815$ | $0.104$ | $0.588/0.621$ | $0.025$ | $0.402/0.454$ | $0.000$ | $0.225/0.286$ | $0.119$ | $0.579/0.602$ |
| Baichuan2-13B | $0.363$ | $0.839/0.827$ | $0.141$ | $0.632/0.650$ | $0.040$ | $0.483/0.460$ | $0.000$ | $0.325/0.313$ | $0.155$ | $0.607/0.606$ |
| Qwen-7B | $0.365$ | $0.802/0.778$ | $0.101$ | $0.643/0.608$ | $0.063$ | $0.517/0.471$ | $0.024$ | $0.331/0.313$ | $0.161$ | $0.610/0.578$ |
| Qwen-14B | 0.758 | $0.867/0.851$ | $0.318$ | $0.713/0.735$ | $0.172$ | $0.529/0.506$ | $0.066$ | $0.398/0.367$ | $0.359$ | $0.623/0.668$ |
| Qwen-72B | $0.754$ | $0.903/-$ | 0.316 | $0.726/-$ | 0.241 | $0.523/-$ | $0.102$ | $0.386/-$ | 0.374 | $0.680/-$ |
| ChatGLM3-6B | $0.000$ | $0.776/0.763$ | $0.000$ | $0.564/0.533$ | $0.000$ | $0.457/0.477$ | $0.000$ | $0.261/0.224$ | $0.000$ | $0.521/0.542$ |

### B.2 EM metric of BIRD Dataset

The EM metric of BIRD dataset are show in Table 7.

**Table 7: Evaluations on BIRD: EM of base models vs fine-tuned models on each split of complexity and overall dataset. “L” and “QL” denote “LORA” and “QLoRA” tuing methods, respectively.**
| Model | Simple | Moderate | Challenge | Overall |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | L/QL | Base | L/QL | Base | L/QL | Base | L/QL |
| Llama2-7B | $0.000$ | $0.068/0.062$ | $0.000$ | $0.015/0.017$ | $0.000$ | $0.000/0.000$ | $0.000$ | $0.046/0.043$ |
| Llama2-13B | $0.000$ | $0.115/0.087$ | $0.000$ | $0.013/0.017$ | $0.000$ | $0.069/0.000$ | $0.000$ | $0.074/0.058$ |
| Llama2-70B | 0.000 | $0.107/-$ | $0.000$ | $0.028/-$ | 0.000 | $0.000/-$ | $0.000$ | $0.072/-$ |
| CodeLlama-7B | $0.000$ | $0.228/0.059$ | $0.000$ | $0.089/0.086$ | $0.000$ | $0.058/0.062$ | $0.000$ | $0.128/0.119$ |
| CodeLlama-13B | 0.088 | $0.293/$0.346 | 0.000 | $0.129/$0.136 | 0.000 | $0.112/$0.124 | 0.029 | $0.256/$0.243 |
| CodeLlama-70B | 0.102 | 0.348$/-$ | 0.059 | 0.124$/-$ | 0.032 | 0.087$/-$ | 0.082 | 0.255$/-$ |
| Baichuan2-7B | $0.000$ | $0.078/0.068$ | $0.000$ | $0.022/0.017$ | $0.000$ | $0.000/0.000$ | $0.000$ | $0.054/0.046$ |
| Baichuan2-13B | $0.010$ | $0.073/0.056$ | $0.000$ | $0.004/0.018$ | $0.000$ | $0.014/0.000$ | $0.035$ | $0.045/0.037$ |
| Qwen-7B | $0.000$ | $0.067/0.082$ | $0.000$ | $0.010/0.015$ | $0.000$ | $0.007/0.013$ | $0.000$ | $0.043/0.055$ |
| Qwen-14B | $0.000$ | $0.089/0.084$ | $0.000$ | $0.028/0.021$ | $0.000$ | $0.014/0.021$ | $0.000$ | $0.064/0.059$ |
| Qwen-72B | 0.154 | $0.243/-$ | 0.023 | $0.048/-$ | $0.012$ | $0.038/-$ | 0.042 | $0.089/-$ |
| ChatGLM3-6B | $0.000$ | $0.124/0.112$ | $0.000$ | $0.045/0.048$ | $0.000$ | $0.026/0.028$ | $0.000$ | $0.068/0.051$ |

### B.3 More Results on Few-Shot Evaluation

The execution accuracy of k-shots prompt on different models with it’s fine-tuned version are shown in Table 8

**Table 8: Few shot evaluations on Spider: base models vs fine-tune models.**
| Model | 0-shot | 1-shot | 3-shot | 5-shot |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | EM | EX | EM | EX | EM | EX | EM | EX |
| Llama2-7B | $3.1$ | $13.0$ | $18.5$ | $25.4$ | $22.1$ | $28.1$ | $22.6$ | $29.3$ |
| Llama2-7B-LoRA | 63.9 | 66.7 | 58.5 | 61.9 | 59.8 | 61.7 | 58.9 | 60.9 |
| Llama2-13B | 2.4 | 20.3 | 13.2 | 30.0 | 15.5 | 32.3 | 16.2 | 32.4 |
| Llama2-13B-LoRA | 62.7 | 67.0 | 62.5 | 66.5 | 60.6 | 66.0 | 61.3 | 66.4 |
| Llama2-70B | 14.2 | 24.1 | 24.8 | 35.7 | 25.4 | 35.2 | 27.7 | 36.6 |
| Llama2-70B-LoRA | 66.3 | 68.7 | 62.8 | 67.1 | 61.6 | 66.6 | 61.5 | 66.6 |
| Qwen-7B | 16.1 | 23.5 | 27.4 | 34.0 | 27.6 | 33.9 | 25.9 | 33.8 |
| Qwen-7B-LoRA | 61.0 | 65.2 | 58.4 | 61.8 | 57.8 | 62.0 | 57.5 | 61.4 |
| Qwen-14B | 32.3 | 52.4 | 40.4 | 55.4 | 43.4 | 56.4 | 44.8 | 57.9 |
| Qwen-14B-LoRA | 67.8 | 69.8 | 64.5 | 66.4 | 64.3 | 65.9 | 64.3 | 66.6 |
| Qwen-72B | 37.4 | 60.0 | 51.5 | 65.4 | 51.3 | 64.8 | 51.3 | 65.0 |
| Qwen-72B-LoRA | 68.0 | 71.2 | 65.1 | 68.9 | 65.5 | 68.5 | 64.2 | 68.4 |

### B.4 LoRA and QLoRA

The performance improvement of LoRA and QLoRA on Spider and BIRD are shown in Table 9

**Table 9: Evaluations on Spider and BIRD: EX improvement on tuning with LoRA / QLoRA over base model.**
| Model | Spider | BIRD |  |  |
| --- | --- | --- | --- | --- |
|  | LoRA | QLoRA | LoRA | QLora |
| Llama2-7B | $\uparrow$0.626 | $\uparrow$0.608 | $\uparrow$0.169 | $\uparrow$0.168 |
| Llama2-13B | $\uparrow$0.680 | $\uparrow$0.664 | $\uparrow$0.167 | $\uparrow$0.163 |
| Llama2-70B | $\uparrow$0.687 | - | $\uparrow$0.186 | $-$ |
| CodeLlama-7B | $\uparrow$0.453 | $\uparrow$0.447 | $\uparrow$0.228 | $\uparrow$0.214 |
| CodeLlama-13B | $\uparrow$0.217 | $\uparrow$0.198 | $\uparrow$0.204 | $\uparrow$0.204 |
| CodeLlama-70B | $\uparrow$0.204 | $-$ | $\uparrow$0.179 | $-$ |
| Baichuan2-7B | $\uparrow$0.268 | $\uparrow$0.289 | $\uparrow$0.133 | $\uparrow$0.123 |
| Baichuan2-13B | $\uparrow$0.286 | $\uparrow$0.267 | $\uparrow$0.141 | $\uparrow$0.101 |
| Qwen-7B | $\uparrow$0.417 | $\uparrow$0.427 | $\uparrow$0.148 | $\uparrow$0.133 |
| Qwen-14B | $\uparrow$0.090 | $\uparrow$0.128 | $\uparrow$0.075 | $\uparrow$0.068 |
| Qwen-72B | $\uparrow$0.112 | $-$ | $\uparrow$0.019 | $-$ |
| ChatGLM3-6B | $\uparrow$0.590 | $\uparrow$0.581 | $\uparrow$0.156 | $\uparrow$0.128 |

## Appendix C Ongoing and Future Work

We are currently exploring several extensions to deal with more complex dialogue and analytics cases
in our system. We are particularly interested in handling

- •
More powerful agents.
Users may want our system not only to perform the analysis but also provide more powerful abilities on text-to-SQL, such as sequential predictions (Jin et al., ; Xue et al., ) based on historical data and predictive decision abilities (Pan et al., ).
- •
Integration of more model training techniques. In addition to pre-training, the community is also interested in continual learning techniques for language models, such as continual pre-training (Jiang et al., ), prompt learning (Wang et al., ) or positional encoding techniques (Zhu et al., ). The integration of these methods will greatly facilitate the research community in these areas.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - I believe the codebase could benefit from additional layers of abstraction to enhance experiment configuration, such as wrappers for adjusting the format of in-context demonstrations or the format of the DB schema used for training and test data instances.
soundness: 2
strengths: - The paper is generally easy-to-follow, and the experiments are clearly described.
- Experiments demonstrating the performance of popular LLMs (including models consisting of up to 72B parameters) across different configurations.
- The provided codebase could be of some value for exploring the performance of fine-tuning-based baselines on the involved benchmark datasets.
summary: The paper proposes a framework for fine-tuning and benchmarking LLMs on popular Text-to-SQL datasets. The paper presents performance scores on the Spider and BIRD datasets, while the codebase supports other similar datasets, such as WikiSQL, CoSQL and Chase. The authors explore the performance of several open-weight LLMs on Spider and BIRD across different configurations (incl. base-model prompting, full-parameters fine-tuning or fine-tuning using LoRA and QLoRA adapters and training/testing with and without in-context demonstrations).
weaknesses: - Nowadays, state-of-the-art performance in NL2SQL is achieved by systems that extend beyond simple fine-tuning of an autoregressive LLM on (question, SQL) pairs. These systems typically incorporate features such as, schema and value linking, SQL validity checks, and multi-turn reflection and refinement. However, the proposed benchmarking framework focuses on exploring the performance of *more baseline* solutions.
- The current approach does not account for unique characteristics or idiosyncrasies present across the various supported datasets. For instance, the BIRD dataset, includes an `evidence` field for each question, and, in many cases this information is necessary, for coming up with the appropriate SQL for each input. However, the paper does not include any considerations for such cases. This may actually explain why the execution accuracy scores provided in Table 2 are lower compared to the ones reported on the official leaderboard page of BIRD (i.e. https://bird-bench.github.io).
- A significant number of systems published in the corresponding leaderboards leverage the DB content for each input DB (Wang et al., 2020; Shen et al., 2024; Sun et al., 2024). However, my understanding is that this is not currently supported by the proposed framework.

Zhili Shen, Pavlos Vougiouklis, Chenxin Diao, Kaustubh Vyas, Yuanyi Ji, and Jeff Z. Pan. 2024. Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruning. Preprint, arXiv:2407.03227.

Ruoxi Sun, Sercan Ö. Arik, Alex Muzio, Lesly Miculicich, Satya Gundabathula, Pengcheng Yin, Hanjun Dai, Hootan Nakhost, Rajarishi Sinha, Zifeng Wang, and Tomas Pfister. 2024. SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended). Preprint, arXiv:2306.00739.

Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online. Association for Computational Linguistics.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: All the weaknesses could be corrected which can improve the overall strength and acceptance of the paper.
soundness: 3
strengths: 1. The paper's focus on creating a framework for evaluating fine-tuned LLMs is very relevant and has high impact potential as current benchmarks only support prompt based methods.
2. Proposed codebase is modular, supported evaluation metrics and fine-tuning methods are satisfactory and enables fair comparison for open sourced models. Additionally, supports closed models which is important.
3. Includes comprehensive analysis of LLMs including small and medium sizes while re-asserting the advantages of fine-tuning for performance critical and challenging tasks such as text to sql. 
4. The experiments and results are well articulated and providing details related to GPU usage and time cost is a strength.
5. The paper is overall well detailed, readable from start to end, includes sufficient technical depth and is a valuable addition to the community.
summary: The paper focuses on creating a benchmarking suite for LLMs in the area of Text to SQL. It introduces DB-GPT-Hub which is proposed to be an open source tool targeting the evaluation of LLMs mainly covering the fine-tuning use cases for text to SQL datasets/tasks. Since traditional fine-tuning is expensive for LLMs, this tool primarily covers PEFT such as LORA and QLORA. Their key contributions include a standard and comprehensive eval framework, modular and extensible codebase and the results of several open source models such as LLAMA, QWEN, BAICHUAN family of models and ChatGLM3 using their framework. The codebase supports many popular datasets such as SPIDER, BIRD, WikiSQL and others.
weaknesses: 1. It is not clear to the readers what are some existing tools that support prompt based evaluation and what are the strengths of the proposed tool compared to existing ones. The discussion on this is too short.
2. The paper's reported results on QLORA vs LORA performance for QWEN models seem unintuitive and probably require more discussion/insights into why the discrepancy compared to other models.
3. No discussion at all on ethical considerations. It is important to talk about how LLMs can hallucinate and generate incorrect answers which might lead to serious concerns for some industrial applications especially in health care, etc.
4. Limitations are not well-thought/rounded. It should include discussions related to limitations/biases of supported datasets, limitations of metrics as some LLMs might require post processing and the supported metrics might not be representative, etc.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Why didn’t the authors use the more recent Qwen2 and Llama3 models? Could these newer models lead to different experimental outcomes due to their enhanced capabilities? Additionally, would a coder-specific model be better suited for the text-to-SQL task?
soundness: 2
strengths: 1. The open-source framework proposed by the paper provides an integrated approach encompassing data construction, model training, and result evaluation. This robust framework is well-suited for research on text-to-SQL fine-tuning methods.
2. The experiments are thorough, examining something from sample sizes in training datasets to comparisons of LORA and QLORA fine-tuning methods and evaluations across different metrics, clearly identifying when fine-tuning proves effective.
3. The writing is clear and detailed, especially in defining the problem and describing the experimental setup.
summary: This paper focuses on generating database queries from natural language queries (text-to-SQL). Most recent research on text-to-SQL has concentrated on improving results by refining LLM prompts and optimizing interaction workflows with LLMs. The paper proposes a convenient tool, DB_GPT_Hub, which integrates data, model, and evaluation processes to form a complete and standardized code framework. However, the paper emphasizes experimental results related to fine-tuning methods more than a detailed introduction of the tool itself. Experiments were conducted using various sizes of Llama2, Qwen, and ChatGLM models, showcasing the effectiveness of fine-tuning methods. Additionally, a series of ablation studies were performed to discuss the impact of different settings during training.
weaknesses: 1. The paper is more of an integration of the entire process for fine-tuning text-to-SQL tasks rather than a research study, making it somewhat lacking in innovation.
2. The paper offers limited detail on the framework’s structure. More granular information would enhance the paper’s completeness.
3. If the experiments included a comparison with results from prompt-based methods in closed-source LLMs, the paper could provide a more comprehensive analysis of the strengths and weaknesses of each approach.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: See Weakness for details, I will stress on most crucial questions here, even if they are fundamental:

1) What is the pain point of previous text-to-SQL benchmarks, such as SPIDER and BIRD also support fine-tuning and there are training set and many works in the leaderboard already.
2) Why do you think such pain problems are important in text-to-SQL benchmark? And how do you resolve them?
3) How do you present your "comprehensiveness" and "extensiveness"? Show a table to compare against other works. Show how to compute "comprehensiveness", "extensibility". Given this paper even didn't include some popular techniques such as COT, TOT, React in prompt methods, DPO in training. Consider them as new techniques that users tried to extend in your workflow. What is the average human costs and efforts to extend them to show your extensibility. How about the previous work. Compare these to show your efficiency. 
4) What does "examplar" mean in task formulation? Giving the first look at it mislead me to consider it as training data since authors use the letter i for all, which is confusing. Please define this clearly.
5) What is your benchmark workflow exactly? Just two lines are not clear. Is anything novel or special? Did authors resolve some issues of previous works which prevents them from popular github projects? If so what is the problem and how do Figure 1 resolve it?
6) What are new conclusions from experiments on your benchmark especially in terms of text-to-SQL aspects? Such as does domains / database schema/ training distribution of knowledge influence LLM performance? Which models in which settings are most robust?, which models are most potential? I encourage authors to read Dr. Spider [3] to learn how to conduct analysis and evaluate models in a more academic level. I highly encourage authors to learn how to bring out more insights from experiments based on your specific tasks. I think the mission of benchmark should bring out crucial research problems and reflect something interesting in this domain.
soundness: 3
strengths: 1. Authors summarize and curate an unified codebase for LLM training and evaluation on many popular benchmarks such as BIRD and SPIDER.
2. Authors conduct experiments across various model sizes and settings including zero-shot and few-shot settings. Also authors implemented lora and Q-lora of partial open-source models on these benchmarks.
summary: This paper investigates capabilities of partial open-source language models across different size on SPIDER and BIRD text-to-SQL benchmarks by fine-tuning approach. Authors summarize and provide an unified training and evaluation pipelines. The experiment summarizes phenomenon and insights for text-to-SQL in PEFT.
weaknesses: In general, the motivations, contributions and paper writing are not good enough for readers to follow. It's hard to follow the paper to see why this work matters in the big scope. The new ideas aren't spelled out clearly, and the writing could use some polish.
## Motivations and Introduction:
1) The logic flow of Introduction in lines 35-37 is unclear. The authors cite seq-to-seq methods as important, but then suddenly discuss decoding techniques without a clear connection. In the paper that authors cite, seq-to-seq modeling involve encoder and decoder usually such as [1] [2]. This jump is confusing, especially since the paper focuses on decoder-only models, not seq-to-seq architectures. If the authors aim to show a progression from seq-to-seq to decoder-only models for SQL generation, they should explain this transition and its rationale. This would help readers better understand the context and relevance of the discussed techniques.

2) The second paragraph is unclear as well. The use of **"While"** suggests a contrast in two clauses, but the relationship between LLMs' improved capabilities and their emergence as a new paradigm seems more **causal** than contrastive. A clearer structure might be: "As LLMs have become more powerful, their application to text-to-SQL tasks has shown promise, as evidenced by [work1, work2, ...]." The claim that LLM-based methods are dominating since 2023 needs stronger support. Including more recent citations (e.g., DIN-SQL, DAIL-SQL) or providing a link to the SPIDER / BIRD leaderboard would be more useful for readers to follow. Additionally, authors mentioned human-like communication, but this seems not related to the focus of papers on single-turn text-to-SQL benchmarks like SPIDER and BIRD. The relevance of communication to these tasks should be clarified or this point reconsidered.

3) In lines 48-49, the phrase **"ambiguity and complexity"** lacks specificity in this context. It's unclear whether the authors are referring to ambiguity in the natural language questions, database schemas, schema linking, SQL generation, or some combination of these elements. Without more precise details, readers may struggle to understand the exact challenges being addressed. The authors should clarify which aspects of the text-to-SQL task they find ambiguous or complex, providing specific examples if possible. And in this paper which aspects they will discuss. 

4) The argument about human effort lacks consistency. They claim in-context learning requires huge expert annotation, but their study uses only 7 examples for it. In contrast, fine-tuning datasets like SPIDER and BIRD contain > 10k examples. This discrepancy undermines the motivation for focusing on fine-tuning approaches. The entire paragraph also presents a confusing narrative. It begins by criticizing in-context learning, then shifts to praising fine-tuning, suggests fine-tuning is under-explored, and abruptly returns to highlighting the advantages of in-context learning. The logic is inconsistent and difficult to follow. The authors need to clarify their position and present a coherent argument about their research focus and its significance. Without a clear progression of ideas, readers may struggle to understand the paper motivation and objectives.

5) The phrase "universally acknowledged" in Line 57 is vague and unsupported. In academic writing, such claims should be backed by concrete evidence, such as citation counts, systematic reviews, or formal user studies. Similarly, the term "relatively smaller" is imprecise without a clear point of comparison. Please define a range such as < 13B or somehow. Without specific definitions or references, these size comparisons remain ambiguous. The authors should provide clear, quantifiable metrics or cite relevant literature to support their claims about model acknowledgment and size categories.

6) The motivation for this work is unclear. The authors try to convey the message that benchmarks for evaluating fine-tuning capabilities of LLMs on text-to-SQL tasks are missing and important. However, this claim is puzzling given the existence of SPIDER and BIRD datasets, which already include training sets which are used for fine-tuning. This paper does **not** introduce new datasets, metrics, or evaluation methodologies, making it difficult to discern its novel contributions, particularly in terms of benchmarking. Without clear advancements in dataset creation or evaluation techniques, the significance of this work in the context of existing benchmarks is questionable. The authors need to articulate more clearly what gap their research fills and how it advances the field beyond the capabilities of current benchmarks.

7) Also, I think one of the biggest problems of this paper is overstatement. All features listed in bullet points cannot be proven in the rest of the paper. Here are in-depth reasons:
   - Standardization: Among the whole paper including appendices, I cannot quite understand how authors standardize pipeline. The author states that their standardized pipelines are **transparent and consistent**. What does **"transparent and consistent"** mean? And why and how are previous works not "transparent and consistent"? Whether this is a better statement by saying what and why previous works are not "transparent and consistent" first, and how do you fix this issue? Also how many contributions of this standardization of your work are? For example, SPIDER and BIRD share similar settings. All them have json as input, sqlite as database formats. Therefore, whether your efficient standardization stems from efforts of these original benchmarks **already**? Or if it's not, please make your contributions clear by stating the difficulties and how you resolve them in the main content since you consider this as one of main contributions here.
   - Comprehensive: I suggest authors could list their aspects of **comprehensiveness** here clearly. According to Line 68-69, it seems the **comprehensiveness** just refers to different sizes of LLMs, one-turn prompt methods. However, this scope is insufficient for a text-to-SQL benchmark, lacking specific elements crucial to the field. so what is the point for this benchmark? Why do authors select text-to-SQL instead of other topics. I have some suggested aspects could be considered as comprehensiveness in text-to-SQL field:
        - Database schema distributions: classify and test LLM training in different database settings. Please note, with the same domain, same questions but **different database schema**, the generated SQLs and difficulties would be very different. 
        - Question distributions: Analyzing performance across various question formats for the same database schema could also reveal some insights. The authors could reference Dr. SPIDER [3] as an example of this approach.
        - Training distributions: Training conditions should be explored since authors focus on fine-tuning mainly in this paper, including domain distributions, training data quantity, and data quality (clean vs. dirty). Compositional generalization data, such as SPIDER-SSP [11], would be also a good reference or example. 
        - Authors just include few-shot or zero-shot one-pass prompting, ignoring more sophisticated but also popular settings such as Tree-of-Thought [4], Agenic workflow [5], also react-like prompting Inter-Coder [6]. Therefore, I think "comprehensiveness" is an over-sold word, potentially leading to reader disappointment after initial excitement.
        - Also authors should compare with other text-to-SQL benchmarks with comprehensive features to show its comprehensiveness by statistic tables. Just linguistic descriptions are not enough.
 - Regarding extensibility, the authors should clarify how new methods can be incorporated into the benchmark. If users need to design prompts and manually integrate multi-step or executed steps into the existing framework. The ease (or difficulty) of extending the benchmark to include these newer techniques should be discussed in detail. If the framework cannot readily support these newer approaches, or if significant human effort is required to integrate them, the authors should acknowledge these limitations. Moreover, if "extensibility" primarily refers to accommodating **new models** rather than **new techniques** specific to text-to-SQL tasks, this should be clearly stated, and the extensibility claims should be moderated accordingly.

8) The introduction is too short and very ambiguous. To improve clarity and impact, the authors could expand this section to include a more thorough discussion of their motivations and general approach, supported by illustrative figures or examples. An explicit discussion of Figure 1 within the introduction is necessary, highlighting the novel aspects of the proposed workflow. The introduction should provide a more comprehensive overview of related contexts, situating the current work within the broader field of text-to-SQL research. 

After reading Introduction several times, I highly suggest that authors should reconsider the **unique** contributions of their benchmark compared to existing works. The introduction requires a more logical structure and careful use of terminology. Statements require to be moderated and supported by experimental or statistical evidence in subsequent sections. Each highlighted point in the introduction must have corresponding proof in the body of the paper. This approach will ensure that the true value and innovations of the benchmark in text-to-SQL research are accurately presented, setting appropriate reader expectations and emphasizing genuine contributions to the field.

## Task Definition:
1) The database D should be clarified as relational database. This paper doesn't consider other formats of databases like GraphDB or MongoDB. Whether database schema already contains db constraints such as PK or FK is a little controversial. In classic database textbooks [7][8], relational database schema **already contains** such constraints. Also, what is the reason authors try to decouple schema and constraints? I didn't see any discussions about distributions or features of such constraints to model performance. What does k mean (no description)? It seems that M just contain one triple of i-th exmaple?
2) In Line 99-100, k appears again, defined as the number of exemplars in few-shot prompting. It is unclear whether this k has the same meaning as the k defined in Section 2.1. If these are different concepts, using distinct variables would improve clarity. If they have the same meaning, authors should define this when it appears at the first time. Formula (1) presents a potential inconsistency. The authors define M as containing q,s,D, but in Formula (1), \sigma(q_i, M) implies \sigma(q_i, (q_i, s_i, D_i)) based on their definition. This raises the question of why q_i is included twice. Additionally, Formula (1) does not provide a clear definition of \theta and does not explicitly state what the **representation space** is, which is not intuitive. The distinction between large \theta and small \theta is also ambiguous since Formula 1 uses small \theta, while the interpretation mentions large \theta. These inconsistencies and absence of clear definitions make the formula and its interpretation difficult to comprehend.
3) In the \paragraph{Fine-Tuning}:
   - In Line 105-106, authors only include q and s as pairs for training. Why is the database not incorporated into the input, given that SQL queries can vary depending on the schema for the same question. The meaning of "serialized input of question" is unclear, because textual input should always be serialized tokens.
   - In Formula 2, what does \hat{s_i} mean? Does it mean gt SQL? if yes, why it models LLM_{theta}, or is \hat{s_i} the gt SQL?
   - Given that PEFT is the only training method tested by the authors, a more detailed definition by formulas would be beneficial. The current general formula does not effectively introduce how authors implement PEFT in such setting. The focus on LoRA, rather than other popular PEFT methods such as prefix-tuning [9] or adapter-tuning [10], seems weird with the claim of the authors of a "comprehensive" study. Including references for these alternative methods would strengthen the paper.

## Benchmark Construction:
The structure of this section should be polished and refined. Authors spent much space in introducing datasets, training, prompting from other works. Their own contributions of benchmarking pipleing by DB-GPT-Hub, just are described in 2 lines with Figure 1?  What are the unique aspects of this pipeline? How does this pipeline achieve comprehensiveness and extensiveness? Why are previous benchmarks insufficient for this task? What specific pain points does DB-GPT-Hub address? The are some suggestions and unambiguities in each part of this Section:
1) CodeBase: What does "suitable format" mean, one-sentence summarization should be clarified. 
2) Metrics: The authors stated and realized the weakness of EM in Line 191-193. Given authors know this metric is not reliable, why did authors include it in experiments? Did authors mitigate the bias? How?
3) Base LLMs: Why didn't authors include more code llms such as Starcoder, Qwen-coder, Deepseek-coder since SQL is also one of popular programming language and they always present extraordinary performance in text-to-SQLs but general LLMs? This may also explain the fine-tuned code-llama had the best performance in this work.
4) Unclear Contributions: After reading this important section, I don't know what is the pain point of concurrent workflow, and what is special method / motivation in the benchmark construction or just unify other works? From my perspective, this work only combines data from other works keeping the same sqlite database, jsons, metadata, it seems not because of authors efforts. Also there are not new dimensions of evaluations, EX, EM, VES has already been utlized in many text-to-SQL works. Therefore, what is the critical contributions of  this work?

## Experiments:
1) The claim that CodeLlama "unsurprisingly" performs better requires more explanation. Why is it unsurprising? Is it due to SQL being a programming language and CodeLlama pre-training on related data? Or does logical reasoning from other coding languages transfer to SQL? To prove this, include more code-focused LLMs like Qwen-Coder, StarCoder, and DeepSeek-Coder for comparison. Dig deeper into what factors drive performance differences and what this means for text-to-SQL tasks. The analysis should go beyond surface-level observations to uncover meaningful insights about model capabilities and text-to-SQL characteristics.
2) Performance on Tuning: I think authors didn't even dig out codes or settings of two datasets. What can the observation "improvement on SPIDER is larger compared to BIRD" prove?  BIRD implements **hash set** comparison while SPIDER does not. Could this difference in evaluation methods affect the observed performance gains? Additionally, does the performance variation stem from domain differences between the datasets? A closer examination of result logs is necessary for meaningful insights. The term "complex" needs clarification. Does it refer to more complicated database schemas, lengthier questions, or a greater diversity of SQL keywords?
3) Fine-tuning vs Few-shot prompting: the decision to conduct experiments solely on SPIDER instead of both datasets raises concerns about potential bias in the conclusions. Why not include both SPIDER and BIRD to ensure more robust and generalizable findings. It is worth noting that the SPIDER leaderboard has not been updated since last year, rendering the current "second-best" open-source model rankings potentially outdated. For instance, DAIL-SQL, which may have been highly ranked on SPIDER, now only ranks **33rd** on the BIRD-SQL leaderboard. Many open-source models have surpassed it. This discrepancy highlights the importance of using up-to-date benchmarks and considering multiple datasets to draw accurate conclusions about model performance. 
4) Fine-Tuning with More Exemplars:
  - First, what does **exemplar** mean in fine-tuning? I think it means in each data from trainning set, authors will give a fixed shots of examples as parts of demonstrations along with instructions, am I right? If so, please define this more clearly in Task Formulation, since it may lead to ambiguity that the number of exemplars refer to number of training data. If not, please tell the difference between exemplars and training data instances in fine-tuning. 
  - In the previous analysis, authors used llama and qwen for one dataset. Now, reduced it further to qwen for one dataset. Why?
  - Why present EM here since authors already state EM is not reliable in Section about Metric? And performance gain distributions between EM and EX are not consistent, why is that?
  - What does random-shot mean here? Please use your formula to define this. Also, what is the random methods implemented exactly? For example, random 3 times, and report the average results or just randomly pick once which is highly biased? If you average several times, report div and mean with error bars please. Does your observation also happens in BIRD set? If not, why is that?

## Related Work:
some important works are missing: CoSQL, BIRD-SQL, Dr. SPIDER [3], SPIDER-SSP [11]. 

Overstatement: we admit it's hard and valuable to achieve a popular work in githubs. However, Github is not an academic place, the number of stars can not be used as the reason to illustrate its academic value since it has different evaluation aspects. A more academic way is to conduct a comprehensive user study to investigate why users like or star your benchmark more? For example, reasons behine higher stars are due to your better advertisement skills or some special components in your codebase that others do not have? The paper sumbission in ICLR is for research, can authors promise all users staring your project are doing research? All of them are active user account? Please give evidence if authors try to state anything in main content of paper.

Along this paper, I do not find anything special compared to SPIDER and BIRD. In BIRD benchmarks, there are works based on training (CodeS, granite, SuperSQL) and ICL works and agenic works. The contributions only are just combining several benchmarks into Githubs. In a short, I highly suggest authors consider an apprioprate way of highlighting their contributions in an academic way by more in-depth analysis, user study, and more taxonomies. Just summarizing and reproducing conlusions from cocurrent work are not qualified for researchers to accept it.

**References:** \
[1] LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations, ACL 2022 \
[2] PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models, EMNLP 2022 \
[3] Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness, ICLR 2022  \
[4] Tree of Thoughts: Deliberate Problem Solving with Large Language Models, NeurIPS 2023 \
[5] MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL \
[6] InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback, NeurIPS 2023 \
[7] Database System Concept, 7th edition \
[8] Fundamentals of Database Systems, 7th edition, Page 160 \
[9] Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021 \
[10] Adapter: Learning multiple visual domains with residual adapters, NeurIPS 2017 \
[11] Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? ACL 2021