CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL

## Abstract

Abstract In tackling the challenges of large language model (LLM) performance for Text-to-SQL tasks, we introduce CHASE-SQL, a new framework that employs innovative strategies, using test-time compute in multi-agent modeling to improve candidate generation and selection. CHASE-SQL leverages LLMs’ intrinsic knowledge to generate diverse and high-quality SQL candidates using different LLM generators with: (1) a divide-and-conquer method that decomposes complex queries into manageable sub-queries in a single LLM call;
(2) chain-of-thought reasoning based on query execution plans, reflecting the steps a database engine takes during execution; and
(3) a unique instance-aware synthetic example generation technique, which offers specific few-shot demonstrations tailored to test questions.
To identify the best candidate, a selection agent is employed to rank the candidates through pairwise comparisons with a fine-tuned binary-candidates selection LLM. This selection approach has been demonstrated to be more robust over alternatives. The proposed generators-selector framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods. Overall, our proposed CHASE-SQL achieves the state-of-the-art execution accuracy of 73.0 % and 73.01% on the test set and development set of the notable BIRD Text-to-SQL dataset benchmark, rendering CHASE-SQL the top submission of the leaderboard (at the time of paper submission).

## 1 Introduction

Text-to-SQL, as a bridge between human language and machine-readable structured query languages, is crucial for many use cases, converting natural language questions into executable SQL commands (Androutsopoulos et al., ; hristidis2003efficient; Li & Jagadish, ; Li et al., ; Yu et al., ). By enabling users to interact with complex database systems without requiring SQL proficiency, Text-to-SQL empowers users to extract valuable insights, perform streamlined data exploration, make informed decisions, generate data-driven reports and mine better features for machine learning
(Wang et al., ; Pourreza & Rafiei, ; Sun et al., ; Chen et al., ; Pourreza et al., ; Pérez-Mercado et al., ; Xie et al., ). Furthermore, Text-to-SQL systems play a pivotal role in automating data analytics with complex reasoning and powering conversational agents, expanding their applications beyond traditional data retrieval (Sun et al., ; Xie et al., ). As data continues to grow exponentially, the ability to query databases efficiently without extensive SQL knowledge becomes increasingly vital for a broad range of applications.

Text-to-SQL can be considered a specialized form of code generation, with the contextual information potentially including the database schema, its metadata and along with the values. In the broader code generation domain, utilizing LLMs to generate a wide range of diverse candidates and select the best one has proven to be effective
(Li et al., ; Ni et al., ; Chen et al., ). However, it is non-obvious what leads to most effective candidate proposal and winner selector mechanisms. A straightforward yet effective approach involves generating candidates using zero-/few-shot or open-ended prompting, followed by selecting the best options utilizing self-consistency (Wang et al., ), which entails clustering candidates based on their execution outputs. This approach has demonstrated promising results in several studies (Maamari et al., ; Talaei et al., ; Lee et al., ; Wang et al., ). However, a single prompt design might not fully unleash the extensive Text-to-SQL knowledge of LLMs, and self-consistency methods might not be always effective.
In fact, as illustrated in Table 1, the most consistent answers would not always be the correct ones, with an upper-bound performance 14% higher than that achieved through self-consistency. This substantial gap highlights the potential for significant improvement by implementing more effective selection methods to identify the best answer from the pool of candidate queries.

**Table 1: Evaluating single-query generation vs. ensemble methods of self-consistency and the upper bound that can be achieved for Text-to-SQL with Gemini 1.5 Pro on the BIRD dev set. EX stands for execution accuracy.**
| Method | EX (%) |
| --- | --- |
| Single query | 63.01 |
| Self-consistency | 68.84 ( + 5.84) |
| Upper-bound | 82.79 ( + 19.78) |

Building on the challenges outlined in the previous section, we propose novel approaches to improve LLM performance for Text-to-SQL by leveraging judiciously-designed test-time computations in an agentic framework. As indicated by the upper bound in Table 1, utilizing LLMs’ intrinsic knowledge offers significant potential for improvement.
We propose methods that generate a diverse set of high-quality candidate responses and apply a selection mechanism to identify the best answer. Achieving both high-quality and diverse candidate responses is critical for the success of scoring-based selection methods. Low diversity limits improvement potential and reduces the difference between self-consistency and scoring-based approaches. While techniques like increasing temperature or reordering prompt contents can boost diversity, they often compromise the quality of the candidates. To address this, we introduce effective candidate generators designed to enhance diversity while maintaining high-quality outputs. Specifically, we propose three distinct candidate generation approaches, each capable of producing high-quality responses. The first is inspired by the divide-and-conquer algorithm, which breaks down complex problems into smaller, manageable parts to handle difficult queries. The second employs a query execution-plan-based chain-of-thought strategy, where the reasoning process mirrors the steps a database engine takes during query execution. Lastly, we introduce a novel online synthetic example generation method, which helps the model better understand the underlying data schema of the test database. These methods, when used independently, can produce highly-accurate SQL outputs. To effectively select the best answer among candidates, we introduce a selection agent, trained with a classification objective, that assigns scores based on pairwise comparisons between candidate queries. With this agent, we construct a comparison matrix for all candidates and select the final response based on the highest cumulative score. By combining these candidate generation methods with the proposed scoring model, we create an ensemble approach that leverages the strengths of each strategy to significantly improve overall performance.

We present comprehensive evaluations on the efficacy of proposed methodologies of CHASE-SQL.
Our innovative candidate generation approaches demonstrate superior performance compared to traditional generic CoT prompts, illustrating their capability in guiding LLMs through the decomposition of complex problems into manageable intermediate steps. Furthermore, the proposed selection agent significantly outperforms conventional consistency-based methods, contributing to the state-of-the-art results. Specifically, CHASE-SQL reaches an execution accuracy of 73.01% and 73.0% on the development set and test set of the challenging BIRD Text-to-SQL dataset which outperforms all of the published and undisclosed methods on this benchmark, by a large margin.

## 2 Related Work

Early Text-to-SQL methods predominantly utilized sequence-to-sequence architectures, encoding user queries and database schemas using models such as Graph Neural Networks (GNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and pre-trained transformer encoders (Hwang et al., ; Cai et al., ; Cao et al., ). On the decoding side, these systems employed either slot-filling or auto-regressive modelling approaches to construct the final SQL queries from the encoded inputs (Choi et al., ; Wang et al., ). Additionally, tabular language models like TaBERT (Yin et al., ), TaPas (Herzig et al., ), and Grappa (Yu et al., ) have been developed to encode both tables and textual data effectively. However, the landscape has evolved with the widespread use of LLMs, which have largely replaced earlier methods with their superior performance (Katsogiannis-Meimarakis & Koutrika, ; Quamar et al., ). Initially, efforts concentrated on optimizing prompt designs for these LLMs (Pourreza & Rafiei, ; Gao et al., ; Dong et al., ). Subsequent advancements have introduced more complex methodologies, including schema linking (Li et al., ; Talaei et al., ; Pourreza & Rafiei, , ), self-correction or self-debugging (Chen et al., ; Wang et al., ; Talaei et al., ), and self-consistency techniques (Lee et al., ; Sun et al., ; Talaei et al., ; Maamari et al., ), further enhancing the performance by proposing complex LLM-based pipelines.

## 3 Methods

### 3.1 Overall Framework

This section outlines the proposed CHASE-SQL framework, which consists of four primary components: 1) Value retrieval, 2) Candidate generator, 3) Query fixer, and 4) Selection agent. As illustrated in Fig. 1. The proposed framework begins by retrieving relevant database values. Subsequently, all contextual information, including retrieved values, database metadata, and schema, is provided to an LLM to generate candidate queries. These candidate queries then undergo a fixing loop, and finally, all candidates are compared in a pairwise way using the trained selection agent to pick the correct answer. The following sections delve into the details of each component.

Figure: Figure 1: Overview of the proposed CHASE-SQL framework for Text-to-SQL, with value retrieval and using a selection agent for improve picking of the answers among the generated candidates along with a fixer to provide feedback for refinement of the outputs.
Refer to caption: extracted/5896753/figures/methodology.png

### 3.2 Value Retrieval

Databases might contain very high number of rows, with often only a few being relevant to a query. Retrieving relevant values is crucial as they can be used in various SQL clauses like ‘WHERE’ and ‘HAVING’. Similar to the approach in (Talaei et al., ), we begin by extracting keywords from the given question using an LLM prompted with few-shot examples. For each keyword, we employ locality-sensitive hashing (LSH) (Datar et al., ) to retrieve the most syntactically-similar words, and re-rank them based on embedding-based similarity and edit distance. This approach is robust to typos in the question and considers keyword semantics during retrieval.

### 3.3 Multi-path Candidate Generation

As shown in Table 1, relying solely on consistency among responses can lead to sub-optimal performance. Therefore, we prioritize diversity in generation of multiple response candidates to increase the likelihood of generating at least one correct answer. Among the diverse responses generated by the candidate generators, we select one as the final response using a selection agent that compares candidates pairwise. To generate diverse responses, we increase the next token sampling temperature, and also shuffle the order of columns and tables in the prompt.

Chain-of-Thought (CoT) prompting (Wei et al., ) has been proposed to enhance LLMs’ reasoning abilities by conditioning their final responses on a step-by-step chain of reasoning. Most CoT prompting approaches rely on few-shot examples in the prompt to guide LLMs on thinking step-by-step, following the format $M={(q\textsubscript{i},r\textsubscript{i},s\textsubscript{i})}$, where qi is the example question, ri is the reasoning path, and si is the ground truth SQL query for qi. We employ two distinct reasoning methods and an online synthetic example generation approach. As shown in Fig. 3(a), different generators can yield different outputs, indicating their effectiveness for specific questions and databases.

##### Divide and Conquer CoT:

Divide-and-conquer perspective brings breaking down complex problems into smaller sub-problems, solving each individually, and then combining the solutions to obtain the final answer. Along these lines, we propose a CoT prompting approach that first decomposes the given question into smaller sub-problems using pseudo-SQL queries. In the ’conquer’ step, the solutions to these sub-problems are aggregated to construct the final answer. Finally, an optimization step is applied to the constructed query to remove redundant clauses and conditions. This approach is particularly powerful handling complex scenarios that involve nested queries, e.g. intricate WHERE or HAVING conditions, and queries requiring advanced mathematical operations. In Appendix Fig. 17, we exemplify a question and its corresponding SQL query that was successfully solved using this generator, a scenario the other methods considered in this paper could not address due to the query’s complex conditions and SQL clauses. For a more detailed view of the divide-and-conquer prompt, please see Appendix Fig. 16. Additionally, Alg. 1 outlines the step-by-step process of this strategy to generate the final SQL output using a single LLM call.

Figure: Algorithm 1 Divide and Conquer Chain-of-Thought (CoT) Strategy for Text-to-SQL.

##### Query Plan CoT:

A query (execution) plan is a sequence of steps that the database engine follows to access or modify the data described by a SQL command. When a SQL query is executed, the database management systems’ query optimizers translate the SQL text into a query plan that the database engine can execute. This plan outlines how tables are accessed, how they are joined, and the specific operations performed on the data (see Appendix Fig. 19 as an example). Inspired by the step-by-step process that database engines use to execute SQL queries, we propose a reasoning strategy to construct the final SQL output. Query plans for any given SQL query can be obtained using the “EXPLAIN" command, which provides a detailed breakdown of execution steps. However, this output is often presented in a format that is difficult to interpret by LLMs (e.g. in SQLite). To address this, we convert the output of “EXPLAIN" command into a human-readable text format that aligns more closely with the pretraining data of LLMs. The human-readable version of query plans consists of three key steps: (1) identifying and locating the relevant tables for the question, (2) performing operations such as counting, filtering, or matching between tables, and (3) delivering the final result by selecting the appropriate columns to return. This reasoning method complements the divide-and-conquer CoT strategy. While the divide-and-conquer approach is better suited for decomposing complex questions, the query plan approach excels when questions require more reasoning over the relationships between different parts of the question and the database schema. It systematically explains which tables to scan, how to match columns, and how to apply filters. Appendix Fig. 20 shows an example of a question that was answered correctly only by this method. Appendix Fig. 18 provides the prompt used for this reasoning strategy.

##### Online Synthetic Example Generation:

Using $M$ demonstrations for few-shot in-context learning has shown promising results on various related tasks (Pourreza & Rafiei, ). Besides helping with specifying the task and illustrate the step-by-step process deriving the output, demonstrations constructed using relevant tables and columns can also help the model understand the underlying data schema. Based on this insight, we propose a synthetic demonstration generation strategy for Text-to-SQL – given the user question $Q_{u}$, the target database $D$, and the selected columns $t_{i}$ (using a column selection approach similar to (Talaei et al., )).

Figure: Algorithm 2 Online Synthetic example generation strategy for Text-to-SQL.

Algorithm 2 outlines the online synthetic example generation approach with two LLM generation steps. The first step focuses on generating illustrative examples with common SQL features described in the guideline $R_{f}$. The SQL features include equality and non-equality predicates, single table and multi-table JOIN, nested JOIN, ORDER BY and LIMIT, GROUP BY and HAVING, various aggregation functions. These are widely applicable SQL clauses and functions – the generated example SQL queries, incorporating these features, follow the BIRD SQL feature distribution (Appendix Fig 23(a)). The second step focuses on generating examples highlighting correct interpretation of the underlying data schema – the model $\theta$ is asked to generate examples using $t_{i}$ and that are similar to the examples outlined in $R_{t}$. Appendix A.10 provides the prompts used for the example generation).

While a relevant example (e.g. showing a nested JOIN query with multiple tables) can be helpful for questions that require complex JOIN queries, it might also mislead the LLM for overuse (e.g. when a simple single table query is sufficient). This and the inherent ambiguity of natural language query $q_{i}$, for which we draw the examples by relevance, make the example selection challenging. Thus, we generate and inject the examples to the prompt online per $q_{i}$. We ask the LLM to generate many input-output pairs for in-context learning. The final set of synthetic examples for $q_{i}$ contains examples generated with both $R_{f}$ and $R_{t}$. This ensures that the example set is diverse both in SQL features/clauses and the choice of relevant tables/columns used. The diversity of the example set is desirable to avoid over-fitting the output to certain patterns (e.g., the model always writes a SQL with JOIN if shown mostly JOIN examples). Mixing various examples for various SQL features and database tables with and without column filtering is observed to result in better generation quality overall (please see Appendix Table 8).

### 3.4 Query Fixer

In some cases, LLMs might generate queries that are syntactically incorrect. These queries are clear candidates for correction, as they fail to provide the correct answers. To address this, we apply an LLM-based query fixer that leverages the self-reflection (Shinn et al., ) method. The fixer reflects on the previously generated query, using feedback such as syntax error details or empty result sets to guide the correction process. We continue this iterative fixing approach up to a specified number of attempts, $\beta$ (set to three in this paper). Appendix Fig. 21 demonstrates the prompt used for this query fixing step.

### 3.5 Selection Agent

With three different methods for generating SQL queries, we can generate a set of candidate queries for any given question. The key challenge in this step is selecting the correct SQL query from this pool of candidates. A naive approach would be to measure consistency among the candidates by executing them, grouping them based on their execution results, and selecting a query from the largest group as the most likely correct answer. However, this would assume that the most consistent answer is always the best one, which is not always the case. Instead, we propose a more refined picking strategy, Algorithm 3, that relies on a selection agent. Given a set of candidates SQL queries $C=\{c_{1},c_{2},...,c_{n}\}$, the final responses are selected by finding the candidate that has the highest score assigned by the selection model. This model $\theta_{p}$ can take $k$ candidates and rank them based on how accurately each of them answers the given question. Concretely, we formulate the selection of the final response as:

$$ $\small{c_{f}}=\arg\max_{c\in C}\left(\sum_{i=1}^{\binom{n}{k}}\theta_{p}(c_{i_ {1}},\ldots,c_{i_{k}}\mid Q_{u},H_{u},D)\right),$ (1) $$

where $Q_{u}$ refers to the user’s question, $H_{u}$ is the provided hint, and $D$ is the target database from which the question is being asked. In Eq. 1, we pass $k$ candidates to the selection model to be ranked, with $k$ being between 1 and $n$. In the extreme case of $k=1$, the model is unable to make comparisons between candidates, which complicates the evaluation process for the model. As $k$ increases, comparing more candidates makes the process more challenging for the model, as it needs to consider different aspects simultaneously. Consequently, we set $k=2$ and train a model with a classification objective to compare only two candidates at a time.

Having a set of high-quality and diverse candidates, the most straightforward solution is to employ off-the-shelf LLMs to make pairwise selections. However, experiments with Gemini-1.5-pro showed that using the LLM without fine-tuning resulted in only 58.01% binary classification accuracy. This is primarily due to the candidates being very similar to one another, requiring a fine-tuned model to learn the nuances and make more accurate decisions. To train the selection agent, we first generate candidate SQL queries on the training set (of Text-to-SQL benchmarks), and group them into clusters based on their execution results. For cases where at least one cluster contains correct queries and others contains incorrect ones, we create training examples in the form of tuples $(Q_{u},C_{i},C_{j},D_{ij},y_{ij})$, where $Q_{u}$ is the user’s question, $C_{i}$ and $C_{j}$ are the two candidate queries being compared, $D_{ij}$ is the database schema used by both candidates, and $y_{ij}\in{0,1}$ is the label indicating whether $C_{i}$ or $C_{j}$ is the correct query. To avoid order bias during training, we randomly shuffle the order of correct and incorrect queries in each pair. Since the number of cases with both correct and incorrect candidates is limited, for instances where no correct candidate exists, we include the ground truth SQL query in the prompt as a hint to guide the model in generating correct candidates.

Figure: Algorithm 3 Picking the final SQL query from a pool of candidates.

In the pseudo-code for Algorithm 3, we begin by initializing a score of zero for each candidate query. Then, for every distinct pair of queries $(c_{i},c_{j})$, we compare both $(c_{i},c_{j})$ and $(c_{j},c_{i})$ to mitigate any order bias, ensuring that both candidates in a pair are fairly evaluated. If both queries produce the same execution result on the database, we mark one as the winner and increment its score, as these results suggest consistency. If the execution results differ, we generate a union of the schema used by both queries and use the binary classifier to determine which query is more likely to be correct. The classifier takes into account the question, the two candidate queries, and the combined schema to make its decision. The winner’s score is then updated accordingly. After all comparisons, the candidate with the highest score is selected as the final query. In the rare case of a tie in the final scores, we break the tie by selecting one of the candidates arbitrarily.

## 4 Experiments

### 4.1 Datasets and Models

We evaluate the performance of the proposed CHASE-SQL framework on two widely-recognized cross-domain datasets: BIRD (Li et al., ) and Spider (Yu et al., ). BIRD contains over 12,751 unique question-SQL pairs from 95 large databases, spanning more than 37 professional domains, with databases designed to resemble real-world scenarios, featuring messy data rows and complex schemas. Spider consists of 10,181 questions and 5,693 unique complex SQL queries across 200 databases, covering 138 domains. The Spider dataset is divided into non-overlapping training, development, and test sets similar to BIRD. For both, we use execution accuracy (EX), the official metric for their respective leaderboard, as the primary evaluation metric to compare methods. Details of the models and their hyperparameters are provided in Appendix section A.2.

### 4.2 BIRD results

We present the end-to-end Text-to-SQL performance of the proposed CHASE-SQL framework using Claude-3.5-sonnet and Gemini 1.5 pro on the BIRD development set, and Gemini 1.5 pro on the BIRD test set. We compare with both published methods (either with an available codebase and/or paper) and undisclosed methods. For a fair comparison with Gemini 1.5 pro, all LLM calls in the Claude-3.5-sonnet setting, except for the selection model, are made using Claude-3.5-sonnet (previously-trained selection model is reused). As shown in Table 3, CHASE-SQL with Gemini 1.5 pro achieves 73.01% accuracy on the BIRD development set and 73.0% on the BIRD holdout test set, outperforming all previous works and setting a new state-of-the-art performance.

**Table 2: Performance Comparison of different Text-to-SQL methods on BIRD benchmark.**
| Method | EX (Dev) | EX (Test) |
| --- | --- | --- |
| Published |  |  |
| CHASE-SQL + Gemini 1.5 (Ours) | 73.01 | 73.0 |
| CHASE-SQL + Claude 3.5 Sonnet (Ours) | 69.53 | – |
| Distillery + GPT-4o |  |  |
| (Maamari et al., ) | 67.21 | 71.83 |
| CHESS |  |  |
| (Talaei et al., ) | 65.00 | 66.69 |
| MCS-SQL + GPT-4 |  |  |
| (Lee et al., ) | 63.36 | 65.45 |
| SuperSQL |  |  |
| (Li et al., ) | 58.5 | 62.66 |
| Undisclosed |  |  |
| Insights AI | 72.16 | 70.26 |
| AskData + GPT-4o | 72.03 | 72.39 |
| OpenSearch-v2 + GPT-4o | 69.3 | 72.28 |
| PURPLE-RED + GPT-4o | 68.12 | 70.21 |
| Arcwise + GPT-4o | 67.99 | 66.21 |
| ExSL + granite-34b-code | 67.47 | 67.76 |

### 4.3 Spider results

We assess the generalizability of the proposed CHASE-SQL by evaluating it in an end-to-end way on the Spider test set without modifying the few-shot samples in the prompts or training a new selection model, i.e. without using and data from the target distribution. This approach allows us to test the performance of CHASE-SQL on different unseen query and database distributions compared to the data from training distributions.
Table 3 demonstrates that CHASE-SQL achieves an execution accuracy of 87.6% on the Spider test set, placing it second among methods that have undergone specific training or prompt optimization for the Spider dataset. This highlights the strong generalizability of CHASE-SQL and its potential for generating high quality Text-to-SQL for unseen samples coming from very different distributions and unique challenges.

### 4.4 Generator and selection performance

**Table 4: Ablation studies on single candidate generation performance of the candidate generators compared with original BIRD prompt + zero-shot CoT with Gemini 1.5 pro on the BIRD dev set. Selector is our final performance applying the selection agent to 21 candidates generated from all generators.**
| Method | Execution Accuracy (%) | $\Delta$(%) |
| --- | --- | --- |
| Baseline | 57.75 | - |
| QP CoT | 63.62 | +5.87 |
| DC CoT | 63.92 | +6.17 |
| OS CoT | 67.09 | +9.34 |
| Baseline w Query Fixer | 61.58 | +3.83 |
| QP CoT w Query Fixer | 65.51 | +7.76 |
| DC CoT w Query Fixer | 65.77 | +8.02 |
| OS CoT w Query Fixer | 68.02 | +10.27 |

##### Generator + Fixer:

To reveal performance of generators, we conducted an ablation study to evaluate the performance of each candidate generation method before and after applying the query fixer. We compare the performance of the proposed generators in producing a single candidate query against the original BIRD prompt (Li et al., ), augmented with zero-shot CoT reasoning (Kojima et al., ), which serves as the baseline for assessing the quality of prompts. The results, shown in Table 4, indicate that the proposed methods significantly improve SQL generation performance, compared to the naive baseline, towards the goal of producing high-quality candidates while maintaining diversity. Among the candidate generators, the online synthetic data generation approach produced an impressive performance of 68.02%, demonstrating its effectiveness in leveraging test-time compute to improve LLM performance by generating high-quality synthetic examples. Furthermore, the query fixer proved crucial, enhancing the quality of the candidate pool and increasing performance by nearly 2% across all candidate generators.

##### Selection:

**Table 5: Evaluating the binary selection accuracy of the different selection models.**
| Selection Model | Binary Acc. (%) |
| --- | --- |
| Claude-3.5-sonnet | 60.21 |
| Gemini-1.5-pro | 63.98 |
| Tuned Gemma 2 9B | 64.28 |
| Tuned Gemini-1.5-flash | 71.01 |

We conducted an analysis on the binary selection accuracy of the selection agent for cases where, in a pairwise comparison, one candidate is correct and the other is incorrect. We exclude cases where both candidates are either correct or incorrect, as the selection would not affect the outcome since both candidates have the same label. We compare the performance of Claude-3.5-sonnet and Gemini-1.5-pro (both out-of-the-box without fine-tuning) with two fine-tuned models: 1) Gemma 2 9B and 2) Gemini-1.5-flash. As shown in Table 5, both fine-tuned models achieve higher accuracy than the untuned counterparts, demonstrating the importance of fine-tuning to teach the model about the specific preferences.

##### Candidate Generation Analysis:

We analyze the performance of each candidate generator method individually. To better understand the performance potential when effectively selecting the correct SQL query from the candidate pool, we generate seven candidate SQL queries from each generator method (21 candidates in total) for all samples in the BIRD development set. We determine this number of candidates based on the observation that increasing the candidate pool beyond 20 did not yield significant improvements, as illustrated in Fig. 2(d). By assuming access to an oracle selection model that always selects the correct SQL query from the seven candidates, we calculate the upper-bound performance achievable for each generator. Conversely, by assuming an adversarial selection model that always selects the wrong SQL query, we determine the lower-bound performance. Fig. 2 illustrates the upper-bound and lower-bound performance for all three methods together with the performance of our selection agent. As shown, the upper-bound performance of the two different CoT methods is generally higher than that of the synthetic example generation method for different number of candidates. However, their lower-bound performance is also lower than the synthetic method. Lower-bound accuracy reflects cases where all candidates are correct, reducing the noise in the selection process since it doesn’t matter which candidate is chosen, so a higher lower-bound is preferred. This is evident in the selection agent’s performance, where a drop in the lower bound leads to diminishing returns from increasing the upper bound, causing the selection agent’s performance to plateau. Additionally, the upper-bound performance of combining all three methods reaches 82.79%, highlighting the significant room for improvement through better candidate picking methods. This demonstrates that the LLM’s parametric knowledge already contains the information needed to solve most questions, highlighting the need for ensemble approaches to effectively extract and utilize this knowledge.

Figure: (a) Upper-bound and lower-bound Accuracy for Divide and Conquer CoT
Refer to caption: extracted/5896753/figures/divide_conquer_cot.png

Additionally, we evaluate the upper-bound performance by combining all candidates from three candidate generation methods across the simple, moderate, and challenging difficulty levels for the BIRD development set. These difficulty categories are assigned by human experts during the creation of the BIRD development set. Fig. 2(d) shows that, as expected, the upper-bound performance increases with the number of candidates across all difficulty levels. However, for the challenging and moderate classes, the improvement plateaus earlier than in the simple class, suggesting that generating more samples does not further improve the upper-bound performance for these two difficulty levels.

Fig. 2 presents a Venn diagram showcasing the performance of three generation methods: Query Plan, Divide and Conquer, and with Synthetic Examples. The numbers within the intersecting regions represent the instances where multiple methods generated at least one correct candidate. This diagram visually highlights the unique contributions of each method, which indicates the necessity of using all three generators. Additionally, in Fig. 3(b), we compare the number of correct queries generated by each SQL generation method that are not correct by the other generators. The divide-and-conquer approach outperforms the others on challenging questions, while the query plan method excels on moderately difficult queries. To further analyze the performance of the generators across different domains and varying numbers of columns and tables, we compare the number of correct queries generated for each database, as shown in Appendix Fig. 4. As illustrated, both CoT methods generally perform similarly across databases, while the online synthetic example generation method significantly increases diversity, resulting in more correct answers overall across different databases.

Figure: (a) Venn diagram illustrating the number of instances for which each method: Query Plan, Synthetic Example, Divide and Conquer, produces at least one correct candidate. The overlap regions represent multiple methods generating correct candidates.
Refer to caption: x1.png

##### Selection Agent Analysis:

We evaluate the query-picking performance by comparing the Text-to-SQL execution accuracy of the selection agent with the self-consistency method (using majority voting) Wang et al. (), an oracle model (upper bound), and an adversarial model (lower bound). To conduct the evaluation, we generate 10 samples from each candidate generation method using two different sampling temperatures: 0.5 and 1.8. The results, shown in Table 6, demonstrate that the selection agent significantly outperforms the self-consistency method with a large margin, roughly 6%. As expected, increasing the sampling temperature raises the upper bound but also lowers the lower bound. This effect is more pronounced for the synthetic data generation method compared to the two CoT methods, mainly because LLMs generate reasoning steps before producing the final SQL query, which helps mitigate the randomness introduced by high-temperature sampling. The performance with self-consistency method generally decreases as temperature increases, since the majority cluster becomes smaller with more random queries. However, the proposed trained selection agent is less affected by temperature scaling and, in two cases, even improved its performance with a more diverse pool of samples.

**Table 6: Performance comparison of different picking methods on the candidates generated by the candidate generators on BIRD development set with two different temperatures. QP refers to query plan COT, DC refers to divide and conquer COT, and OS is the online synthetic example generation method.**
| Picking Method | QP (T=0.5) | QP (T=1.8) | DC (T=0.5) | DC (T=1.8) | OS (T=0.5) | OS (T=1.8) |
| --- | --- | --- | --- | --- | --- | --- |
| Lower Bound | 50.46 | 48.63 | 51.37 | 47.39 | 60.43 | 50.98 |
| Upper Bound | 78.55 | 80.44 | 78.42 | 79.34 | 74.77 | 79.66 |
| Self-consistency | 65.78 | 65.51 | 66.43 | 64.41 | 67.34 | 66.88 |
| Our Selection Agent | 71.7 | 71.73 | 71.31 | 70.53 | 70.4 | 71.38 |

### 4.5 Ablation Studies

In the previous sections, we evaluate the importance of the selection agent and each candidate generation method. Next, we focus on the analysis of the remaining components of CHASE-SQL: LSH for value retrieval, the query fixer, and three reasoning strategies (QP, OS, and DC). Table 7 shows the performance of CHASE-SQL without each of these steps, highlighting their significance in achieving higher-quality performance. The results demonstrate the contribution of each component, where removing LSH, the query fixer, or any of the candidate generators leads to a reduction in execution accuracy, further validating the importance of these components of CHASE-SQL. Moreover, the table compares the performance of our binary selection agent with two other selection methods: self-consistency (Wang et al., ) and a ranker agent. The ranker agent receives all candidates generated by our three candidate generators in a single prompt, compares them, and produce a ranking for each. For the ranker agent, we select the query with the lowest rank as the best answer. The binary selection agent significantly outperforms both the self-consistency and ranker agents, demonstrating the effectiveness of the proposed method.

**Table 7: Ablation studies on the performance of CHASE-SQL after removing the query fixer, LSH for value retrieval, and reasoning strategies, i.e., QP, OS, and DC.**
| Method | Execution Accuracy (%) | $\Delta$(%) |
| --- | --- | --- |
| CHASE-SQL All | 73.01 | - |
| CHASE-SQL w self-consistency | 68.84 | -4.17 |
| CHASE-SQL w ranker agent | 65.51 | -7.5 |
| CHASE-SQL w/o LSH | 70.09 | -2.92 |
| CHASE-SQL w/o Query Fixer | 69.23 | -3.78 |
| CHASE-SQL w/o QP | 72.36 | -0.65 |
| CHASE-SQL w/o OS | 72.16 | -0.85 |
| CHASE-SQL w/o DC | 71.77 | -1.24 |

## 5 Conclusion

We introduce a novel agentic framework, CHASE-SQL, to leverage test-time compute for generating diverse, high-quality SQL queries and accurately selecting the correct one. We propose multiple chain-of-thought prompting methods and an online synthetic example generation technique, along with a query selection mechanism that scores candidates based on pairwise comparisons. Our framework, CHASE-SQL, sets a new state-of-the-art in the notable public Text-to-SQL leaderboard (at the time of the submission), demonstrating the effectiveness of test-time computation for both generating diverse queries and selecting the most accurate response. CHASE-SQL addresses key issues like query diversity and selection optimization, paving the way for further improvements in complex reasoning tasks encountered at real-world Text-to-SQL challenges.

#### Acknowledgments

We would like to thank Per Jacobsson, Raj Sinha, Zeke Miller, Reza Sherkat, James Su, Zhixian Yan, David Culler, and Xiance Si, for their valuable comments and feedbacks on our paper. We would also like to thank the BIRD team for their invaluable assistance with the evaluation of the BIRD test set.

## Appendix A Appendix

### A.1 Related works

As previously discussed, one approach to enhance Text-to-SQL performance is based on the consistency of LLM responses. The self-consistency approach, as proposed by Wang et al. (), involves sampling multiple responses from an LLM and selecting the most consistent answer based on the majority vote. In the Text-to-SQL context, this technique extends to generating multiple SQL queries for a given question, grouping these queries by their execution results, and selecting a query from the largest cluster as the most consistent answer (Gao et al., ; Sun et al., ; Talaei et al., ). However, recent studies have pointed out the limitations of this method in reliably identifying the correct answer. In response, MCS-SQL (Lee et al., ) introduced an approach that utilizes an LLM to rerank the most consistent answers, moving beyond simple majority voting. Despite these advancements, reliance on consistency as a filtering mechanism can inadvertently exclude correct queries that are less frequent among generated candidates, as a critical bottleneck.

### A.2 Models

All experiments are conducted using models from the Gemini and Claude, known for their ability to handle long contextual information (Maamari et al., ), which is crucial for the Text-to-SQL task involving queries from large databases. For candidate generation, online synthetic example generation, query fixing, column filtering, and keyword extraction, we reported the performance with two models of Gemini 1.5 Pro and Claude-3.5-Sonnet. For the query-picking model, we train a Gemini 1.5 Flash model (which has much less latency than the Gemini 1.5 Pro model) on a dataset of 3.8K samples generated by running the candidate generators on the BIRD training dataset. The Gemini 1.5 Flash model is trained for 10 epochs using a LoRA adapter with a rank of 16 using Vertex AI tuning API.

### A.3 Performance Based On Database

In this section we provide a detailed analysis on the performance of each candidate generator method on the BIRD development set databases.

Figure: Figure 4: Number of correct queries by each method across different databases of BIRD development set.
Refer to caption: extracted/5896753/figures/candidate_generators_db_id.png

### A.4 Error Analysis

Figure: Figure 5: Distribution of system performance based on the final answer correctness. The chart shows the proportion of correct final answers, correct queries existing among candidates but not chosen (wrong selection), no correct candidate cases, and cases were the golden SQL query is wrong.
Refer to caption: extracted/5896753/figures/pie_chart.png

Fig. 5 provides a pie chart that breaks down the system’s performance into four categories: correct final answer (72.9%), correct exists among candidates but not chosen (10.4%), wrong generations or no correct candidate (6.7%), and wrong golden query (10.0%). The majority of responses are correct final answers, but a notable portion falls under correct answers not being chosen by the system. This breakdown helps in understanding areas where the system excels and where improvements can be targeted.

Figure: Figure 6: Correctness comparison of the system across different databases in two metrics: (1) percentage where the correct query exists among the candidates, and (2) percentage where the correct query is chosen by the selection agent.
Refer to caption: x2.png

Fig. 6 presents a comparative analysis of system correctness across multiple databases. The x-axis lists various databases or categories such as California Schools, Formula 1, and Superhero, while the y-axis represents the percentage performance. Two key metrics are visualized: the first is the percentage where the correct answer exists among the candidates (shown by one bar per category), and the second is the percentage where the correct answer is chosen by the selection system (depicted by a second bar for each category).

#### A.4.1 Selection Agent Error Analysis

In this section, we examine cases where at least one of the candidate SQL queries generated by the three generators matched the ground truth answer, but the selection agent assigned the highest score to another, incorrect candidate. We categorized these errors into four groups: (1) Vague questions, (2) Wrong picking, (3) Data Integrity Error, and (4) Incorrect gold query. Fig. 7 illustrates the distribution of each category among the sample queries. In the following sections, we will discuss each of these categories in more detail.

Figure: Figure 7: Error analysis on the cases where selection agent failed to pick the correct SQL query which was among the candidates.
Refer to caption: extracted/5896753/figures/picker_error_analysis.png

##### Wrong Picking Errors:

The largest portion of errors occurs when the candidate with the highest score from the selection agent is missing a required column, table, or SQL clause. In our analysis, we could not identify a specific types of patterns in the model’s mistakes as these mistakes includes different types of the errors almost for each instance. Fig. 8 provides an example where the selected SQL query incorrectly uses * to return all columns, instead of just returning the id as specified in the ground truth answer.

Figure: Figure 8: An example of selection agent preferred SQL query which is incorrect.
Refer to caption: x3.png

##### Wrong Golden Query Error:

The second largest portion of errors occurs when the ground truth SQL query is incorrect, and one of the candidate queries generated by our model replicates the same mistake. However, the selection agent ultimately picks another candidate that correctly answers the question. Fig. 9 provides an example of such a case, where the ground truth query includes an extra molecule_ID column in the SELECT clause, which was not specified in the question.

Figure: Figure 9: An example of an error case where the selection agent picked a correct SQL query and the gold query was wrong.
Refer to caption: x4.png

##### Vague Question:

Another significant portion of errors occurs when the question does not specify which column to return or use for filtering, and multiple columns could satisfy the query. In these cases, although one of the candidates was the correct SQL query, the selection model favored another response that could also be considered correct. Fig. 10 illustrates such a case where "Fresno" could refer to either a city or a county, but the question doesn’t specify which one to return. The selection model chose the query that used "city" and did not select the candidate that used "county.

Figure: Figure 10: An example of an error case where the selection model picked a query which could be considered as correct as the question is vague.
Refer to caption: x5.png

##### Data Integrity Error:

Finally, the smallest category of errors involves cases where two or more columns are supposed to have consistent values, but one or more columns contain missing values. For example, Fig. 11 shows a case where the "School" and "School Name" columns were both expected to contain the names of schools, but one of the columns has missing values.

Figure: Figure 11: An example of an error case where the selection agent picked a correct candidate but because of the data inconsistency the execution accuracy was zero for this candidate.
Refer to caption: x6.png

#### A.4.2 Error Analyses

We present the manual error analysis we conducted on one-third of the cases where none of the generated candidate queries were correct. We categorized these errors into five main types: (1) Schema linking errors, (2) Incorrect logic, (3) SQL function errors, (4) JOIN issues, and (5) Ignoring evidence. Fig. 12 illustrates the distribution of these error categories. As shown, the most common errors occur when none of the candidate queries correctly utilized the columns or tables required to answer the question. In the following section, we describe the specific types of errors that fall under each category.

Figure: Figure 12: Error analysis on the cases where all candidate generators failed to produce a single correct answer.
Refer to caption: extracted/5896753/figures/pipeline_error_analysis.png

##### Schema Linking Errors:

The schema linking errors category includes cases where none of the generated candidate SQL queries correctly use the columns required to answer the question. These errors often occur in databases where column names are ambiguous or confusing for the model. Fig. 13 provides an example where the LLM failed to correctly calculate the average to return the correct column that was expected.

Figure: Figure 13: An example of schema linking error category.
Refer to caption: x7.png

##### Wrong Logic Error:

This category, which represents the second largest portion of errors, includes cases where the logic of the generated candidate queries is incorrect. These errors involve missing elements such as the DISTINCT keyword, NOT NULL conditions, missing columns in the SELECT clause, or incorrect or missing conditions in the WHERE or HAVING clauses. An example is shown in Fig. 14, provides an example where the LLM failed to correctly calculate the average total price due to incorrect logic computing the average total price.

Figure: Figure 14: An example of wrong logic error category.
Refer to caption: x8.png

##### SQL Functions Error:

This category, the third-largest source of errors, includes queries where the error results from the incorrect use of, or failure to include, SQL functions such as COUNT(), CAST(), AVG(), ROUND(), and others. Fig. 15 illustrates a case where none of the candidate queries used the ROUND() function as required by the question.

Figure: Figure 15: An example of wrong usage of SQL functions error category.
Refer to caption: x9.png

##### Joins Error:

This category represents a smaller portion of the error cases, where the source of the error is missing one or more tables in the JOIN clauses of the predicted SQL queries.

##### Ignoring Evidence Error:

This category represents the smallest portion of error cases, where the correct column or table is explicitly mentioned in the provided evidence, but the LLM disregards it and uses its own selection instead.

### A.5 Divide and Conquer Prompt

In this section, we provide an example of a divide-and-conquer prompt used in one of the few-shot in-context learning demonstrations to illustrate the decomposition and aggregation steps.

Figure: Figure 16: An example of the divide and conquer CoT method

An example of question/SQL pair where the divide-and-conquer CoT approach was the only generator that solved the problem.

Figure: Figure 17: An example of query that has been solved only with the divide and conquer CoT method.
Refer to caption: x10.png

### A.6 Query Plan Prompt

In this section, we provide an example of a query (execution) plan prompt used in one of the few-shot in-context learning demonstrations to illustrate the steps.

Figure: Figure 18: An example of the query plan CoT method

Fig. 19 provides an example of a query plan generated by using the EXPLAIN command.

Figure: Figure 19: An example of SQLite query plan generated by using the EXPLAIN command.
Refer to caption: extracted/5896753/figures/db_query_plan_example.png

Additionally Fig. 20 provides an example question that was solved by using the query plan-based CoT strategy.

Figure: Figure 20: An example of query that has been solved only with the query plan CoT method.
Refer to caption: x11.png

### A.7 Query Fixing Prompt

In this section, we provide the prompt template for the SQL query fixing step.

Figure: Figure 21: The prompt template used for query fixing

### A.8 Selection Agent Prompt

In this section, we provide the prompt template used for training and query picking at test time by the trained selection agent. Note that the database schema used in this step is the union of the columns and tables by the two candidates instead of using the full-schema of all tables in the database.

Figure: Figure 22: The prompt template used for query fixing

### A.9 Generated Synthetic Examples Analysis

Figure: (a) Distribution (normalized) of synthetic vs. ground truth examples in different characteristics. The examples are generated for the questions and schemas from the BIRD-Bench dev dataset.
Refer to caption: extracted/5896753/figures/synthetic_ex.png

Fig. 23(a) shows the SQL feature distribution of the generated synthetic examples for the BIRD dev dataset, which closely follows the actual SQL features distribution, except CASE statement. We omit CASE statement examples, since showing examples with CASE statement did not help with the generation, unless the ground-truth SQL query actually used it. Fig. 23(b) shows the input question and output SQL query pair examples generated for a question $q_{i}$ and its associated database from the BIRD dev dataset.

**Table 8: Ablation studies on synthetic example generation guidelines, $R_{f}$ with common SQL features and $R_{t}$ with filtered schema. The baseline is the original BIRD prompt Zero-shot CoT with Gemini 1.5 pro on the BIRD dev set. Total of 75 examples are generated for each example set ($R_{f}$, $R_{t}$) and for the mixed ($R_{f}$ + $R_{t}$)**
| Method | Execution Accuracy (%) | $\Delta$(%) |
| --- | --- | --- |
| Baseline (Zero-shot) | 57.75 | - |
| OS w/ $R_{f}$ | 65.45 | +7.7 |
| OS w/ $R_{t}$ | 66.75 | +9.0 |
| OS w/ $R_{f}$ + $R_{t}$ | 67.09 | +9.34 |

Table 8 illustrates the ablation studies done with different guidelines and their generated example sets. Compared to the baseline (no example), the user question and its associated data schema targetted synthetic examples can help; we try to promote the diversity of the examples to avoid over-fitting the output to certain patterns (e.g., the model always writes a SQL with JOIN if shown mostly JOIN examples).

### A.10 Synthetic Example Generation Prompts

In this section we provided the prompt template for the online synthetic example generation step.

Figure: Figure 24: Synthetic example generation prompt used for common SQL features examples generation . TARGET_DATABASE_SCHEMA contains all the tables from the target database.

Figure: Figure 25: Synthetic example generation prompt. This is use TARGET_DATABASE_SCHEMA filtered with column selection result, and the model is asked to generate simple examples similar to the ones taken from the training dataset (separate from the test or dev dataset).

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: + How about experimenting with the above pipeline for an open-source LLM, is it possible?
+ Can you give some examples of selecting rows to match the schema and user query?
+ Why schema union is so effective in evaluating your query pairs.
soundness: 3
strengths: + Proposed a pipeline that provides high efficiency on text2sql data and has proven its efficiency and high results on difficult benchmarks such as Bird and Spider.
+ There is a thorough review and explanation, making it easy to combine and remove components throughout the system.
+ The paper is presented fully and in detail, with thorough explanations of mathematical formulas, complete conclusions and details in the evaluation and testing.
summary: The article discusses the formation of a pipeline for text2sql, which includes the following main parts: combining multiple methods (including DC-CoT, Query Plan-CoT and OS-CoT) to generate a set of candidate sql, with each sql query running through a fixer to fix the query so that it can be run, and finally, comparing each pair of queries and ranking to choose the most suitable query. The pipeline is implemented on close-source LLMs and is carefully promoted and evaluated to come up with a good method when testing multi-agent LLM for the text2sql task. The method reaches an execution accuracy of 73.01% and 73.0% on the development set and test set of the challenging BIRD Text-to SQL dataset which outperforms all of the published and undisclosed methods on this benchmark by a large margin.
Contribution:
+ Experiment multi method (include few-shot, CoT) to generate candidate SQL query from natural language text.
+ Develop evaluator agent to score every candidate and ranking for choose the best candidate.
+ Experiment combination and ablation many method choose to the effective text2sql pipeline.
+ Adapt Multi-agent LLM approach.
weaknesses: + CoT, Few Shot or query fixer methods are quite classic in testing LLM, so they have not mentioned the highlights in generating this query.
+ Beside the method uses available closed source models such as claude, gemini pro mainly to perform prompting and connect agents together, it is not possible to verify whether it is good to fine-tune open source for the above methods. The article would be better if there was more in-depth research on fine-tuning these models, which could be a useful scientific research and could be more easily deployed in practice. Overall, I rate the article very well in terms of testing and conclusions drawn.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. As a reviewer, I highly appreciate the detailed error analysis in this paper. However, I suggest that the authors include some parts of the error analysis in the main content instead of the Appendix, as it is gaining increasing attention from the text-to-SQL community.

2. As shown in Appendix A.3, why is the number of correct samples for some databases zero across various methods? Could this result be combined with the difficulty level of the corresponding questions for further analysis?

3. For the prompt used in the Query Fixer module, there are few-shot examples provided, as shown in Appendix A.7. What type of execution results do the examples represent? How does the performance of this module vary across different types of execution results (errors)? As an assumption, could including a diverse range of few-shot examples that cover as many types of execution errors (e.g., Column Not Found, Data Type Mismatch) enhance the correcting capability of the Query Fixer module?

4. The authors could consider using a different abbreviation instead of "CHASE" since there is a related work in the text-to-SQL community with a similar name [7].

[7] Jiaqi Guo, et al. "Chase: A Large-Scale and Pragmatic Chinese Dataset for Cross-Database Context-Dependent Text-to-SQL" In Proceedings of ACL, 2021.
soundness: 3
strengths: 1. The performance of the proposed CHASE-SQL framework is highly promising. It achieves SOTA accuracy on the well-recognized BIRD benchmark, outperforming both published and undisclosed methods, thereby demonstrating its effectiveness in complex text-to-SQL tasks.

2. The experiment of the paper is sufficient, and the error analysis section is detailed and highly informative. The authors provide comprehensive analyses, including performance across different databases, error analysis, and selection agent errors, offering clear insights into the model's strengths and areas for improvement. Notably, the error analysis section is outstanding and is garnering increasing attention within the text-to-SQL community, as it helps readers understand not only how the framework works but also why it works.

3. The overall framework is novel, and the paper is well-organized and easy to understand. The structure effectively presents the methodology, experiments, and results, making it accessible for readers to comprehend the contributions and significance of the proposed approach.
summary: This paper introduces the CHASE-SQL framework, a novel approach for improving text-to-SQL task with LLMs. The framework proposes multi-path reasoning techniques that decompose complex queries and optimize candidate generation for SQL, which involves three main strategies: a divide-and-conquer approach for breaking down queries, a CoT reasoning based on query execution plans, and an instance-aware synthetic example generation to enhance LLM performance. To select the best SQL candidate, a fine-tuned selection agent ranks generated queries through pairwise comparisons, achieving high accuracy. Extensive experiments have validated the effectiveness of the proposed framework. CHASE-SQL demonstrates SOTA performance on the well-recognized BIRD benchmark, achieving an execution accuracy of 73%, making it the top place on the leaderboard.
weaknesses: 1. The paper does not include the cost of the proposed framework. Prompting proprietary LLMs for SQL generation has become a mainstream approach in text-to-SQL research. When the performance of various methods shows no significant differences, the method with lower API costs is typically preferred. Previous work has focused more on models released by OpenAI (e.g., ChatGPT, GPT-4) [1]. Since the Gemini series is also an outstanding proprietary LLM, this paper presents a good opportunity to introduce the series to the community by comparing its performance and API costs.

2. The model ablation study is lacking in the paper, and the selection of open-source models has not been discussed. The paper would benefit from verifying the proposed framework on a broader range of models, including GPT series models (e.g., ChatGPT, GPT-4) and open-source models (e.g., LLaMA-3.1, Qwen-2.5). Recently, well-designed frameworks for open-source models have achieved promising progress [2][3], demonstrating particular effectiveness for local deployment and real-world applications. Therefore, experiments on open-source models in this paper could further advance this development.

3. The description of the Query Fixer module is relatively brief. The authors could consider adding a detailed algorithm for correcting incorrect SQL in Section 3.4. Additionally, I suggest including a separate limitations section to discuss the potential challenges for application, such as the framework's complexity, generalization and extension capabilities [4], and the potential limitations in handling ambiguous questions [5][6].

[1] BIRD Leaderboard. https://bird-bench.github.io/

[2] Haoyang Li, et al. "CodeS: Towards Building Open-source Language Models for Text-to-SQL" In Proceedings of SIGMOD, 2024.

[3] Mohammadreza Pourreza, et al. "DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models" arXiv preprint, 2024.

[4] Zijin Hong, et al. "Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL" arXiv preprint, 2024.

[5] Yujian Gan, et al. "Towards Robustness of Text-to-SQL Models against Synonym Substitution" In Proceedings of ACL, 2021.

[6] Xiang Deng, et al. "Structure-Grounded Pretraining for Text-to-SQL" In Proceedings of NAACL, 2021.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. How many examples are included in the few-shot prompting (online synthetic example generation)?
2. What is the total number of tokens processed if I use CHASE-SQL to generate the SQL for a single question?
3, What is the end-to-end latency of CHASE-SQL?
soundness: 4
strengths: 1. The paper is well-written and provide a lot of details and insights for readers to learn about. The evaluation is solid with comprehensive comparison, lower/upper bound analysis and ablation studies. The design of all components within CHASE-SQL are well-justified.

2. Although the pipeline of generating a candidate pool and then selecting from it is not new, CHASE-SQL introduces novel strategies for candidate generation and fine-tunes the selector in a unique way, focusing on simple tasks, i.e., pairwise selection, rather than relying on reranking or selecting from a large pool. I particularly appreciate the approach used to construct the few-shot examples. Instead of tailoring examples to specific SQL types, it includes examples for both the full database and the specific database relevant to the question. Moreover, it aims to cover a broad range of SQL features rather than only complex examples, which significantly reduces the risk of the model overfitting to the provided examples.

3. By using examples generated on BIRD and fine-tuning the selector solely on BIRD, CHASE-SQL also achieves competitive performance on the Spider benchmark, demonstrating its generalizability.
summary: The paper introduces CHASE-SQL, a state-of-the-art Text-to-SQL approach that achieved top performance on the BIRD benchmark at the time of submission. It employs three prompting strategies — Divide-and-Conquer, Query Plan, and Few-Shot prompting with synthetic example generation — to first generate a diverse set of SQL candidates. These candidates are then evaluated by a selector, which is fine-tuned on BIRD to select the correct SQL from two SQLs. The selector does pair-wise comparison over the pool of candidates and outputs the one having the highest score.
weaknesses: The main concern is the cost and latency of CHASE-SQL. Assuming each generator produces 7 candidates, this results in 21 LLM calls. The prompt length for each call is also substantial, especially with the few-shot prompting strategy, which includes examples for both the full database and the specific database, aiming to cover a wide range of SQL features. For the selector, CHASE-SQL employs a pairwise comparison strategy, leading to O(n^2) LLM calls, where n is the total number of generated candidates. It would be helpful if the authors reported the total number of tokens processed by CHASE-SQL to generate SQL for a user query and provided the end-to-end latency. Given these factors, I am uncertain whether CHASE-SQL can achieve interactive Text-to-SQL.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Q1: The paper's novelty requires clarification, particularly in light of the concerns raised regarding the originality of its individual algorithms (see weaknesses W1, W2, and W3).

Q2: The technical contributions and key innovations compared to existing studies need to be clearly justified (refer to weaknesses W4 through W6).

Q3: The experimental design needs to be improved to more effectively verify the authors' claims (see weakness W8).
soundness: 3
strengths: The main strength of the paper is its robust framework, which effectively integrates multiple LLM strategies to optimize SQL query generation, leading to state-of-the-art results. This framework not only demonstrates the versatility of techniques such as divide-and-conquer and self-debugging but also highlights the potential for real-world applications in natural language processing. It achieves superior performance on well-established benchmarks like BIRD and Spider.
summary: This paper introduces Chase-SQL, which combines SQL queries generated by various LLM strategies to enhance query quality for natural language questions. It utilizes diverse approaches, including divide-and-conquer, few-shot learning with demonstration synthesis, and self-debugging. Leveraging Gemini Pro 1.5, Chase-SQL sets a new state-of-the-art performance on two prominent benchmarks, BIRD and Spider.
weaknesses: The main weaknesses of the paper stem from the limited novelty of the framework and its individual techniques. A recent comprehensive survey on NL2SQL, available at https://arxiv.org/pdf/2408.05109, addresses many of the technical innovations discussed here, encompassing a wide range of recent studies that leverage LLMs. This suggests that the contributions may not be as groundbreaking as implied, as they largely reiterate established concepts in the field. 

First, let’s discuss the individual algorithms.

W1. Divide-and-Conquer: This concept has been widely utilized in various NL2SQL studies, including DTS-SQL, TKK, and DEA-SQL, as illustrated in Figure 6 of the NL2SQL survey under the "Decomposition" branch. The papers of all the referred methods can be found in the survey. In order to highlight the technical contributions, it would be helpful to clarify how the proposed divide-and-conquer approach differs from or improves upon these existing methods. 

W2. Chain-of-Thought: The Chain-of-Thought (CoT) approach has also been extensively applied in NL2SQL research, with examples such as CHESS, ACT-SQL, and COE-SQL, which can be found in Figure 6 of the NL2SQL survey under the "Chain-of-Thought" branch. 

W3. Instance-Aware Synthetic Example Generation: The authors assert that they introduce a "unique instance-aware synthetic example generation." However, the use of few-shot examples has already been explored in DIN-SQL and CodeS. Additionally, rather than synthesizing examples, it is often more straightforward to select existing examples from the training data. This raises the question: what advantages does synthesis offer over selection? Please provide empirical evidence or theoretical justification for why the proposed synthetic example generation approach outperforms or differs from simply selecting examples from training data.

- DIN-SQL: https://arxiv.org/abs/2304.11015
- CodeS: https://arxiv.org/abs/2402.16347

Next, let’s explore weaknesses from other aspects.

W4. Method Ensemble: The paper claims that ensembling multiple generated SQL paths is beneficial. However, it raises the question of why not include even more methods by incorporating readily available off-the-shelf solutions.

W5. Method Ensemble vs. Module Combination: In the reference titled "The Dawn of Natural Language to SQL: Are We Fully Ready?", a method for NL2SQL automated architecture search is discussed, which explores a predefined design space of NL2SQL solutions. This prompts a consideration of the pros and cons of coarse-grained method ensembles compared to fine-grained module ensembles.

W6. Self-Consistency and Self-Reflection Methods: The authors propose self-consistency and self-reflection methods, yet both approaches have been extensively studied in the context of NL2SQL. Similarly, value retrieval and candidate generation techniques have been addressed in previous research, as noted in Table I of the aforementioned survey.

W7. Algorithm 3 Execution Comparison: In Algorithm 3, the need to "compare both (ci, cj) and (cj, ci)" raises questions. Why is it necessary to evaluate execution results in both directions? Please provide a specific example or explanation of how comparing in both directions impacts the results or addresses potential biases in the selection process.

W8. Experiments: The experimental results lack clarity regarding whether they stem from the capabilities of Gemini or the proposed framework. (1) If the benefits arise from the framework, the experiments should involve substituting Gemini with various LLMs to assess whether the framework enhances performance across different models. (2) The impact of ensembling different methods remains uncertain. (3) It is unclear how the proposed techniques individually compare to existing methods in areas such as value retrieval, candidate generation, and query fixing. Hence, the following experiments are important to enhance the experimental section:

- Conduct ablation studies comparing their framework with different LLMs.
- Provide a detailed analysis of the impact of ensembling different methods.
- Include comparisons of individual components (value retrieval, candidate generation, query fixing) against existing state-of-the-art methods for each task.

Meta-review: This paper proposes the CHASE-SQL framework, which improves text-to-sql tasks using LLMs. Overall the reviews are quite positive as CHASE-SQL is novel, efficient, and has good results on benchmarks like Bird and Spider. The paper is also well written and provides plenty of detail and insights. I therefore recommend accepting the paper.

Decision: Accept (Poster)