CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL
Authors: Mohammadreza Pourreza
Hailong Li
Ruoxi Sun
Yeounoh Chung
Shayan Talaei
Gaurav Tarlok Kakkar
Yu Gan
Amin Saberi
Fatma Ozcan
Sercan O Arik
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-SQL
LLM
Databases
Primary area: foundation or frontier models, including LLMs
TL;DR: CHASE-SQL is a new framework that improves LLM performance by increasing test-time compute. It achieves state-of-the-art execution accuracy on the BIRD dataset.

Abstract: We present CHASE-SQL, a novel framework addressing large language model (LLM) performance challenges for Text-to-SQL tasks by leveraging multi-agent modeling and test-time compute for improved candidate generation and selection. CHASE-SQL uses LLMs to generate diverse SQL candidates with: (1) a divide-and-conquer approach to break down complex queries, (2) chain-of-thought reasoning based on query execution plans, and (3) instance-aware synthetic example generation for tailored few-shot demonstrations. A selection agent ranks candidates via pairwise comparisons using a fine-tuned binary selection LLM, offering robust performance. This framework improves SQL query quality and diversity, achieving state-of-the-art execution accuracy of 73.0% on the BIRD Text-to-SQL benchmark test set, topping the leaderboard at the time of submission.

Published as a conference paper at ICLR 2025
CHASE-SQL: MULTI-PATH REASONING AND PREF- ERENCE OPTIMIZED CANDIDATE SELECTION IN TEXT- TO-SQL
Mohammadreza Pourreza' Hailong Li Ruoxi Sun', Yeounoh Chung', Shaaan Talaei' Gaurav Tarlok Kakkar Yu Gan', Amin Saberi', Fatma Özean', Sercan Ö. Arik! 'Google Cloud, Sunnyvale, CA, USA
Stanforr University, Stanford, CA, USA
(pourreza, hailongli, ruoxis, yeounch) y @google. com {gkakkar, gany, fozcan, soarik)egoope com {stalaei, baberi)@stanforr
"Equal contribution
ABSTRACT
We present CHASE-SQL, a novel framework addressing large language model (LLM) performance challenges for Text-to-SQL tasks by leveraging multi-agent modeling and test-time compute for improved candidate generation and selection. CHASE-SQL uses LLMs generate diverse SQL candidates with: (1) divide- and-conquer approach to break down complex queries, (2) thain-of-thouuht rea- soning based on query execution plans, and (3) instance-aware synthetic example generation for tailored few-shot demonstrations. A selection agent ranks candi- dates via pairwise comparisons using fine-tuned binary selection LLM, offering robust performance. This framework improves SQL query quality and diversity, achieving state-of-the-an execution accuracy of 73.0% on the BIRD Text-to-SQL benchmark test set, topping the leaderboard at the time of submission.
INTRODUCTION
Text-to-SQL, as a bridge between human language and machine-readable structured query lan- guages, is crucial for many use cases, converting natural language questions into executable SQL commands (Androutsopoulos et al., 1995; HHistidis et al.) 20033
Jagadish| 2014 Li et al.) 2024c| Yu et 2018). By enabling users to interact with complex database systems without requir- ing SQL proficiency, Texi-to-SQL empowers users to extract valuable insights, perform streamlined data exploration, make informed decisions, generate data-driven reports and mine better features for machine learning (Wang et al. 2019 Pourreza & Rafici 22024a] Sun et al.) 20023] Chen et al.]
2023 Pourreza et al. 2024 PerezaMercado et al. 2023] Xie al. (023). Furthermore, Text-to-SQI. systems play pivotal role in automating data analytics with complex reasoning and powering con- versational agents, expanding their applications beyond traditional data retrieval (Sun et al.) [0033]
Xie et al. 2023). As data continues to grow exponentially, the ability to query databases efficiently without extensive SQL knowledge becomes increasingly vital for a broad range of applications.
Text-to-SQL can be considered a specialized form of code generation, with the contextual informa- tion potentially including the database schema, its metadata and along with the values. In the broader code generation domain, utilizing LLMs to generate a wide range of diverse candidates and select the best one has proven to effective (Li al.) 2022, Ni al. 2023 Chen et al., 2021). How- ever, it is non-obvious what leads to most effective candidate proposal and winner selector mecha- nisms. A straightforward yet effective approach involves generating candidates using zero-/few-shot or open-ended prompting, followed by selecting the best options utilizing self-consistency (Wang et al. 2022), which entails clustering candidates based on their execution outputs. This approach has demonstrated promising results in several studies (Maamari et al. 2024 Talaei et al. 2024 Lee et al. 2024 Wang et al. 2023). However, a single prompt design might not fully unleash the exten- sive Text-to-SQL. knowledee of L.LMs, and self-consistency methods might not be always effective.

---

Published as a conference paper at ICLR 2025
In fact, as illustrated in Table I the most consistent answers would not always be the correct ones, with an upper-bound bound performance 14% higher than that achieved through self-connistency. This substantial gap highlights the potential for significant improvement by implementing more effective selection methods to identify the best answer from the pool of candidate queries.
Building the challenges outlined in the previous sec- tion, we propose novel approaches to improve LLM performance for Text-to-QLL by leveraging judiciously- designed test-time computations in an agentic framework. As indicated by the upper bound in Table I utilizing LLMs' intrinsic knowledge offers significant potential for improvement. We propose methods that generate diverse set of high-quality candidate responses and apply selec- tion mechanism to identify the best answer. Achieving both high-quality and diverse candidate responses is crit- ical for the success of scoring-based selection methods. Low diversity limits improvement potential and reduces the difference between censissnny and scoring- based approaches. While techniques like increasing temperature or reordering prompt contents can boost diversity, they often compromise the quality of the candidates. To address this, we introduce ef- fective candidate generators designed enhance diversity while maintaining high-quality outputs. Specifically, we propose three distinct candidate generation approaches, each capable of produc- ing high-quality responses. The first is inspired by the wwvide-and-conquee algorithm, which breaks down complex problems into smaller, manageable parts to handle difficult queries. The second employs a query execution- -aane--laa-baaaeae baadd chain-of-thought strategy, where the reasoning process mir- rors the steps database engine takes during query execution. Lastly, we introduce novel online synthetic example generation method, which helps the model better understand the underlying data schema of the test database. These methods, when used independently, can produce aighly-accurate SQL outputs. To effectively select the best answer among candidates, we introduce a selection agent, trained with classification objective, that assigns scores based on pairwise comparisons between candidate queries. With this agent, we construct a comparison matrix for all candidates and select the final response based on the highest cumulative score. By combining these candidate genera- tion methods with the proposed scoring model, we create an ensemble approach that leverages the strengths of each strategy to significantly improve overall performance.
Table Evaluating singlc-query gen- cration vs. ensemble methods of self- consistency and the upper bound that can be achieved for Text-to-SLL with Gemini 1.5 Pro on BIRD dev set, EX stands for exccution accuracy.
Method
EX (%)
Single query
63.01
Self-consistency
68.84 (+ 5.84)
Upper bound
82.79 19.78)
We present comprehensive evaluations on the efficacy of proposed methodologies of CHASE-SQL. Our innovative candidate generation approaches demonstrate superior performance compared to tra- ditional generic CoT prompts, illustrating their capability in guiding LLMs through the decomposi- tion of complex problems into manageable intermediate steps. Furthermore, the proposed selection agent significantly outperforms conventional consistency- based methods, contributing to the state- of-the-art results. Specifically, CHASE-SQL reaches an execution accuracy 73.01 % and 73.0 % on the development set and test set of the challenging BIRD Text-to-SQL dataset which outperforms all of the published and undisclosed methods on this benchmark, by a large margin. Moreover, by leveraging entirely open-source tor and fine-tuned Qwen-2.5 2.5-coder model (Team) 2024) as the selector- our method achieved a state-of-the oftt-of-the-art performance of 70.33 the BIRD development set with open-source models.
Large Model (Al 2024) the candidate genera-
2 METHODS
2.1 OVERALL FRAMEWORK
This section outlines the proposed CHASE-SQL framework, which consists of four primary com- ponents: 1) Value retrieval, 2) Candidate generator, 3) Query fixer, and 4) Selection agent. As illustrated in Fig. [ The proposed framework begins by retrieving relevant database values. Sub- sequently, all contextual information, including retrieved values, database metadata, and schema, is provided to an LLM to generate candidate queries. These candidate queries then undergo fixing loop, and finally, all candidates are compared in a pairwise way using the trained selection agent to pick the correct answer. The following sections delve into the details of each component.
2

---

Published as a conference paper at ICLR 2025
USEE
asrolment for stadente In grade11
mit sohaslo
(6 003 NON
Medi Couct (e/12) TRrroteme
DARCT Percent (S) Plggite Free (6.19)
FO
(1-4213
WITH Lasact Peroent sigi (io1) Foom
(X-0a) FROH Brpin
BELECTION ACENT
Figure I: Overview the proposed CHASE-SQL framework for Texl-to-SQL, with value retrieval and using a selection agent for improve picking of the answers among the generated candidates along with a fixer to provide feedback for refinement of the outputs.
2.2 VALUE RETRIEVAL
Databases might contain very high number of rows, with often only a few being relevant to a query. Retrieving relevant values is crucial as they can be used in various SQL clauses like 'WHERE' and 'HAVING'. Similar to the approach in (Talaei et al. (2024), we begin by extracting keywords from the given question using an LLM prompted with few-shot examples. For each keyword, we employ locality-sensitive hashing (LSH) (Datar et al.) (2004) to retrieve the most
words, and re-rank them based on embedding- based similarity and edit distance. This approach is robust to typos in the question and considers keyword semantics during retrieval.
2.3
MULTII-ATH CANDIDATE GENERATION
As shown in Table 1 relying solely on consistency among responses can lead sub-optimal perfor- mance. Therefore, we prioritize diversity in generation of multiple response candidates to increase the likelihood of generating at least correct answer. Among the diverse responses generated by the candidate generators, we select one as the final response using a selection agent that compares candidates pairwise. To generate diverse responses, we increase the next token sampling tempera- ture, and also shuffle the order of columns and tables in the prompt.
Chain-of-Te Thoouhh (CoT) prompting (Wei al. 2022) has been proposed to enhance LLMs' reason- ing abilities by conditioning their final responses on a step-by-step chain of reasoning, Most CoT prompting approaches rely on few-shot examples by-step, following the format M = (g, ri, 8,), where qi is the example question, r; is the reasoning path, and s; is the ground truth SQL query for qi We employ two distinct reasoning methods and an online synthetic example generation approach. As shown in Fig, 3a different generators can yield different outputs, indicating their effectiveness for specific questions and databases.
the prompt to guide L.LMs on thinking step-
Divide and Conquer CoT: Divide-ands svride-and-comee and-conquer perspective brings breaking down complex prob- lems into smaller sub-problems, solving each individually, and then combining the solutions to obtain the final answer. Along these lines, we propose a CoT prompting approach that first decom- poses the given question into smaller sub sub-problems using pseudo-SQL. queries. In the 'conquer' step, the solutions to these sub-problems are aggregated to construct the final answer. Finally, an op- timization step is applied to the constructed query to remove redundant clauses and conditions. This approach is particularly powerful handling complex scenarios that involve nested queries, e.g. intri- cate WHERE HAVING conditions, and queries requiring advanced mathematical operations. In Appendix Fig. 18] we exemplify a question and its corresponding SQL query that was successfully solved using this generator, scenario the other methods considered in this paper could not address due to the query's complex conditions and SQL clauses. For a more detailed view of the divide- and-conquer prompt, please see Appendix Fig. 17 Additionally, Alg. [ outlines the step-by-step process of this strategy to generate the final SQL output using single LLM call.
A query (execution) plan a sequence of steps that the database engine fol- lows to access modify the data described by SQL command. When SQL query is executed, the database management systems' query optimizers translate the SQL text into a query plan that the database engine can execute. This plan outlines how tables accessed, how they are joined, and
Query Plan CoT:
3

---

Published as a conference paper at ICLR 2025
Algorithm Divide and Conquer Chain-of- -aii--offf-oohhttth (CoT) Strategy for Text-to-SQL.
Require;
Set of human-annotated few-shot examples M. user question Qu. target database associated with the question, and large language model (LLM) 6.
Divide:
S, + 8(M, D. Q.) Il Decompose the original question Qa into a set of sub-questions S4
2: Seat +- D.l Initialize an empty set Sagt to store partial SQL queries for each sub question
Conquer:
3: for each sub-questtin in S. do
4: II Generate a partial SQL query each sub-question q:
S.gt + Sagl U (0(M, D, Qu. Yo sght, t.... sql;-1) -13}
6: end for
Assemble:
7: Sy +- B(M, D. Q,Qu.. Sug)) it Assemble the final SQI. Sy from all sub queries in Sagl return Sy
the specific operations performed on the data (see Appendix Fig. 20 as an example). Inspired by the step-by-step process that database engines use to execute SQL queries, we propose reasoning strategy to construct the final SQL output. Query plans for any given SQL query can be obtained using the "EXPLAIN" command, which provides a detailed breakdown of execution steps. How- ever, this output is often presented in a format that is difficult to interpret by LLMs (e.g. in SQLite). To address this, we convert the output of "EXPLAIN" command into a juman-readale that aligns more closely with the pretraining data of LLMs. The human-readabl version of query plans consists of three key steps: (1) identifying and locating the relevant tables for the question, (2) performing operations such as counting, filtering, or matching between tables, and (3) delivering the final result by selecting the appropriate columns to return. This reasoning method complements the divide-and-congue decomposing complex questions, the query plan approach excels when questions require more rea- soning over the relationships between different parts of the question and the database schema. It systematically explains which tables to scan, how to match columns, and how to apply filters. Ap- pendix Fig. 21 shows example of question that was answered correctly only by this method. Appendix Fig. 19 provides the prompt used for this reasoning strategy.
text format
CoT strategy. While the divide-and-conuue approach is better suited for
demonstrations for few- shot in-context learning has shown promising results on various related tasks (Pourreza & Rañeil 2024a). Besides using few select demonstrations helping with specifying the task and illustrate the step-by-step process deriving the output, question and SQL example pairs are also used for few-shot in-context learning for text-to-SQL (Liu et al. 2022 Nan et al., 2023). While prior works focused on selecting few handful of relevant examples from existing example pools (e.g., training dataset), we synthesize many example pairs using different schema elements and SQL features per incoming question. Unlike prior few-shot in-context learning approaches, we generate many more than just a few (3-5) examples (Pourreza & Rafiei, 2024a] et al.] (022b)) as we observe that many-shot learning consistently outperforms few-shot learning (Agarwal et al. 2024).
Online Synthetic Example Generation: Using a set of
Algorithm 2 Online Synthetic example generation strategy for Text-to-SQL.
Require
User question Qu, additional user hint H., target datahuse 1) and filtered relevant table columns associated with the question. LLM 6, guidelines R2 for generating examples by SQL features, guide- lines R; for generating examples with filtered schema. and the numbers of examples to generate ny. respectively
1: P - Bfl (19,.5) 9: E S1") where y is input question. is outpu SQt. for the i-sh example
2: P (O),,, Generate examples with entire database by common SQL features
Rr),
P 1-+77 (#(t, Re, Generate examples with filtered colamns to highlight correct schena usage
return P
Algorithm 2 outlines the online synthetic example generation approach with two LLM generation steps. The first step focuses on generating illustrative examples with common SQL features de- scribed in the guideline Ry. The SQL features include equality and non-equality predicates, single table and multi-table JOIN, nested JOIN, ORDER BY and LIMIT, GROUP BY and HAVING, var- ious aggregation functions. These are widely applicable SQL clauses and functions the generated example SQL queries, incorporating these features, follow the BIRD SQL feature distribution (Ap- pendix Fig[5a)) The second step focuses on generating examples highlighting correct interpretation of the underlying data schema- the model 0 is asked generate examples using i, (column selec-
4

---

Published as a conference paper at ICLR 2025
tion using approach similar to (Talaci et al. 2024 and that are similar to the examples outlined in RR. Appendix A.13 provides the prompts used for the example generation).
While relevant example (e.g. showing nested JOIN query with multiple tables) can be helpful for questions that require complex JOIN queries, might also mislead the LLM for overuse (e.g. when a simple single table query is sufficient). This and the inherent ambiguity of natural language query qi for which we draw the examples by relevance, make the example selection challenging. Thus, generate and inject the examples to the prompt online per g. We ask the LLM generate many input-output pairs for in-context learning. The final set of synthetic examples for contains examples generated with both R, and Rt. This ensures that the example set diverse both in SQL features/clauses and the choice of relevant tables/columns used. The diversity of the example set desirable to avoid over-fitting the output to certain patterns (e.g., the model always writes a SQI. with JOIN if shown mostly JOIN examples). Mixing various examples for various SQL features and database tables with and without column filtering is observed result in better generation quality overall (see Appendix Table (). The generated synthetic examples can guide the model for more accurate text-to-SQL generation. In Appendix AA.3 Table g
we discuss how our synthetic example generation performs compared to selecting examples from often limited examples pools (e.g.. training dataset and cross- data augmentation (Li et al. 2024b)).
2.4 QUERY FIXER
In some cases, LLMs might generate queries that are syntactically incorrect. These queries are clear candidates for correction, as they fail to provide the correct answers. To address this, we apply an LLM-based query fixer that leverages the self-reflection (Shinn et al., 2024) method. The fixer reflects the previously generated query, using feedback such syntax error details empty result sets to guide the correction process. We continue this iterative fixing approach up to a specified number of attempts, & (set to three in this paper). Appendix Fig. 22 demonstrates the prompt used for this query fixing step. Additionally, Appendix section A.4 provides a detailed algorithm of the query fixing step.
2.5 SELECTION AGENT
With three different methods for generating SQL queries, for any given question. The key challenge in this step is selecting the correct SQL. query from this pool of candidates. A naive approach would be measure consistency among the candidates by executing them, grouping them based on their execution results, and selecting a query from the largest group as the most likely correct answer. However, this would assume that the most consistent answer is always the best one, which is not always the case. Instead, we propose a more refined picking strategy, Algorithm that relies on selection agent. Given a set of candidates SQL queries C' = {o,, C22 ... ). the final responses are selected by finding the candidate that has the highest score assigned by the selection model. This model 6p can take & candidates and rank them based on how accurately each of them answers the given question. Concretely, we formulate the selection of the final response as:
generate set of candidate queries
cJ = arg max
Σ
Qu, Hu, D)
(I)
GEC
where Qu refers to the user's question, H₂ is the provided hint, and D is the target database from which the question being asked. Eq. [ we pass candidates the selection model to be ranked, with k being between and n. In the extreme case of k 1, the model is unable to make comparisons between candidates, which complicates the evaluation process for the model. As k increases, comparing more candidates makes the process more challenging for the model, as it needs to consider different aspects simultaneously. Consequently, we set = 2 and train a model with classification objective compare only candidates at time.
Having a set of high-quality and diverse candidates, the most straightforward solution is to em- play off-the- off-the-shelf LLMs to make pairwise selections. However, experiments with 1omini-1.5-ae showed that using the LLM without fine-tuming resulted in only 58.01% % binary classification accu- racy. This is primarily due to the candidates being very similar to another, requiring fine-tuned
5

---

Published as a conference paper at ICLR 2025
model to learn the nuances and make more accurate decisions. To train the selection agent, first generate candidate SQL queries on the training set (of Text-to-SQL benchmarks), and group them into clusters based on their execution results. For cases where at least one cluster contains correct queries and others contains incorrect ones, we create training examples in the form of tuples (Qu, Ci,, Cj, Pij. PijsDij): where Qu is the user's question, C2 and Cj are the two candidate queries being compared, Dij is the database schema used by both candidates (Using the union of candidate schemas is important as it reduces cost and eliminates unnecessary information during comparison.), and Vij E 0, is the label indicating whether C₁ Cj is the correct query. To avoid order bias dur- ing training. we randomly shuffle the order of correct and incorrect queries in each pair. Since the number of cases with both correct and incorrect candidates is limited, for instances where no correct candidate exists, we include the ground truth SQL query the prompt as a hint to guide the model in generating correct candidates.
Algorithm& 3 Picking the final SQL query from a pool of candidates.
Require:
Set of candidate SQL queries = {ci, C2, Ca..., C.)) user question Q., hint Hu, target database D. and a selection model 9. er(a. D) the execution result of e; on D
1: r. 4- 0 for a E C I Initialize the score ri for each condidate query to zem for each distinct pair (c),6)) where + do
3: if ere,, D) = er orlej, D) then winner if the execution results match
5:
else
S., f- wnieema. Gy. D) Construct union of schemas used in os and oj
7:
0, (i. E {5,3) If Use binary classifier 92 to select the winner, è {t.3)
end
9: Yo + To + 11 Increase the score of the winner co by
10: end
1I: ef + m, Hf Select the candidate with the highest score as the final qQl-query cz
12: returney
In the pseudo-code for Algorithm 3 we begin by initializing score of zero for each candidate query. Then, for every distinct par of queries (c), we compare both (c,,c) and (c,, c.) to mitigate any order bias, ensuring that both candidates in a pair are fairly evaluated (ignoring the both side comparison will reduce the final performance by roughly 2%). If both queries produce the same execution result on the database, we mark one as the winner and increment its score, these results suggest consistency. If the execution results differ, we generate a union of the schema used by both queries and use the binary classifier determine which query is more likely be correct. The classifier takes into account the question, the two candidate queries, and the combined schema to make its decision. The winner's score then updated accordingly. After all comparisons, the candidate with the highest score is selected as the final query. In the rare case of tie in the final scores, we break the tie by selecting one of the candidates arbitrarily.
3 EXPERIMENTS
Datasets and Models We evaluate the performance of the proposed CHASE-SQL framework on two widely-recognized cross-domain datasets: BIRD Li al. 2024c) and Spider (Yu et al. 2018). BIRD contains over 12,751 unique question-SQL pairs from 95 large databases, spanning more than 37 professional domains, with databases designed to resemble real-world scenarios, featuring messy data rows and complex schemas. Spider consists of 10,181 questions and 5,693 unique com- plex SQL queries across 200 databases, covering 138 domains. The Spider dataset is divided into non-overlapping training, development, and test sets similar to BIRD. For both, we use execution accuracy (EX), the official metric for their respective leaderboard, as the primary evaluation metric to compare methods. Details of the models and their hyperparameters provided in Appendix section A.3
BIRD results We present the end-to-end Text-to-SQL performance of the proposed CHASE-SQL framework using Claude-3.5-somme Gemini 1.5 pro, and Mistral Large models on the BIRD devel- opment set, and Gemini 1.5 pro on the BIRD test set. We compare with both published methods (either with an available codebase and/or paper) and undisclosed methods. For a fair comparison with Gemini 1.5 pro, all LLM calls in the Claude-3.5-some setting, except for the selection model, made using 3.5-onntttt (previously-trainedd selection model is reused). To evaluate the

---

Published as a conference paper at ICLR 2025
performance of fully open-source models, we used fine-tuned Qwen2.5-coder model Team 2024 as the selection model for the Mistral Large model. As shown in Table CHASE-SOR SSL with Gemini 1.5 pro achieves 73.01% % accuracy the BIRD development set and 73.0% the BIRD holdout test set, outperforming all previous works and setting a new state-of-the-art performance.
Table 2: Performance Comparison of different Table 3: Performance Comparison of different Text- to-SQL methods on Spider test set.
to-SQL methods on BIRD benchmark.
Method
EX (Dev)
EX (Test)
Method
EX
Training with Spider
Published
MCS-SOL GPT-4
CHASE-SQL + Gemini 1,5 (Ours)
73.01
73.0
(T at 2024
89.6
CHASE-SQL Claude Sonuct (Ours)
69.53
CHASE-SQU QL Gemini 1,5 (Ours)
87.6
x
CHASE-SQL Mistral Large (Ours)
70.33
CHESS
Distiller GPT
Talaci ct 2024
87.2
X
Maamari al. 2024
67.21
71.83
DXU SQQ GPT4
CHESS
Gao ct 2023
86.6
4
[alaci 2024
65.00
66.69
DIN-SQ 4-
SCL GPT
Pourreza Rañccl
2024a
85.3
4
al. 2024
63.36
65.45
C Chiil
SupaSQL
Dong et al. 2025
82.3
( ct
2024a
58.5
62,66
RESDSOL 3B
Undisclosed
Li al 2023g
79,9
DIN-QQL Codex
Insights
70.26
Poorreza Rañiei 2024a
78.2
AskData 4p
72.03
72.39
13:354NNSQL
OpenSearch-x24 GPT-4o
72.28
Rai et 2023
78.0
PÜRPLE-RED- + GPT-4o
Arewise- GPT-4o
68.12
67.99
66.21
70.21
Grapiic B4FICARD
ExSI. granite 34b-code
67.47
67.76
[L 20236
77.6
Spider results We assess the generalizability of the proposed CHASE-SQL by evaluating it in an end-to-end way on the Spider test set without modifying the few-shot samples in the prompts or training new selection model, i.e. without using and data from the target distribution. This ap- proach allows us to test the performance of CHASE-SQL on different unseen query and database dis- tributions compared to the data from training distributions. demonsseetss that CHASE-SQL achieves an execution accuracy of 37.6% on the Spider test set, placing it second among methods that have undergone specific training or prompt optimization for the Spider dataset. This highlights the strong generalizability of CHASE-SQL. and its potential for generating high quality Text-to-SII. for unseen samples coming from very different distributions and unique challenges.
3.1 GENERATOR AND SELECTION PERFORMANCE
Generator + Fixer: To reveal performance of generators, we conducted an ablation study to evaluate the performance of each candidate generation method before and after applying the query fixer using two models of Gemini- 1.5-pro and Mistral Large (Al 2024). We compare the performance of the proposed gen- erators in producing single candidate query against the original BIRD prompt (Li et al.) 2024c), augmented with zero-shot COT reason- ing (Kojima et al. 2022), which serves as the baseline for assessing the quality of prompts. The results, shown in Table indicate that the proposed methods significantly improve SQL generation performance, compared to the naive baseline, towards the goal of producing high-quality candidates while maintaining diversity. Among the candidate generators, the online synthetic data generation approach produced impressive per- formance of 68.02% with Gemini- 1.5-pro model, demonstrating its effectiveness in leveraging test- time compute to improve LLM performance by generating high-quality synthetic examples. Fur- thermore, the query fixer proved crucial, enhancing the quality of the candidate pool and increasing performance by nearly 2% across all candidate generators.
Table 4: Ablation studies on single candidate genera- tion performance of the candidate generators compared with original BIRD prompt zero-shot CoT with Gem- ini 1.5 pro and Mistral Large on the BIRD dev set.
Gemini 1.5 pro
Mistral Large
Method
(96)
A(58)
EX (%)
A(%)
Baseline
$7.75
54,88
CoT
63.62
+5.87
59.64
4.76
DC Col
63.92
+6.
58.99
4.11
OS ICL
67.09
+9.34
56.32
1.44
Baseline w Query Fixer
61.58
+3.83
60.03
5.15
CPT Query Hixer
65.51
47.76
7.26
DC CoT w Query Fixer
65.77
+8.02
63.75
8,87
OS ICL Query Fixer
68.02
+10.27
61.47
6.59
7

---

Published as a conference paper at ICLR 2025
Selection: We conducted an analysis the binary selection accuracy of the selection agent for cases where, in a pairwise comparison, one candidate is correct and the other is incorrect. We exclude cases where both candidates are either correct or incorrect, the selection would not affect the outcome since both candidates have the same label. We compare the performance of Claude-3.5- sonnet and Gemini-1.5-pre (both out-of-the-box without fine-tuning) with two fine-tuned models: 1) Gemma 2 9B and 2) Gemini-I.3 f.5-lahhh As shown in Table[5] both fine-tuned models achieve higher accuracy than the untuned counterparts, demonstrating the importance of fine-tuning to teach the model about the specific preferences.
Candidate Generation Analysis: We analyze the performance of each candidate generator method indi- vidually. To better understand the performance poten- tial when effectively selecting the correct SQL query from the candidate pool, we generate seven candidate SQL queries from each generator method (21 candi- dates in total) for all samples in the BIRD development set. We determine this number of candidates based the observation that increasing the candidate pool be- yond 20 did not yield significant improvements, as il-
Table 5: Evaluating the binary selection accu- racy of the different selection models.
Selection Model
Binary Acc. (%)
Blaude-3.5-sommn
60.21
Gemini-1.5 1.5-ro
63.98
Tuned Gemma 9B
64.28
Tuned Gemini- 1.5--lash
71.01
lustrated in Fig. 2d By assuming access to an oracle
selection model that always selects the correct SQL query from the seven candidates, we calculate the upper-bound performance achievable for each generator. Conversely, by assuming adver- sarial selection model that always selects the wrong SQL query, we determine the lower-bound performance. Fig. Dillustrates the upper-bound and lower-bound performance for all three methods together with the performance of our selection agent. As shown, the upper-bound performance of the two different CoT methods is generally higher than that of the synthetic example generation method for different number of candidates. However, their lower-bound performance is also lower than the synthetic method. nower-bound accuracy reflects cases where all candidates reducing the noise in the selection process since it doesn't 't matter which candidate is chosen, so a higher lower-bound is preferred. This is evident in the selection agent's performance, where a drop in the lower bound leads to diminishing returns from increasing the upper bound, causing the selec- tion agent's performance to plateau. Additionally, the upper-bound performance of combining all three methods reaches %e.79% highlighting the significant room for improvement through better candidate picking methods. This demonstrates that the L.LM's parametric knowledge already con- tains the information needed to solve most questions, highlighting the need for ensemble approaches to effectively extract and utilize this knowledge.
correct,
Additionally, we evaluate the upper-bound performance by combining all candidates from three candidate generation methods across the simple, moderate, and challenging difficulty levels for the BIRD development set. These difficulty categories are assigned by human experts during the cre- ation of the BIRD development set. Fig. 2d] shows that, as expected, the upper-bound performance increases with the number of candidates across all difficulty levels. However, for the challenging and moderate classes, the improvement plateaus earlier than in the simple class, suggesting that gen- erating more samples does not further improve the upper-bound performance for these two difficulty levels.
Fig. 2 presents a Venn diagram showcasing the performance of three generation methods: Query Plan, Divide and Conquer, and with Synthetic Examples. The numbers within the intersecting re- gions represent the instances where multiple methods generated least one correct candidate. This diagram visually highlights the unique contributions of each method, which indicates the necessity of using all three generators. Additionally, in Fig. 3bb we compare the number of correct queries generated by each SQL generation method that are not correct by the other generators. The divide- and-conquer approach outperforms the others on challenging questions, while the query plan method excels on moderately difficult queries. To further analyze the performance of the generators across different domains and varying numbers of columns and tables, we compare the number of correct queries generated for each database, shown in Appendix Fig. 5 As illustrated, both CoT methods generally perform similarly across databases, while the online synthetic example generation method significantly increases diversity, resulting in more correct answers overall across different databases.
8

---

Published as a conference paper at ICLR 2025
Divide and Conquer Examples
Ontire Synthetic Eumrples
Loare
Reader Creakittes
Suadey Cookiaaion
(a) Upper-bound and lower-bound Accuracy for Divide and Conquer CoT
(h) Upper-hound and lower-hound Accuracy for Online Synthetic Example
Rccertcy Chasser
Query Pian Exemples
Liee wertd
weard
Number Condblrtes
(d) Upper-bound performance of all three can- didate generators across different difficulty cate- gories.
(c) Upper-bound and lower-bound performance for Query Plan COT.
Figure 2: Comparison of the upper- and lower-bound performance of different candidate generators.
Haaz
Syathesi Enanpie
Comperison of Methods by Number Correct Samples
23
(h/Me Caf)
X4
72
21
uebber of Cammer Aampaee
Conquar
Utsihee Questimss 358
(a) Venn diagram illustrating the number of instances for which each method: Query Plan, Synthetic Example, Divide and Con- quer, produces at least one correct candi- date. The overlap regions represent multi- ple methods generating correct candidates.
Oif level
(b) Number correct queries across different complexity levels that were answered by each method.
Figure 3: Comparison of SQL generation methods: Venn diagram showing unique and overlapping correct answers (left) and the performance across different complexity levels (right).
We evaluate the query-picking performance by comparing the Text- to-SQL execution accuracy of the selection agent with the self-consistency method (using majority voting) Wang et al. (2022), an oracle model (upper bound), and an adversarial model (lower bound). To conduct the evaluation, we generate samples from each candidate generation method using two different sampling temperatures: 0.5 and 1.8. The results, shown in Table demonstrate that the selection agent significantly outperforms the self-consistency method with large margin, roughly 6%. As expected, increasing the sampling temperature raises the upper bound but also lowers the lower bound. This effect more pronounced for the synthetic data generation method compared to the two CoT methods, mainly because LLMs generate reasoning steps before producing the final SQL query, which helps mitigate the randomness introduced by high-temperature sampling. The performance with self-consistency method generally decreases temperature increases, since the
Selection Agent Analysis:

---

Published as a conference paper at ICLR 2025
majority cluster becomes smaller with more random queries. However, the proposed trained selec- tion agent is less affected by temperature scaling and, in two cases, even improved its performance with more diverse pool of samples.
Table 6: Performance comparison of different picking methods the candidates generated by the candidate generators on BIRD development set with two different temperatures. QP refers to query plan COT, DC refers to divide and conquer COT, and OS is the online synthetic example generation method.
Picking Method
QP (T=0.5)
(P(T-=)8)
(T=00))
(T=1.))
(T=0.5)
OS (T=1.8)
Lower Bound
50.46
48.63
51.37
47.39
60.43
50.98
Upper Bound
78.55
80.44
78.42
79.34
74.77
79,66
Self-gonsistency
65.78
65.51
66.43
64.41
67.34
Our Selection Agent
71.7
71,73
71.31
70.53
70.4
71,38
3.2 ABLATION STUDIES
In the previous sections, we evaluate the importance of the selection agent and each candidate generation method. Next, we focus on the analysis of the re- maining components of CHASE-SQL: LSH for value retrieval, the query fixer, and three reasoning strategies (QP, OS, and DC). Table 7 shows the perfor- mance of CHASE-SQL without each of these steps, highlighting their signifi- cance in achieving higher-quality per- formance. The results demonstrate the contribution of each component, where removing LSH, the query fixer, or any of the candidate generators leads to reduction in execu- tion accuracy, further validating the importance of these components of CHASE-SQL. Moreover, the table compares the performance of our binary selection agent with two other selection methods: self-consistency (Wang et al. (022) and a ranker agent. The ranker agent receives all candidates generated by our three candidate generators in single prompt, compares them, and produce a rank- ing for each. For the ranker agent, we select the query with the lowest rank as the best answer. The binary selection agent significantly outperforms both the self-consisteng and ranker agents, demonstrating the effectiveness of the proposed method.
Table 7: Ablation studies the performance of CHASE-SQL after removing the query fixer, LSH for value retrieval, and rea- soning strategies, i.e. QP. OS, and DC.
Method
Execution Accurccy (%) A(%)
CHASE-SQI. All
73.01
CHASE-SQL w self-consistency
CHASE-SQL. w ranker agent
65.51
-7.5
CHASE-SQL wlo LSH
70.09
-2.92
CHASE-SQL wlo Query Fixer
69.23
-3.78
CHASE-SQL wlo QP
72.36
-0.65
CHASE-SQL OS
-0.85
CHASE-SQL wlo DC
71.77
-1.24
4 CONCLUSION
We introduce a novel agentic framework, CHASE-SQL, to leverage test-time compute for generat- ing diverse, high-quality SQL queries and accurately selecting the correct one. We propose multiple chain-of-thougg prompting methods and an online synthetic example generation technique, along with query selection mechanism that scores candidates based pairwise comparisons. Our frame- work, CHASE-SQL, sets a new state-of-the-art in the notable public Text-to-SQL leaderboard (at the time of the submission), demonstrating the effectiveness of test-time computation for both gen- erating diverse queries and selecting the most accurate response. CHASE-SQL addresses key issues like query diversity and selection optimization, paving the way for further improvements in complex reasoning tasks encountered at real-world Text-to-SQL. challenges.
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In ICML. 2024 Workshop on In- In-Context Learning, 2024. URL httpt:://///.....
goi7DFH1qS
Mistral AL. Mistral large 2024. Accessed: 2024-11-16.
10

---

Published as a conference paper at ICLR 2025
Ion Androutsopoulos, Graeme D Ritchie. and Peter Thanisch. Natural language interfaces to databases an introduction. Natural language engineering. 1(1):29 81, 1995.
Ruichu Cai, Jinjie Yuan, Boyan Xu, and Zhifeng Hao. Sadga: Structure-awee aware dual graph aggrega- tion network for text-to-sql. Advances Neural Information Processing Systems, 34:7664-7676, 2021.
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. Lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations. arXiv preprint arXiv:2106.0102) 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint MxXiv:2107.03377 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05//28 2023.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql cross-domain databases. Computa- lional Linguistics, 47(2):309-332. 2021.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry. pp. 253-262. 2004.
Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Jinshu Lin, Dongfang Lou, et al. C3: Zero-shot text-to-sql with chatgpt. arXiv preprint arXiv:2307.07306, 2023.
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308. 15363, 2023.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Mar-
tin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349. 2020.
Vagelis Hristidis, Yannis Papakonstantinou, and Luis Gravano. Efficient ir-style keyword search over relational databases. In Proceedings 2003 VLDB Conference, pp. 850-861. Elsevier, 2003.
Wonseok Hwang. Jinyeong Yim, Seunghyun Park, and Minjoon Sco. A comprehensive explorroi on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01066 2019.
George Katsogianni--eennan Meemarakis and Georgia Koutrika. A survey on deep learning approaches for text-to-sql. The VLDB Journal, 32(4):905-95) -966,996377 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
Dongjun Lee. Choongwon Park, Jachyuk Kim, and Heesoo Park. Mcs-sql: Leveraging multiple prompts and multiple-choice selection for text-to-sql generation. arXiv preprint arXiv:2405.07467, 2024.
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to sql: Are we fully ready? arXiv preprint arXiv:2406.0/265 2024a.
Fei Li and Hosagrahar V Jagadish. Constructing interactive natural language interface for rela- tional databases. Proceedings of the VLDB Endowment, 8(1):73-84 2014.
Haoyang Li. Jing Zhang, Cuiping Li. and Hong Chen. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13067-13075. 2023a.
11

---

Published as a conference paper at ICLR 2025
Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang. Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data. 2(3):1 2024b.
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. arXiv preprint wxXiv2331.07557, 2023b.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024c.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-Ieve code generation with alphacode. Science, 378(6624)11092-19999 2022.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirre, Marianna Apidianaki, and Ivan Vulić (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-1 14, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 10. 18653v1// 2022.deelio-1.
Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-sql in the age of well-reasoned language models. arXiv preprint arXiv:2408.000 07700, 2024.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. arXiv preprint arXiv:2305. 12586, 2023.
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106-26128. PMLR, 2023.
Rubén Pérez-Mercado, Antonio Balderas, Andrés Muñoz, Juan Francisco Cabrera, Manuel Palomo- Duarte, and Juan Manuel Dodero. Chatbotsql: Conversational agent to support relational database query language learning. SoftwareX, 22:10134: 2023.
Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to- sql with self-correction. Advances in Neural Information Processing Systems, 36, 2024a.
Mohammadreza Pourreza and Davood Rafici. Dts-sql: Decomposed text-to-sql with small large language models. arXiv preprint arXiv:2402.01111 2024b.
Mohammadreza Pourreza, Ruoxi Sun, Hailong Li, Lesly Miculicich, Tomas Pfister, and Sercan O Arik. Sql-gen: Bridging the dialect gap for text-to-sql via synthetic data and model merging. arXiv preprint arXiv:2408. 12733, 2024.
Abdul Quamar, Vasilis Efthymiou, Chuan Lei, and Fatma Özcan. Natural language interfaces to data. Found. Trends Databases, 11(4):3194114 2022. doi: 10. 10.1561/1900000007. URL https:
7/doi. org. 1111111000009990000155
Daking Rai, Bailin Wang, Yilun Zhou, and Ziyu Yao. Improving generalization in language model- based text-to-sql semantic parsing: Two simple semantic boundary- techniques. arXiv preprint arXiv:2305. 17378, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Ruoxi Sun, Serean Ô Arik, Alex Muzio, Lesly Miculicich, Satya Gundabathula, Pengcheng Yin, Hanjun Dai, Hootan Nakhost, Rajarishi Sinha, Zifeng Wang, et al. Sql-palm: Improved large language model adaptation for text-to-sql (extended). arXiv preprint arXhv:2306.00739 2023.
Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprini arXi::2405.16775, 2024.
Qwen LM Team. Qwen2.5-coder.
htttps://ggenmmmggihbbb....00
5 5-coder/ 2024. Accessed: 2024-11-16.
Bailin Wang. Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat- sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint 19.04:111 2019.
Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian- Wen Zhang, Zhao Yan, and Zhoujun Li. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arX1v:2312.11222, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint wtXh::220..1177: 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24883 248337, 2022.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Jun- ning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv:2310 10. 10634, 2023.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08314, 2020.
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingming Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv: 180.088889 2018.
Tao Yu, Chien-Sheng WU, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang. Dragomir Radev, Richard Socher, and Caiming Xiong. Grappa: Grammar- araamma--ugmented pre-training for table semantic parsing. arXiv preprint arXiv:2009. 1845,, 2020.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LIMITATIONS AND FUTURE WORKS
Based on the analysis presented in this paper, we demonstrate that the parametric knowledge of recent large language models, such Gemini-1.5-pro and Mistral Large, contains the necessary information to answer most challenging questions in notable text-to-SQL. benchmarks, as evidenced by their high pass @ass@K performance. This highlights the challenge of effectively utilizing the rea- soning ability of these models to select the best answer among the candidates. In our work, we concluded that pairwise comparison an effective approach to identify the best candidate. How- ever, we believe this performance could be further improved by leveraging the reasoning capabilitiess of the models, either through thain-of-tthoueee prompting or employing more sophisticated search methods, which we leave directions for future work. Additionally, further research is needed to address the detection of ambiguous questions and to improve the reliability of text-to-SQL. sys- tems. Most methodologies in this domain assume that all questions are answerable, which remains a significant limitation in current text-to-SQL approaches.
A.2 RELATED WORKS
Early Text-to-SQI. methods predominantly utilized sequence-to-sequeee architectures, encoding user queries and database schemas using models such as Graph Neural Networks (GNNs), Re- current Neural Networks (RNNs). Long Short-Term Memory (LSTM) networks, and pre-trained transformer encoders (Hwang et al. 2019] Cai et al.] 2021] Cao et al. 2021). On the decoding side, these systems employed either slot-filling auto regoorregressive modelling approaches construct the final SQL queries from the encoded inputs (Choi et al. 2021 Wang et al. 2019). Additionally. tab- ular language models like TaBERT (Yin et al.) 2020). TaPas (Herzig al. 2020)) and Grappa (Yul eet al., (220)) have been developed to encode both tables and textual data effectively. However, the landscape has evolved with the widespread use of LLMs, which have largely replaced earlier meth- & Koutrika [0233 Quamar et al.)
ods with their superior performance (Katsogianmi--Miimaae
2022]. Initially, efforts concentrated on optimizing prompt designs for these LLMs (Pourreza & Rañeii (204a) (ao et al.) [023]] Dong et al. [002). Subsequent advancements have introduced more complex methodologies, including schema linking (Li et al.] 2024b Talaei et al. 2024|] Pourreza
& Rafiei 2024a b). self-correction or self-debugging (Chen et al. 2023 Wang et al.) 2023 Ta-
laei et al. 2024), and self-consistency techniques (Lee et al. 2024 Sun et al., 2023] Talaei et a
2024 Maamari et al. 2024], further enhancing the performance by proposing complex LLM-based
pipelines.
As previously discussed, one approach enhance Text-to-SQL performance is based on the con- sistency of LLM responses. The self-consistency approach, as proposed by Wang et al. (0022), involves sampling multiple responses from an LLM and selecting the most consistent answer based on the majority vote. In the Text-to-SQL context, this technique extends to generating multiple SQL queries for given question, grouping these queries by their execution results, and selecting query from the largest cluster as the most consistent answer (Gao et al., [2233] Sun et al. 2023 Talaci et al., 2024]. However, recent studies have pointed out the limitations of this method in reliably identifying the correct answer. In response, MCS-SQL (Lee et al. 2024) introduced an approach that utilizes an L.M to rerank the most consistent answers, moving beyond simple majority voting. Despite these advancements, reliance on consistency as filtering mechanism can inadvertently exclude correct queries that are less frequent among generated candidates, as a critical bottleneck.
A.3 MODELS
All experiments are conducted using models from the Gemini and Claude, known for their ability to handle long contextual information (Maamari et al. 2024), which is crucial for the Text-to-SQL task involving queries from large databases. For candidate generation, online synthetic example generation, query fixing, column filtering, and keyword extraction, we reported the performance with two models of Gemini 1.5 Pro and Claude-3.5 S.n-one.... For the query-picking model, train Gemini 1.5 Flash model (which has much less latency than the Gemini 1.5 Pro model) on a dataset of 3.8K samples generated by running the candidate generators the BIRD training dataset. The
14

---

Published as a conference paper at ICLR 2025
Gemini 1.5 Flash model is trained for 10 epochs using LoRA adapter with rank of 16 using Vertex Al tuning API.
A.4 QUERY FIXING ALGORITHM In this section, we present the algorithm of the query fixing step of our propose approach.
Algorithm 4 Query fixing method.
Require:
Set of candidate SQL queries C'= fci, C2.... Ein user question Qu. hint Hu, larget databuse
query lixing threshold and a fixer model 6
1: Crized +- D/ Initialize on empty set Ciixed to store the foxed queries for each candidate query do
3: Exeecction, +- Earecddefe,, D); Execute the SQL query over the database
error Execution, Exccution, then for j E {1.2.....
times 6, +- 9y cu, Exrecution.) Hf Fix the query using the fixer model
do fixing
Execution, +- Exrecutel(a. D2 Re-execute the fixed query
if not (ervor in Execution, or Exccution; () then
Cresa Cyeee {c,) Add fixed query
10:
break Il Exit the fixing loop if successuul
11:
end
12:
end for
13:
else
14:
Crised Crised U (o.) HI Add the query is if fixing needed
15:
end if
16:
end
17:
return Criced Return the set of fixed queries
15

---

Published as a conference paper at ICLR 2025
A.5 VALUE RETRIEVAL EXAMPLE
In this section, we provide an example of the value retrieval step. For the Given Question: "What the highest eligible free rate for K-12 students in the schools in Alameda County?", from the celifornia_schooll Database, the closest database values that are retrieved from the LSH and after reranking are follows:
Table "schools":
"SOCType": ["Preschool"],
"EILName": ("Preschool"),
"School":
"Preschool",
"MethodSchools",
Alamed County Community",
Alameda County Opportunity",
Allameda High"].
"MailStreet": ["4600 Student Lane"], "Street": 4600 Student Lane"].
'Amme_Namel" I": ["Free"],
" NAdm!_ ["Freeman"),
"MailCity": [" IlAlmedaa").
"City": |"Alameda"),
"AdmFName 1": ["Kate", "Nate", "Bree"],
"GSserved": ["K-
"GSoffered"": ["K-12"],
"StreetAbr": ("4600 Student Ln."),
"MailStrAbb":: ["4600 Student Ln."],
"AdmLNamee": ["Yount"],
"AdmFName"": ["Bree"]]
"County":: ["Alameda"],
"District": ["Alameda Unified", "Tri- PTri-County ROP"],
Table "frpm":
"School Type": ["Preschool"]]
"School Name":
"MethodSchools".
Alaaeea County Community",
Allameda High"
J. "County Name": ["Alameda"),
Table "satscores":
"sname": ["Alameda High"),
"cname": ["Allmeda")],
"dname": [" l'Alameda County Office of Education")
Figure 4: An example the divide and conquer CoT method
A.6 PERFORMANCE BASED ON DATABASE
In this section, we present the number of samples across different databases where only one of the candidate generators produces correct result, meaning the other two generators fail to provide a correct answer. A value of zero for any generator in this figure indicates that whenever that generator produces a correct result, the other two generators also manage to generate at least one correct answer.
16

---

Published as a conference paper at ICLR 2025
Comparison of Methods by Correct Samples on Different Databases
Query Plan CoT
10
Orline Syniheii
Stodd Concuer
Number Cornec.
tarcelooy
Ilcontioiis prediction
crdettane community
formula 1
californi schools
car gemes
europeen, bestast 2
student chib
debit card specialtingg
financial
superero
Detabases
Figure 5: Number of correct queries by each method across different databases of BIRD development set.
A.7 ERROR ANALYSIS
wronnn bold quary
6.7%
No CORTICE conditnne
72
COFTEC ARSAV"
Figure 6: Distribution of system performance based the final answer correctness. The chart shows the proportion of correct final answers, correct queries existing among candidates but not chosen (wrong selection), correct candidate cases, and cases the golden SQL query wrong.
Fig. provides a pie chart that breaks down the system's performance into four categories: correct final answer (72.9%), correct exists among candidates but not chosen (10.4%), wrong generations or no correct candidate (6.7%), and wrong golden query (10.0%). The majority of responses are correct final answers, but notable portion falls under correct answers not being chosen by the system. This breakdown helps understanding areas where the system excels and where improvements can be targeted.
17

---

Published as a conference paper at ICLR 2025
02.0%
62.4%
82.1%
$6.0%
12.9%
Card Ganvy
Prbrt Card Sperkatidn
Parrapun Hoshhl
Noraala
Makket
Asperbery
Thrmmbont Fredittion
Torbology
Schaok
Cameandiy
Figure 7: Correctness comparison of the system different databases in metries: (1) per- centage where the correct query exists among the candidates, and (2) percentage where the correct query chosen by the selection agent.
Fig. presents comparative analysis of system correctness across multiple databases. The x-axis lists various databases or categories such California Schools, Formula 1, and Superhero, while the y-axis represents the percentage performance. Two key metrics are visualized: the first the percentage where the correct answer exists among the candidates (shown by one bar per category), and the second is the percentage where the correct answer is chosen by the selection system (depicted by second bar for each category).
A.7.1 SELECTION AGENT ERROR ANALYSIS
In this section, we examine cases where at least one of the candidate SQL queries generated by the three generators matched the ground truth answer, but the selection agent assigned the highest score to another, incorrect candidate. We categorized these errors into four groups: (1) Vague questions, (2) Wrong picking, (3) Data Integrity Error, and (4) Incorrect gold query. Fig. & illustrates the distribution each category among the sample queries. In the following sections, we will discuss each of these categories in more detail.
whing bold uuery
Data Integnty Eror
Sague Quastion
Picking
Figure 8: Error analysis on the cases where selection agent failed to pick the correct SQL query which among the candidates.
18

---

where the selection agent picked a correct SQL query and Published as a conference paper at ICLR 2025
Wrong Picking Errors: The largest portion of errors occurs when the candidate with the highest score from the selection agent is missing a required column, table, or SQL clause. In our analysis, could not identify specific types of patterns in the model's mistakes as these mistakes includes different types of the errors almost for each instance. Fig. provides an example where the selected SQL. query incorrectly * to return all columns, instead of just returning the id as specified in the ground truth answer.
Question: List all patients who were followed up at the outpatient clinic who underwent laboratory test in October 1991 and had total blood billrubin level within the normal range.
Evidence: followed up outpatient
Admission bilirubin
laboratory April
Date
within
Gold SQL: SELECT DISTINGT TLJD FROM Patient AS T1 INNER JOIN Laboratory TST2 ON T1,D T2.ID WHERE T1Addmission + AND T2. T-BIL' 2.0 AND T2.Date LIKE 1991-10-%
Ploked SQL: SELECT DISTINCT TL. FROM Patient AS T1 INNER JOIN Laboratory AS T2 ON TIID T2.ID WHERE Tl.Admission AND T-BL'- 2.0 AND T2.Date 1991-10%%
Figure 9; An example of selection agent preferred SQL query which is incorrect.
The second largest portion of errors occurs when the ground truth SQL query is incorrect, and one of the candidate queries generated by our model replicates the same mistake. However, the selection agent ultimately picks another candidate that correctly answers the question. Fig. 10 provides example of such extra molecule ID column in the SELECT clause, which was not specified the question.
Wrong Golden Query Error:
where the ground truth query includes
Question:
What are the labels for T2000, TROO1 and TROD2?
Evidence: TR0O0, TR001 and TR002 are molecule label = mean molecules carcinoganic; label means molecules are non- carcinogenic
SELECT molecule T.label T.molecule_id ('TR000', 'TRO01. TRO02')
Plaked SQL: SELECT label FROM molecule WHERE molecule, jid IN ('TR00D', 'TROO,, (TRO02))
Figure 10: An example of an the gold query was wrong.
Vague Question: Another significant portion of errors occurs when the question does not specify which column to return or use for filtering, and multiple columns could satisfy the query. these cases, although one of the candidates was the correct SQL. query, the selection model favored another response that could also considered correct. Fig. illlsttrate such case where "Fresno" could refer to either city or county, but the question doesn't specify which one to return. The selection model chose the query that used "city" and did not select the candidate that used "county.
Question: How many schools in Fresno (directly funded) have number of test takers not more than 250?
Evidence:
CDSCode) frpm T1.CSCode WHERE Charter Funding 'Directly funded AND T1. County Name 'Fresno' AND Num TatTakr 250
Picked SQL: SELECT COUNT(T.cDSCodd)] FROM schools AS T1 INNER JOIN satscores AB T2 ON T.CDSCode T2.ods WHERE Ti.city FFresnoo AND TI.FundingType 'Directly funded' AND N2.uumTarTakr 250
Figure 11:
An example of an error case where the selection model picked query which could be considered as correct as the question vague.
Data Integrity Error: Finally, the smallest category of errors involves cases where two or more columns supposed have consistent values, but more columns contain missing values.
19

---

Published as a conference paper at ICLR 2025
shows where the "School" and "School Name" columns both expected to contain the names of schools, but one the columns has missing values.
For example, Fig. 12
than 30 difference enrollements between K-12 and 5-177 Please also give the full street adress of the sohools
Question: List
of schools
Evidance: Diffrence in enrollement 'Enrollment (K-12) 'Enrollment (Ages 5-17)
TLSohool. T1.8treet schools T1.DDSCode 122CD8Code (Agea
frpm
Enrollment (X-12)"
Picked SELECT School Name: T1.Street FROM schools AS INNER JOIN from AS ON M.CDDCCde 12.CDSCode WHERE T2. Enrollment (K- 12) T2. Enroliment (Ages 5-17)
Figure 12:
An example of because of the data inconsistency the execution accuracy was zero for this candidate.
where the selection agent picked correct candidate but
A.7.2 ERROR ANALYSES
We present the manual error analysis we conducted on one-third of the cases where none of the generated candidate queries were correct. We categorized these errors into five main types: (1) Schema linking errors, (2) Incorrect logic, (3) SQL function errors, (4) JOIN issues, and (5) Ignoring evidence. Fig. 13 illustrrass the distribution of these error categories. As shown, the most common emrors occur when none of the candidate queries correctly utilized the columns or tables required to answer the question. In the following section, we describe the specific types of errors that fall under each category.
Wrong Logic
ignoring Evidence
Missing JOIN
3.2
6.5%
19 4%
SQL
Schema linking
Figure 13: Error analysis on the cases where all candidate generators failed to produce single correct answer.
Schema Linking Errors: The schema linking errors category includes cases where of the generated candidate SQL queries correctly use the columns required to answer the question. These errors often occur in databases where column names ambiguous confusing for the model. Fig. p4povvidss an example where the L.M failed to correctly calculate the average to return the correct column that was expected.
Wrong Logic Error: This category, which represents the second largest portion of errors, includes cases where the logic of the generated candidate queries is incorrect. These errors involve missing elements such as the DISTINCT keyword, NOT NULL conditions, missing columns in the SELECT clause, incorrect missing conditions in the WHERE or HAVING clauses. An example is shown in Fig. [5] provides an example where the LLM failed to correctly calculate the average total price due to incorrect logic computing the average total price.
20

---

Published as a conference paper at ICLR 2025
Question:
Who
perform better crossing actions?
indicate their player
Evidence: perform better crossing actions refers to MAX(erossing)
Player,
Random Candidate SQL: BOLEET player_apl_id FROM Player_ Attributes Player.Attributes ORDER BY "crossing" DESC LIMIT
Figure 14: An example of schema linking error category.
Question:
average stations Czech Republic?
transactions
Evidence: Oas station in the Czech Republic implies that Country CCZE'
Gold SQL: SELECT (TG(Ce)Price) FROM transactione. 1k AS T1 INNER JOIN gasstations AS T2 ON T1.GasStationD T2.9asStationID WHERE C2.Cuuyyyy 'ZZE
AVG(TlAmmuut gasstations TI.GasStationiD T2.0asStationID WHERE Co.nouy
FROM
Figure 15: An example of wrong logic error category.
SQL Functions Error: This category, the third-largest source of errors, includes queries where the error results from the incorrect use of, or failure include, SQL functions such as COUNTO, CASTO, AVGO, ROUNDO, and others. Fig. [6] illustrates a case where none of the candidate queries used the ROUND() function as required by the question.
Question: How much of the hydrogen in malecule TR206 accounted for? Please provide enswer as percentage with four decimal places.
hydragen refers to element 'h;; TR206 is the molecule id: percentage = DIVIDE 'h'), CoUNT id)) as percent where molecule_id 'TR206"
Evidence:
ROUND(CASTCCOUNTCCAS
WHERE T.molecule_id REAL)
CouNT(T.atom, id).4)
Random Candidate SQL: SELECT (SST(SUM WHEN Tt.alement # 'hy THEN 1 ELSE o END) As REAL) 100 COUNT(Tl.atom) id) FROM atom AS WHERE Ti.molecule_ié 'TR206"
Figure 16: An example of wrong usage of SQL functions category.
Joins Error: This category represents smaller portion the error cases, where the source the error is missing more tables in the JOIN clauses of the predicted SQL queries.
This category represents the smallest portion of error cases, where the correct column or table explicitly mentioned in the provided evidence, but the LLM disregards and uses its own selection instead.
Ignoring Evidence Error:
A.8 DIVIDE AND CONQUER PROMPT
In this section, we provide an example divide-and andie--ed--ooquer prompt used in one of the few-shot in-context learning demonstrations to illustrate the decomposition and aggregation steps.
21

---

Published as a conference paper at ICLR 2025
Database Info
CETTABASESCCEMME
Question
Question: What is the gender of the youngest client who opened account in the lowest average salary branch?
Hint: Given that Later birthdate refers younger age; A11 refers average salary
#$1. Divide and Conquer:**
MMiin Question:** What is the gender of the youngest client who opened account in the lowest average salary branch?
** Analysis:**** Question asking about 'gender', and it appears in table 'client'. We will use this as the output column, selecting it from the youngest client in the lowest average salary branch. **Pseudo SQL:** SELECT 'TI' ''endder' FROM 'client" AS 'TI' WHERE youngest client in the lowest average salary branch
"*Sub-question 1:** youngest client in the lowest average salary branch *** ***Analsis:** According to the hint, need info, and the youngest client can be obtained from using the 'birth_daa datee column of table "client'. The items between these tables can be INNER JOIN using district_id.
the 'A11' from 'district' to get the salary
P*esudo SQL:** SELECT 'T1". ".",'"o" "client_dd' FROM 'client" AS 'T1' INNER JOIN 'district' AS "T2' ON "TI" 'district id 'T2" 'district id' WHERE lowest average salary branch ORDER BY 'T1' 'birth date' DESC NULLS LAST LIMIT
** Sub-questtion 1.1:*# lowest average salary branch
* ** Analysis:** We can get the lowest average salary branch using order by 'All' ASC and pick 1. The column 'All' is not NULLABLE,
do not need to add "IS NOT NULL" filter *sesudo SQL:** SELECT "district.id" FROM 'district' ORDER BY 'A11' ASC LIMIT
#. Assembling SQL:**
Sub-queestion 1.1 (lowest average salary branch):** **SQ ** SELECT 'district.d" FROM 'district' ORDER BY 'Al1' ASC LIMIT
* **Sub-question (youngest client in the lowest average salary branch):
**SQL.*** SELECT 'T1'." 'client ' id' FROM "client" AS "T1" INNER JOIN 'district' AS "T2" ON "TI', 'district = "T2'. 'district id' WHERE 'T2'. 'district id' (SELECT 'district FROM 'district' ORDER BY 'A11' LIMIT 1) ORDER BY 'TI', "i.r"birth_dtte"" DESC NULLS
LAST LIMIT
**iain Question (gender of the client):**
**SQL:** SELECT "T1' 'gender' FROM 'client" AS 'T1' WHERE 'Tl'''clientiid" (SELECT "T1'.' 'client id' FROM 'client' AS 'T1' INNER JOIN 'district' AS "T2' ON 'TI' 'isstrict "T2', "district_id' WHERE 'T2' 'district id' IN (SELECT 'district.id" FROM 'district' ORDER BY 'A11' ASC LIMIT 1) ORDER BY 'T1'. "1'..birth_date DESC NULLS LAST LIMIT
**3. Simplification and Optimization:**
The nested queries can be combined using single 'INNER JOIN and the filtering can be done within a single 'ORDER BY clause.
**Final Optimized SQL Query:
SELECT "T1" 'gendrr'' FROM 'client" AS "T1' INNER JOIN 'district" AS 'T2' ON 'T1'. 'district id' ' id' ORDER BY 'T2:'111' ASC, 'T1 'birth date' DESC
NULLS LAST LIMIT
Figure 17: An example of the divide and conquer CoT method
22

---

Published as a conference paper at ICLR 2025
An example of question/SQL pair where the divide-and-comd con-cor CoT approach was the only gener- ator that solved the problem.
Question: How much faster in percentage the champion than the driver who finished the race last in the 2008 Australian Grand Prix?
Query: SELECT
T1.postionOrder,
CASE
WHEN T1.positionOrder =1 THEN
(CAST(SUBSTR(TA)) (Ti.tmm, 1, AS REAL) 33600) (OAST(SUBSTa (AT.BBSss..imme, 3, AS REAL) 660)
(AAS(SSBSTR((1Liin 6) AS REAL)
ELSE CAST[GUBSTR(TT.tim REAL)
timo_seconds
INNER JOIN
ON Ti.racedd T2.raceld
WHERE
T2.name 'Australian Grand Prix'
AND T1.time 18 NOT NULL
AND T2.year 2008
}.
champion time FROM seo_ses
WHERE positionOrder
).
last driver incremental AS
SELECT time, seoonds
FROM time, in, seconds
WHERE positionOrder
SELECT MAX(positionOrder)
FROM time, in_einneecodd
SELECT
[CAST((BELECT time seconda driver incremenalmm REAL]*
100)
(SELECT time, seconds
(SELECT time_seconde
seconds FROM intermmnrrrlerremnntta)
FROM champion,timo
Figure 18: example of query that has been solved only with divide and conquer CoT method.
23

---

Published as a conference paper at ICLR 2025
A.9 QUERY PLAN PROMPT
In this section, we provide an example of a query (execution) plan prompt used in one of the few-shot in-context learning demonstrations illustrate the steps.
Database Info
(DATABASE SCHEMA}
Answer Repeating the question and evidence, and generating the SQL with query plan.
** **Question*** **; How many Thai restaurants can be found in San Pablo Ave, Albany?
"Evidence**: Thai restaurant refers to food = 'thai'; San Pablo Ave Albany refers to street_name 'san pablo ave' AND T1.city "albany'
** *Query *lan**:
** Preparation *
Initialize the process: Start preparing execute the query.
2. Prepare storage: Set up storage space (registers) to hold temporary results, initializing them to NULL.
3. Open the location table: Open the location table so we can read from it.
Open generalinfo table: Open the generalinfo table so we can read from it.
** Matching Restaurants:
Start reading the location table: Move to the first row in the location table.
Check if the street matches: Look at the street name column of the current row in location.
it's not "san pablo ave," skip this row.
Identify the matching row: Store the identifier (row ID) of this location entry.
Find the corresponding row in generalinfo: Use the row ID from location directly find the matching row in generalinfo.
Check if the food type matches: Look at the food_type column generalinfo. If not "thai,"
skip this row.
Check if the city matches: Look at the city column in generalinfo. it's "albany," skip this row.
** Counting Restaurants:
Prepare to count this match: If all checks pass, prepare to include this row in the final count.
2. Count this match: Increment the count for each row that meets all the criteria. Move to the next row location: Go back to the location table and move to the next row,
repeating the process until all rows are checked.
Finalize the count: Once all rows have been checked, finalize the count of matching rows.
Prepare the result: Copy the final count prepare it for output.
** Delivering the **
1. Output the result: Output the final count, which is the number of restaurants that match all the specified criteria.
2. End the process: Stop the query execution process.
Setup phase: Before starting the actual query execution, the system prepares specific values it will be looking for, like "san pablo ave," "thai," and "albany."
**Final Optimized SQL Query:
SELECT COUNT(T1 _id_restaurant) FROM generalinfo AS TI INNER JOIN location AS T2 ON restaurrua = 12.id WHERE TI.food 'thai' AND T1.city = "albany' AND 22.street_name 'san pablo ave'
Figure 19: An example of the query plan CoT method
24

---

Published as a conference paper at ICLR 2025
Fig. 20 provides an example of query plan generated by using the EXPLAIN command.
Qpentice
1962
10a-d: volite auttlidee sotseares
bell
PSB DIMBELL THEE Teselt
Detupes
214.41
HILL
LIMIT counter
Sperdaed
T000-A Sin FOOm
213t- CARLor colan
DICLITCO
([4]
((5)m, gole
kepes(33)
Cilln
Petarn
CeeBOZ
Bayer1s)
Oclam
11
cilll+ OUETRE e estone
Iroegez
Figure 20: An example of SQLite query plan generated by using the EXPLAIN command.
Additionally Fig. 21 provides an example question that was solved by using the query plan-based CoT strategy.
Question: State the names and full communication address of high schools in Monterey which has more than 800 free or reduced price meals for ages 15-17?
Query SQL:
T2.city,
T2.Zip
FROM
INNER JOIN schools AS 12 ON T1CDDSCode T2.CDSCode
WHERE T1. County Name's = 'Monterey' AND Ft.'reee Meal Count (Ages 5-17)"
> 800 AND T1. School Type's "High "High Schools (Publio):.
and Conquer COT FROM Sohool
city',
"1''
INNER JOIN
ON CDSCode' CDSCode'
WHERE SOCType Schools (Public)' AND 'T2. County' 'Monterey' AND FRPM Count (Ages 5-17) 800
Figure 21: An example of query that has been solved only with the query plan CoT method.
25

---

Published as a conference paper at ICLR 2025
A.10 QUERY FIXING PROMPT
In this section, we provide the prompt template for the SQL query fixing step.
* **Task Description:**
You are an SQL database expert tasked with correcting SQL query. A previous attempt to run query did not yield the correct results, either due to errors in execution or because the result returned was empty or unexpected. Your role is to analyze the error based on the provided database schema and the details of the failed execution, and then provide a corrected version of the SQL query.
** *Procedure:**
Review Database Schema:
Examine the table creation statements understand the database structure.
Analyze Query Requirements:
Original Question: Consider what information the query is supposed retrieve.
Hint: Use the provided hints to understand the relationships and conditions relevant to the
query.
Executed SQL Query: Review the SQL query that was previously executed and led to an error or incorrect result.
Execution Result: Analyze the outcome of the executed query to identify why it failed (e.g.. syntax errors, incorrect column references, logical mistakes).
3. Correct the Query:
Modify the SQL query to address the identified issues, ensuring correctly fetches the requested data according the database schema and query requirements.
**Output *
Present your corrected query as single line of SQL code, after Final Answer. Ensure there are no line breaks within the query.
Here are some examples: (EXAMPLES)
Your task
***8***** & $833838888888
Table creation statements [DATABASE SCHEMA)
#** wtw..att.t...................... #e
The original question is:
Question:
[QUESTION]
Evidence:
[HINT}
The SQL query executed was:
(QUERY)
The execution result:
[RESULT] b
Based the question, table schema and the previous query, analyze the result try to fix the query.
Figure 22: The prompt template used for query fixing
26

---

Published as a conference paper at ICLR 2025
A.11 SELECTION AGENT PROMPT
In this section, we provide the prompt template used for training and query picking at test time by the trained selection agent. Note that the database schema used in this step is the union of the columns and tables by the two candidates instead of using the full-schema of all tables in the database.
Instruction:
Given the DB info and question, there are two candidate queries. There is correct one and incorrect one, compare the two candidate answers, analyze the differences the query and the the original question and the provided database info, choose the correct one.
result. Based
#88 is ------------------- is a * * w ***.*****.000. *******
Database Schema [DATABASE SCHEMA]
# **
Question:
[QUESTION]
Evidence:
(HINT)
Candidate
(CANDIDATE.A.QUU))
Execution result
(CANDIDATE.A.RESUIT
Candidate B [CANDIDATE_B B.QURRY
Execution result
BANNDDDAE...RESUUCE
Just output the correct answer "A" or "B".
Figure 23: The prompt template used for query fixing
27

---

Published as a conference paper at ICLR 2025
A.12 GENERATED SYNTHETIC EXAMPLES ANALYSIS
Question: What is the highest oligible free rate for K-12 students the achools Alameda County?
SQL:
(K-12)' /Enrollment (X-12))
from
WHERE
''Alamede'
ORDER BY (CAST( Meal Count (K-12) REAL) Ennollment (K-12))
DESC LIMIT
Common SQL feature example:
Input: What the total enroilment of schools in distriots with an avorage
SAT score above 600?
Enroliment (K-12)) satscores
JOIN frpm
T2.CDSCode WHERE T1.AvgScrMath 600
Simple exemple with filtered columns:
input: What the average 'Enrollment (K-12) for sohools in Alameda County?
output: SELECT A'G('Enrlllment (K-12)) FROM frpm WHERE 'oounty Name 'Alameda'
Which school Alameda County
Meal
(K-12)"?
output: SELECT Sohool FROM sahools INNER JOIN 11.CDSCode T2,CDSCode WHERE T1.County = 'Alameda' ORDER BY T2. Free Meal Count (K-12) DESC LIMIT
ON
Synthetic examples generated for the celifornia,schooke database question with different guide- lines common SQL features and filtered columns.
Figure 24:
Table 8: Ablation studies on synthetic example generation guidelines, Ry with common SQL features and Rs with filtered schema. The baseline is the original BIRD prompt shot CoT with Gemini 1.5 pro on the BIRD dev set. Total of 75 examples generated for each example set (Ry, R2) and for the mixed (Rş R,)
Method
Execution Accuracy (%)
A(%)
Baseline (Zero-shot)
57.75
OS w/ R, OS w/ R
65,45
+7.7
66.75
+9.0
OS w/ RG + R.
67.09
+9.34
Table illustrates the ablation studies done with different guidelines and their generated example sets. Compared the baseline (no example), the question and its associated data schema targetted synthetic examples can help; we try to promote the diversity of the examples to avoid over- fitting the output certain patterns (e.g., the model always writes SQL with JOIN if shown mostly JOIN examples).
Figure 25: Distribution (normalized) of synthetic vs. ground truth examples in different SQL features/clauses. All examples are generated using gemini-l.5-pre for the questions and schemas from the BIRD-Bas BeDBBB dev datasct.
Oreundvatt
0.30
Synthetic examples
0.25
0.20
0.00
080
zar por
curer Lerr movir
Loster
(a) Our synthetic examples with explicit guidelines,
(b) Synthetic examples following cross domain data augmentation strategy (Li ct 2024b)
Ry and Rs
Fig. 25a shows the SQL feature distribution of the generated synthetic examples for the BIRD dev
dataset, which closely follows the actual SQL. features distribution, except CASE statement. We
28

---

Published as a conference paper at ICLR 2025
omit CASE statement examples, since showing examples with CASE statement did not help with the generation, unless the ground-trum SQL query actually used it.
Example quality comparison between our synthetic examples (OS) and other example sets, prepared by two other approaches: relevant training examples by question similarity (o((tain)) and a different synthetic example generation strategy (CodeS). To isolate the impact of example quality, we only generated single output candidate without any self-correction with varying numbers of examples
Table 9:
5
25
75
125
(train)
58.80
58.54
57.69
56.91
CodeS
60.76
60.63
59.97
59.06
OS
62.13
64.02
64.41
63.69
demonstteesssssees how the online synthetic example generation (OS) yields more useful example set for in-context learning, compared to two other approaches. Training dataset is commonly used source of few-shot examples, where examples are selected by question similarity (o(train)). Data augmentation is another technique used for cross-domain adaptation, where either extra fine-tuning data or few-shot examples are synthesized. The proposed technique in (Li al.) 2024b) uses two- step example generation, where the model is 1) asked to come up with questions to ask given the schema; 2) asked to fill in the blanks of example templates with the schema elements. There is no specific guidelines to the example and SQI structures for the first step, and the second step uses set of universal question/SQL templates with limited complexity (e.g., only single table, non-nested queries with up to 3 columns). The resulting example SQL feature distribution shown in Fig. 25b
29

---

Published as a conference paper at ICLR 2025
A.13 SYNTHETIC EXAMPLE GENERATION PROMPTS
In this section we provided the prompt template for the online synthetic example generation step.
You are a SQLite SQL expert. Your job is to create {k} examples, where each example consists of question and SQL query to fetch the data for it. I want each example to look like this, question input and SQL output pairs:
"input": "What's the description of the series code SM.POP.TOTL for Aruba? (Hints: Aruba is the name of the country where ShortName 'Aruba')"
"output": "SELECT T2.Descritiion FROM Country AS TI INNER JOIN CountryNotes T2 ON To.CountryCode T2. C2.nCuntreeode WHERE TT..ShotNaame 'Aruba" AND 12.reeisee "SM.POP.TOT.."
You should generate examples that examine and showcase different aspects and relation- ships of the following table schemas, described in "Table creation statements". Understand the database tables and their relationships. Understand the columns and their types and meanings to construct intresting examples.
Generate mixture of SQL examples that include:
some simple SQL query examples without JOIN
some SQL query examples with aggregates, like COUNT
some simple SQL query examples with JOIN
some complex SQL query examples with nested JOIN
###Table creation statementstifff
[TARGET. DATABSSS
Generate total {k} examples. Only outputs the examples (question input and SQL output pairs), and each example be separated by new line.
Figure 26: Synthetic example generation prompt used for common SQL features examples genera- tion. DATABASSDT contains all the tables from the target database.
30

---

Published as a conference paper at ICLR 2025
You SQLite SQL expert. Your job is to create set of examples, where each example consists of question and a SQL query to fetch the data for it. You should generate examples that examine and showcase different aspects and relationships of the following table schemas. Understand the database tables and their relationships. Understand the columns and their types and meanings construct intresting examples. I will also show you multiple examples generated for the other database and its table schemas, so you can see what kind of examples can be generated for given database.
###Examples from other databasetitt# The following is the table schemas and column examples for other database:
The database ((TRAIN DATABASE NAMEJ) structure is defined by the following table schemas (comments after provide additional column descriptions).
(TRAIN PATABASE.SCHEMA
The folloiwing are the examples generated for the above database schemas: Example 1) "input": " "Among the countries in the group Heavily Indebted Poor Countries, how many of them are under the lending category of the International Development Associations?
(Hints: group of Heavily Indebted Poor Countries is OtherGroups = 'HIPC"; International Development Associations refers to lendingcategory 'IDA")"
"output": "SELECT COUNT(CountryCote) FROM Country WHERE LendingCategory 'IDA' AND OtherGroups = "HIPC""
Example 10) "input": "What is the description of the footnote on the series code AG.LND.FRST.R. in 1990 for Aruba? (Hints: Year = 1990; Aruba is the name of country where ShortName 'Aruba')"
"output": "SELECT T2. FROM Country AS TI INNER JOIN FootNotes AS T2 ON T1. Co.CCuntryCode = T2, T2. T22.Contrycode WHERE T1 Shorttmme 'Aruba' AND T2.Seriescode 'AG.LND.FRST.22 AND T2. Year "YR1990""
Now similarly, generate examples (question input and SQL output pairs) for the table schemas defined below, in "Table creation statements".
*** ***4988
###leble creation statements#ifff
TARGET DATABASE SCHEMA
* *ewe#eiio*eeo**eee
Only outputs the examples (question input and SQL output pairs), and each example can be separated by a new line.
Synthetic example generation prompt. This is use TARGET DATABASE SCHEMA filtered with column selection result, and the model asked to generate simple examples similar to the ones taken from the training dataset (separate from the test or dev dataset).
Figure 27:
31


Reviews:
[{"review_id": "ApLSl1nFbv", "reviewer": "Reviewer_xZFJ", "summary": "The article discusses the formation of a pipeline for text2sql, which includes the following main parts: combining multiple methods (including DC-CoT, Query Plan-CoT and OS-CoT) to generate a set of candidate sql, with each sql query running through a fixer to fix the query so that it can be run, and finally, comparing each pair of queries and ranking to choose the most suitable query. The pipeline is implemented on close-source LLMs and is carefully promoted and evaluated to come up with a good method when testing multi-agent LLM for the text2sql task. The method reaches an execution accuracy of 73.01% and 73.0% on the development set and test set of the challenging BIRD Text-to SQL dataset which outperforms all of the published and undisclosed methods on this benchmark by a large margin.\nContribution:\n+ Experiment multi method (include few-shot, CoT) to generate candidate SQL query from natural language text.\n+ Develop evaluator agent to score every candidate and ranking for choose the best candidate.\n+ Experiment combination and ablation many method choose to the effective text2sql pipeline.\n+ Adapt Multi-agent LLM approach.", "questions": "+ How about experimenting with the above pipeline for an open-source LLM, is it possible?\n+ Can you give some examples of selecting rows to match the schema and user query?\n+ Why schema union is so effective in evaluating your query pairs.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Proposed a pipeline that provides high efficiency on text2sql data and has proven its efficiency and high results on difficult benchmarks such as Bird and Spider.\n+ There is a thorough review and explanation, making it easy to combine and remove components throughout the system.\n+ The paper is presented fully and in detail, with thorough explanations of mathematical formulas, complete conclusions and details in the evaluation and testing.", "weaknesses": "+ CoT, Few Shot or query fixer methods are quite classic in testing LLM, so they have not mentioned the highlights in generating this query.\n+ Beside the method uses available closed source models such as claude, gemini pro mainly to perform prompting and connect agents together, it is not possible to verify whether it is good to fine-tune open source for the above methods. The article would be better if there was more in-depth research on fine-tuning these models, which could be a useful scientific research and could be more easily deployed in practice. Overall, I rate the article very well in terms of testing and conclusions drawn.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zAvZzV1A0U", "reviewer": "Reviewer_2iLv", "summary": "This paper introduces the CHASE-SQL framework, a novel approach for improving text-to-SQL task with LLMs. The framework proposes multi-path reasoning techniques that decompose complex queries and optimize candidate generation for SQL, which involves three main strategies: a divide-and-conquer approach for breaking down queries, a CoT reasoning based on query execution plans, and an instance-aware synthetic example generation to enhance LLM performance. To select the best SQL candidate, a fine-tuned selection agent ranks generated queries through pairwise comparisons, achieving high accuracy. Extensive experiments have validated the effectiveness of the proposed framework. CHASE-SQL demonstrates SOTA performance on the well-recognized BIRD benchmark, achieving an execution accuracy of 73%, making it the top place on the leaderboard.", "questions": "1. As a reviewer, I highly appreciate the detailed error analysis in this paper. However, I suggest that the authors include some parts of the error analysis in the main content instead of the Appendix, as it is gaining increasing attention from the text-to-SQL community.\n\n2. As shown in Appendix A.3, why is the number of correct samples for some databases zero across various methods? Could this result be combined with the difficulty level of the corresponding questions for further analysis?\n\n3. For the prompt used in the Query Fixer module, there are few-shot examples provided, as shown in Appendix A.7. What type of execution results do the examples represent? How does the performance of this module vary across different types of execution results (errors)? As an assumption, could including a diverse range of few-shot examples that cover as many types of execution errors (e.g., Column Not Found, Data Type Mismatch) enhance the correcting capability of the Query Fixer module?\n\n4. The authors could consider using a different abbreviation instead of \"CHASE\" since there is a related work in the text-to-SQL community with a similar name [7].\n\n[7] Jiaqi Guo, et al. \"Chase: A Large-Scale and Pragmatic Chinese Dataset for Cross-Database Context-Dependent Text-to-SQL\" In Proceedings of ACL, 2021.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The performance of the proposed CHASE-SQL framework is highly promising. It achieves SOTA accuracy on the well-recognized BIRD benchmark, outperforming both published and undisclosed methods, thereby demonstrating its effectiveness in complex text-to-SQL tasks.\n\n2. The experiment of the paper is sufficient, and the error analysis section is detailed and highly informative. The authors provide comprehensive analyses, including performance across different databases, error analysis, and selection agent errors, offering clear insights into the model's strengths and areas for improvement. Notably, the error analysis section is outstanding and is garnering increasing attention within the text-to-SQL community, as it helps readers understand not only how the framework works but also why it works.\n\n3. The overall framework is novel, and the paper is well-organized and easy to understand. The structure effectively presents the methodology, experiments, and results, making it accessible for readers to comprehend the contributions and significance of the proposed approach.", "weaknesses": "1. The paper does not include the cost of the proposed framework. Prompting proprietary LLMs for SQL generation has become a mainstream approach in text-to-SQL research. When the performance of various methods shows no significant differences, the method with lower API costs is typically preferred. Previous work has focused more on models released by OpenAI (e.g., ChatGPT, GPT-4) [1]. Since the Gemini series is also an outstanding proprietary LLM, this paper presents a good opportunity to introduce the series to the community by comparing its performance and API costs.\n\n2. The model ablation study is lacking in the paper, and the selection of open-source models has not been discussed. The paper would benefit from verifying the proposed framework on a broader range of models, including GPT series models (e.g., ChatGPT, GPT-4) and open-source models (e.g., LLaMA-3.1, Qwen-2.5). Recently, well-designed frameworks for open-source models have achieved promising progress [2][3], demonstrating particular effectiveness for local deployment and real-world applications. Therefore, experiments on open-source models in this paper could further advance this development.\n\n3. The description of the Query Fixer module is relatively brief. The authors could consider adding a detailed algorithm for correcting incorrect SQL in Section 3.4. Additionally, I suggest including a separate limitations section to discuss the potential challenges for application, such as the framework's complexity, generalization and extension capabilities [4], and the potential limitations in handling ambiguous questions [5][6].\n\n[1] BIRD Leaderboard. https://bird-bench.github.io/\n\n[2] Haoyang Li, et al. \"CodeS: Towards Building Open-source Language Models for Text-to-SQL\" In Proceedings of SIGMOD, 2024.\n\n[3] Mohammadreza Pourreza, et al. \"DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models\" arXiv preprint, 2024.\n\n[4] Zijin Hong, et al. \"Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL\" arXiv preprint, 2024.\n\n[5] Yujian Gan, et al. \"Towards Robustness of Text-to-SQL Models against Synonym Substitution\" In Proceedings of ACL, 2021.\n\n[6] Xiang Deng, et al. \"Structure-Grounded Pretraining for Text-to-SQL\" In Proceedings of NAACL, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oo6YKBYLO", "reviewer": "Reviewer_s8pW", "summary": "The paper introduces CHASE-SQL, a state-of-the-art Text-to-SQL approach that achieved top performance on the BIRD benchmark at the time of submission. It employs three prompting strategies — Divide-and-Conquer, Query Plan, and Few-Shot prompting with synthetic example generation — to first generate a diverse set of SQL candidates. These candidates are then evaluated by a selector, which is fine-tuned on BIRD to select the correct SQL from two SQLs. The selector does pair-wise comparison over the pool of candidates and outputs the one having the highest score.", "questions": "1. How many examples are included in the few-shot prompting (online synthetic example generation)?\n2. What is the total number of tokens processed if I use CHASE-SQL to generate the SQL for a single question?\n3, What is the end-to-end latency of CHASE-SQL?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and provide a lot of details and insights for readers to learn about. The evaluation is solid with comprehensive comparison, lower/upper bound analysis and ablation studies. The design of all components within CHASE-SQL are well-justified.\n\n2. Although the pipeline of generating a candidate pool and then selecting from it is not new, CHASE-SQL introduces novel strategies for candidate generation and fine-tunes the selector in a unique way, focusing on simple tasks, i.e., pairwise selection, rather than relying on reranking or selecting from a large pool. I particularly appreciate the approach used to construct the few-shot examples. Instead of tailoring examples to specific SQL types, it includes examples for both the full database and the specific database relevant to the question. Moreover, it aims to cover a broad range of SQL features rather than only complex examples, which significantly reduces the risk of the model overfitting to the provided examples.\n\n3. By using examples generated on BIRD and fine-tuning the selector solely on BIRD, CHASE-SQL also achieves competitive performance on the Spider benchmark, demonstrating its generalizability.", "weaknesses": "The main concern is the cost and latency of CHASE-SQL. Assuming each generator produces 7 candidates, this results in 21 LLM calls. The prompt length for each call is also substantial, especially with the few-shot prompting strategy, which includes examples for both the full database and the specific database, aiming to cover a wide range of SQL features. For the selector, CHASE-SQL employs a pairwise comparison strategy, leading to O(n^2) LLM calls, where n is the total number of generated candidates. It would be helpful if the authors reported the total number of tokens processed by CHASE-SQL to generate SQL for a user query and provided the end-to-end latency. Given these factors, I am uncertain whether CHASE-SQL can achieve interactive Text-to-SQL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ssQ7hAAFMQ", "reviewer": "Reviewer_qs8r", "summary": "This paper introduces Chase-SQL, which combines SQL queries generated by various LLM strategies to enhance query quality for natural language questions. It utilizes diverse approaches, including divide-and-conquer, few-shot learning with demonstration synthesis, and self-debugging. Leveraging Gemini Pro 1.5, Chase-SQL sets a new state-of-the-art performance on two prominent benchmarks, BIRD and Spider.", "questions": "Q1: The paper's novelty requires clarification, particularly in light of the concerns raised regarding the originality of its individual algorithms (see weaknesses W1, W2, and W3).\n\nQ2: The technical contributions and key innovations compared to existing studies need to be clearly justified (refer to weaknesses W4 through W6).\n\nQ3: The experimental design needs to be improved to more effectively verify the authors' claims (see weakness W8).", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is its robust framework, which effectively integrates multiple LLM strategies to optimize SQL query generation, leading to state-of-the-art results. This framework not only demonstrates the versatility of techniques such as divide-and-conquer and self-debugging but also highlights the potential for real-world applications in natural language processing. It achieves superior performance on well-established benchmarks like BIRD and Spider.", "weaknesses": "The main weaknesses of the paper stem from the limited novelty of the framework and its individual techniques. A recent comprehensive survey on NL2SQL, available at https://arxiv.org/pdf/2408.05109, addresses many of the technical innovations discussed here, encompassing a wide range of recent studies that leverage LLMs. This suggests that the contributions may not be as groundbreaking as implied, as they largely reiterate established concepts in the field. \n\nFirst, let’s discuss the individual algorithms.\n\nW1. Divide-and-Conquer: This concept has been widely utilized in various NL2SQL studies, including DTS-SQL, TKK, and DEA-SQL, as illustrated in Figure 6 of the NL2SQL survey under the \"Decomposition\" branch. The papers of all the referred methods can be found in the survey. In order to highlight the technical contributions, it would be helpful to clarify how the proposed divide-and-conquer approach differs from or improves upon these existing methods. \n\nW2. Chain-of-Thought: The Chain-of-Thought (CoT) approach has also been extensively applied in NL2SQL research, with examples such as CHESS, ACT-SQL, and COE-SQL, which can be found in Figure 6 of the NL2SQL survey under the \"Chain-of-Thought\" branch. \n\nW3. Instance-Aware Synthetic Example Generation: The authors assert that they introduce a \"unique instance-aware synthetic example generation.\" However, the use of few-shot examples has already been explored in DIN-SQL and CodeS. Additionally, rather than synthesizing examples, it is often more straightforward to select existing examples from the training data. This raises the question: what advantages does synthesis offer over selection? Please provide empirical evidence or theoretical justification for why the proposed synthetic example generation approach outperforms or differs from simply selecting examples from training data.\n\n- DIN-SQL: https://arxiv.org/abs/2304.11015\n- CodeS: https://arxiv.org/abs/2402.16347\n\nNext, let’s explore weaknesses from other aspects.\n\nW4. Method Ensemble: The paper claims that ensembling multiple generated SQL paths is beneficial. However, it raises the question of why not include even more methods by incorporating readily available off-the-shelf solutions.\n\nW5. Method Ensemble vs. Module Combination: In the reference titled \"The Dawn of Natural Language to SQL: Are We Fully Ready?\", a method for NL2SQL automated architecture search is discussed, which explores a predefined design space of NL2SQL solutions. This prompts a consideration of the pros and cons of coarse-grained method ensembles compared to fine-grained module ensembles.\n\nW6. Self-Consistency and Self-Reflection Methods: The authors propose self-consistency and self-reflection methods, yet both approaches have been extensively studied in the context of NL2SQL. Similarly, value retrieval and candidate generation techniques have been addressed in previous research, as noted in Table I of the aforementioned survey.\n\nW7. Algorithm 3 Execution Comparison: In Algorithm 3, the need to \"compare both (ci, cj) and (cj, ci)\" raises questions. Why is it necessary to evaluate execution results in both directions? Please provide a specific example or explanation of how comparing in both directions impacts the results or addresses potential biases in the selection process.\n\nW8. Experiments: The experimental results lack clarity regarding whether they stem from the capabilities of Gemini or the proposed framework. (1) If the benefits arise from the framework, the experiments should involve substituting Gemini with various LLMs to assess whether the framework enhances performance across different models. (2) The impact of ensembling different methods remains uncertain. (3) It is unclear how the proposed techniques individually compare to existing methods in areas such as value retrieval, candidate generation, and query fixing. Hence, the following experiments are important to enhance the experimental section:\n\n- Conduct ablation studies comparing their framework with different LLMs.\n- Provide a detailed analysis of the impact of ensembling different methods.\n- Include comparisons of individual components (value retrieval, candidate generation, query fixing) against existing state-of-the-art methods for each task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)