ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL
Authors: Yang Qin
Chao Chen
Zhihang Fu
Ze Chen
Dezhong Peng
Peng Hu
Jieping Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-SQL
LLMs
Primary area: foundation or frontier models, including LLMs
TL;DR: A novel multitask collaborative solution for Text-to-SQL via LLMs.

Abstract: Despite the significant advancements in Text-to-SQL (Text2SQL) facilitated by large language models (LLMs), the latest state-of-the-art techniques are still trapped in the in-context learning of closed-source LLMs (e.g., GPT-4), which limits their applicability in open scenarios. 
To address this challenge, we propose a novel RObust mUltitask Tuning and collaboration mEthod (ROUTE) to improve the comprehensive capabilities of open-source LLMs for Text2SQL, thereby providing a more practical solution.  Our approach begins with multi-task supervised fine-tuning (SFT) using various synthetic training data related to SQL generation.  Unlike existing SFT-based  Text2SQL methods, we introduced several additional SFT tasks, including schema linking, noise correction, and continuation writing.  Engaging in a variety of SQL generation tasks enhances the model's understanding of SQL syntax and improves its ability to generate high-quality SQL queries. Additionally, inspired by the collaborative modes of LLM agents, we introduce a Multitask Collaboration Prompting (MCP) strategy.  This strategy leverages collaboration across several SQL-related tasks to reduce hallucinations during SQL generation, thereby maximizing the potential of enhancing Text2SQL performance through explicit multitask capabilities. Extensive experiments and in-depth analyses have been performed on eight open-source LLMs and five widely-used benchmarks. The results demonstrate that our proposal outperforms the latest Text2SQL methods and yields leading performance.

Published as a conference paper at ICLR 2025
ROUTE: ROBUST MULTITASK TUNING AND COLLAB- ORATION FOR TEXT-TO-SQL
Yang Qin' Chao Chen Zhihang Fu² Ze Chen Dezhong Peng Peng Hu
Jieping Ye²
Sichuan University, ?Independent Researcher, TTinfu Jincheng Laboratory
ABSTRACT
Despite the significant advancements in Text-to-SQL (Text2SQL) facilitated by large language models (LLMss, the latest state-of-the-art techniques still trapped in the in-context learning of closed-source LLMS (e.g.. GPT-4), which limits their applicability in open scenarios. To address this challenge, we propose a novel RObust m Ultitaakk Tuning and collaboration mEthod (ROUTE) to improve the comprehensive capabilities of open-source LLMs for Tex12SQL, thereby pro- viding a more practical solution. Our approach begins with multi-task supervised fine-tuning (SFT) using various synthetic training data related SQL genera- tion. Unlike existing SFT-based Text2SQL. methods, we introduced several ad- ditional SFT tasks, including schema linking, noise correction, and continuation writing. Engaging in variety of SQL generation tasks enhances the model's s understanding of SQL syntax and improves its ability to generate high-quality SQL queries. Additionally, inspired by the collaborative modes of LLM agents, we introduce a Multitask Collaboration Prompting (MCP) strategy. This strat- egy leverages collaboration across several SQL-related tasks reduce halluci- nations during SQL generation, thereby maximizing the potential of enhancing Text2SQL performance through explicit multitask capabilities. Extensive exper- iments and in-depth analyses have been performed on eight open-source LLMs and five widely-used benchmarks. The results demonstrate that our proposal out- performs the latest Tex12SQL methods and yields promising performance. The code and data are available at here.
INTRODUCTION
Text2SQL has emerged as popular and practical technology for question answering based on large- scale databases, serving as crucial link between natural language and database systems (Zhang et al., 2024). Recently, Large Language Models (LLMs) have proven to be effective solu- tion in Text2SQL (Pourreza & Rafiei, 2024a). Unlike pioneer works Katsogiannis-MMimmae & Koutrika, 2021; Xiao et al., 2016; Bogin et al., 2019; Li et al., 2023a;b; Gu et al., 2023), LLM- based methods (Pourreza & Rafici, 2024a; Gao et al., 2024a; Wang et al., 2023) primarily develop effective prompt engineering techniques, e.g., in-context learning, to motivate LLMs to understand the database schema and generate accurate SQL query. However, accurately aligning entities in natural language questions and databases for SQL generation remains challenging, especially when dealing with complex database schema or semantically complex questions (Li et al., 2024c).
To address these challenging scenarios, recent efforts (Lee et al., 2024; Talaei al., 2024; Li al., 2024d) have developed various pipelines that enhance the entire SQL generation process and reduce potential error risks. These improvements include techniques such as Schema Linking (Pourreza & Rafici, 2024a), Self-correction (Wang et al., 2023), Chain-of-Thoow ability Voting (Li et al., 2024d), etc. Although these methods have achieved promising results, they often rely on closed-souse source models (e.g.. GPT-4/4o (Achiam et al., 2023)), which can raise potential privacy risks and incur significant overheads when deploying LLMs practical scenarios. More- over, while these techniques perform well on GPT-4 or other large-sized LLMs, their effectiveness
(CoT) (Tai et al., 2023), Reli-
Coorreponning authors.

---

Published as a conference paper at ICLR 2025
may diminish when applied smaller open-source LLMs (refer to Table 1). This is due to the lim- ited capacity of smaller LLMs to understand complex instructions, showing lower generalizability.
An alternative involves transforming general LLM into specialized LLM by injecting Tex12SQL- related knowledge through pre-training or SFT (Li et al., 2024b; Yang et al., 2024b; Gu et al., 2023; Roziere et al., 2023). However, most training- based methods only incorporate the SQL genera- tion task in the SFT stage, resulting in a degraded performance in other tasks that are important for Text2SQL capability, such as schema linking. Additionally, training LLMs on single SQL generation task poses a substantial risk of diminishing performance in understanding instructions, potentially reducing the model's effectiveness in other important SQL-related tasks beyond SQL generation (Appendix A.6). To avoid this dilemma, DTS-SQL separates the schema linking task from Text2SQL. and trains specialized L.MMs for each to simplify the process (Pourreza & Rafiei, 2024b). However, this approach requires deploying two separate L.Mss during the inference stage, which introduces additional overhead and is impractical in real-world applications.
Considering the limitations of existing prompting training methods, we propose robust multitask tuning and collaborative prompting framework (ROUTE) for Text2SQL generation. The motivation is intuitive: (1) Multitask training not only enhances the model's SQL generation capabilities but also preserves other abilities such as schema linking. (2) Training LLM in various SQL-related tasks, such schema linking and noise correction, is expected to enhance the model's 's understanding of SQL syntax. (3) By employing multitask collaborative prompting, the complex Text2SQL task can be decomposed into several simpler sub-tasks, thereby enhancing the accuracy of Text2SQL.
To achieve this goal, in the training stage, we explore a Multitask Supervised Fine-Tuning paradigm (MSFT), which endows the LLM with the capabilities in Tex12SQL (TS), Schema Linking (SL), Noise Correction (NC) and Continuation Writing (CW). In the inference stage, we develop a Mul- titask Collaboration Prompting (MCP) approach to selectively and incrementally generate the SQL queries, potentially reducing the risk of ballucination in complex SQL generation.
Our contributions can be summarized as follows:
We introduce multitask SFT training framework, which equips the LLM with variety of SQL-related specialized capabilities.
We propose a multitask collaborative prompting strategy that enables the decomposition of the Text2SQL task into several simpler sub-tasks, leveraging specialized capabilities of LLMs.
We conduct a thorough evaluation of recent open-source LLMs and perform extensive exper- iments to demonstrate the effectiveness of multitask SFT and to showcase the generalization capabilities of collaborative prompting.
2 RELATED WORK
Text-to-SQU SQL (Li al., 2023b; Gao et al., 2024a; Pourreza & Rafiei, 2024a) aims to understand user intent and convert natural language questions into SQL queries. Existing Text2SQL methods can be roughly categorized into Pre-LLM and L.MM-based methods. Pre-ILMM approaches mainly exploit the rule modeling (Katsogiannis-Mer Meimnnnisi & Koutrika, 2021), specialized neural networks (Xiao al., 2016; Bogin et al., 2019) and pre-trained models (Fu et al., 2023; Li al., 2023a;b; Gu et al., 2023) to parse and improve SQL generation. The latter has made significant progress recently thanks LLM's unique emergent abilities in developing Text2SQL solutions, including prompt engineering (Rajkumar& et al., 2022; Gao et al., 2024a; Pourreza et al., 2024; Maamari et al., 2024) and LLM iine-tunigg/pre-treeee training (Li et al., 2024b; Pourreza & Rafiei, 2024b; Yang al., 2024b). In this paper, we mainly focus on the M-beed methods.
Prompt Engineering. To unleash the potential of LLMs in the Text2SQL task, a straightforward attempt is to design effective prompting techniques (Liu et al., 2023; Pourreza & Rafiei, 2024a; Gao et al., 2024a; Wang et al., 2023; Lee et al., 2024; Gao et al., 2024b) to guide LLMs. Recent approaches mainly focus on leveraging closed-source models (e.g.. ChatGPT and GPT-4) to de- sign innovative instructions or pipelines through the techniques of ch (Zhang et al.), LLMs' agents (Wang al., 2023), question/task decomposition (Pourreza & Rafiei, 2024a), self- debugging (Wang et al., 2023; Chen et al., 2023), schema linking (Li et al., 2024b), few-shot ex- ample selection (Gao et al., 2024a), etc. For example, DIN-SQL. (Pourreza & Rafiei, 2024a) adopts
2

---

Published as a conference paper at ICLR 2025
Input: Database d & Question y
Step3: Continuation Writing
SFT
Text-to-SQL
Step1: Schema Linking Text2SQL
d
SELBCT
Schema Linking
FROM A
SPIDER BIRD
Single-task SFT
SQL (s"
SELECT
Eq2
FROM A T1
Text-to Text-to-Sol to-SQL
Noise Filtering Data Synthesis
B4B
Noise Correction
Schema Linking
Continu. Writing
Step2: Noise Correction
NC
False
hard?
Yes
JOIN BT2 ON XXX
JOIN C T3 ON XOX
No
Multitask SFT
+
True s
s"
Final SQL 22
(a) Multitask Supervised Fine- Tuning
(b) Multitask Collaboration Prompting
Figure 1: Overall framework of our ROUTE. Our approach consists of two core stages, i.e., Multi- task Supervised Fine-tuning (MSFT) and Multitask Collaboration Prompting (MCP). Different from existing methods that only focus on unitask learning in a single LLM, our MSFT aims empower LLMs to handle multiple SQL-specific tasks by utilizing synthetic data for supervised fine-tuning. MCP mainly leverages the capabilities of individual tasks for given database and question col- laboratively generate accurate SQL queries. It enhances the final SQL query incrementally through a shree-step process that leverages the multitasking capabilities of LLMs. Note that TS the abbre- viation of Text2SQL, and the task definitions of SL, NC, and CW can be found in Section 3.1.
multiple steps to reduce the complexity of the task for an accurate SQL query. MAC-SQL (Wang al., 2023) decomposes the original question into several sub-questions and refiner agents to check and correct the final SQL generation. However, these approaches come with huge inference over- head, and may no longer be feasible for small-sized LLMs. In contrast, our method can effectively alleviate these problems through multi-task training and collaborative prompting.
Fine-tuning LLMs. Although prompting with source models such as GPT-4 has achieved promising performance, the cost of inference and concerns about data privacy make it particularly urgent develop specialized LLMs for Text2SQL (Li al., 2024b; Pourreza & Rafiei, 2024b; Yang et al., 2024b). To break such limitations, CODES (Li et al., 2024b) exploits SQL-related corpus for incremental pre-training with StarCoder (Li et al.) and designs some plug-and-play utils to enhance Texi2SQL performance on several benchmarks. DTS-SQL. (Pourreza & Rafiei, 2024b) performs supervised fine-tuning (SFT) with a specialized model for schema linking to reduce complexity and improve performance. Recently, SENSE (Yang et al., 2024b) enhanced the preference of LLMs for correct SQL answers by synthesizing strong data and performing Direct Preference Optimization (DPO) (Rafailov et al., 2024) on weak data from weak LLMs. In this paper, we propose to aggregate multiple tasks to enhance the comprehensive SQL-related capabilities and then employ multitask collaboration minimize potential risks in schema linking errors or SQL clause errors.
METHODOLOGY
In this section, we will delve into our proposed RObust multitask Tuning mEthod (ROUTE), de- signed to enhance SQL generation capabilities. As shown in Figure 1, ROUTE consists of two core stages, iee, Multitask Supervised Fine-Tuning (MSFT) and Multitask Collaboration Prompting (MCP) for SQL Generation. In the following, first show the notations and definitions used in this work in Section 3.1, then introduce the details of our ROUTE in Sections 3.2 and 3.3 for Tex12SQL.
3.1 NOTIONS AND PROBLEM FORMULATION
Given an instructional Text2SQL dataset D ((d,, qs > where d; is SQL database, qi is a natural language question that may be accompanied by a question hint, and is a ground-truth SQL query, the purpose of Text2SQL is to exploit an LLM(M) to generate a SQL query sf based on a prompt constructed by d, and g. and the execution results of predicted SQL query are consistent with those of the ground-truth SQL query 8,. In this paper, in addition to the standard Tex12SQL task, approach incorporates three additional SQL-related tasks in the SFT and SQL generation stages, including schema linking, noise correction, and continuation writing, as shown in Figure 2. All these tasks are defined follows:
3

---

Published as a conference paper at ICLR 2025
Database (d)
Question (e)
Text-to- Soe-tt--SSS0 So-L
Schemna Linking
Tables:
CREATE TABLE mavies(--)::
What the average number of
Question:
(d. o)
H LLMS
SQL (5-))
movess pww.lantpyyyyyyyo drrector name
Simplified (4)
CREATE TABLE users--i
Musi users
mavies
5, (d.
CREATE TABLE ratings(-)::
directed by Staniey Kabnick?
Rows:
Question Hint:
Noise Correction
Continu. Writing
"movies:: [row]], row12, --]
everage AVG (movie_ eepoppl- arity): number of Mubi users who loves the movie refers to movie_pope provlepoopulartty
True No operation
og ogdd.g.5
'userss: [row21, row22, -"]
SILECT AuGammoo.eee Aew
tratings: [row33, row32, m]]
False SQL (5)
SULDCT ArGaronee pepplarryll RDM
Illustration of all SQL-related tasks involved in our ROUTE. These tasks are based on given database and user question. We form the prompt for each task by extracting a schema (table) description, a few rows of examples from the database, and the given question (possibly with a hint).
Figure 2:
(1) Text-to-SQL. (Text2QQL, TS) aims to generate a SQL query (sā)) based on a prompt constructed by the database (d,) and question (gt) and the execution results of S consistent with those of its ground-truth SQL. query (s²). The function of prompt formatting based on the database and question is represented as or (dc,)))
(2) Schema Linking (SL. aims to identify relevant tables and columns in the database (d:) for the given question (q:), thus avoiding verbose information in the prompt to reduce complexity, which has been proven to improve Text2SQL performance by recent studies (Pourreza & Rafiei, 2024b;a). The function of prompt formatting is represented as ax(di, 4s).
(3) Noise Correction (NC) is required to determine whether the execution results of the predicted SQL query (s²) can correctly answer the question (g:) based on the database d, If not, LLM (M) will be asked to provide a revised SQL query ($))) The function of prompt formatting is represented as on (d: qi, 57).
(4) Continuation Writing (CW) is another strategy to refine SQL. Given an incomplete SQL query (8:), LLM (M) is required to continue writing it into a complete and valid SQL query (5,) whose execution results can correctly answer the question (q)) Likewise, the function of prompt formatting can be represented (d qi, 8:).
From the task definitions provided, it is evident that the tasks of TS and NC have the potential to directly enhance the quality of the SQL queries, as they are directly related to SQL genera- tions. Empirical evidence from previous works (Pourreza & Rafiei, 2024a; Lee et al., 2024) has also demonstrated that SL can significantly reduce prompt complexity by simplifying database formation, thereby improving performance. In addition, we define new task named Continuation Writing (CW), which indirectly contributes to the final SQL generation. The motivation of Continu- ation Writing (CW) stems from the fact that LLMs possess inherent continuation capabilities, which make it easier than requiring the model to generate complete SQL queries. To unleash the potential of these tasks, in this paper, we first enhance the specialized capabilities of LLMs by aggregating multiple tasks for supervised fine-tuning. Then, we exploit the collaboration of multiple tasks to reduce potential risks in schema linking errors or SQL clause errors during SQL generation, thus further improving performance. Note that the prompt templates shown in Figure 2 for all the above tasks can be found in Appendix A.
3.2 MULTITASK SUPERVISED FINE-TUNING
Existing SFT-based methods (Li et al., 2024b; Yang et al., 2024b) mainly focus on the task of Text2SQL to improve final performance. However, previous prompting- based methods have shown that SQL-related tasks can also effectively improve performance. Unfortunately, due to the lack of specialized pre-training or instruction fine-tuning for these related tasks, is difficult for open- source LLMs to complete these tasks with high accuracy. In this section, present the details of our Multitask Supervised Fine-tuning (MSFT) on the dataset D that combines training sets from two wide-used cross-domain datasets, i.e., SPIDER (Yu al., 2018) and BIRD (Li et al., 2024c).
Noisy correspondence filtering. Recent empirical evidence (Wretblad et al., 2024; Wretblad & Gordh Riseby, 2024) indicates that even carefully annotated Text2SQL datasets exhibit semantic inconsistencies between the given question and the ground-trumt -tunt SQL query, as shown in Figure 3, which we call noisy correspondences and widely exist in various field (Qin et al., 2022; 2023; Sun
4

---

Published as a conference paper at ICLR 2025
et al., 2024; Qin et al., 2024). is widely recognized that noise remains primary factor contributing to hallucinations in existing LLMs. To mitigate this, we fine-tuned the selected LLM (Llama3-8B) to serve a noise discriminator, specifically designed to detect potential noise in the corpus. To con- struct corresponding SFT' data, we construct a positive discriminant example (an (di, li 8i), Apa)) and negative discriminant example (d,(di, 9i: s)); Ane (s+)) for each Text-SQL. pair, where 8; is the ground-truth SQL query and Apos/Ancg means the alfirmative/neeatiee answer, i.e. Apos: 'The execution results of the SQL query can correctly answer the question." ; Angg (s:): 'The execution results of the SQL query cannot correctly answer the question. The correct SQL query should be: {sa}'
To obtain negative SQL examples for the construction of Anyy we employ open-source LLMss Qwen2-7B and Llama3-8B to generate all SQL. responses for the questions in 'D in zero-shot manner and then compare the execu- tion results each predicted SQL query with those of the ground-truth SQL query. If they do not match, we view the predicted SQL query as negative example (s'). The goal is consis- tent with the defined task of NC shown in Sec- tion 3.1. In addition, to enrich the diversity of SFT data, we artificially and randomly intro- duce five types of errors in ground-truth SQL queries, including schema linking errors, nest- ing errors, GROUP BY errors, JOIN errors, and symbol errors. See Appendix A.7 for more details. Then, we fine-tune Llama3-8B model through a standard SFT process, and finally, we perform inference on D to identify and filter noise, achieving self- sulfipurificttion of the training data set. For convenience, we represent the purified data as DD
Example 1:
Q1: What is the number of inhabitants and income of geographic identifier 239?
A1: SELECT INHABITANTS FROM Demog WHERE GEOID 239; RI: SELECT INHABITANTS K. INCOME K FROM Demog WHERE
GEOID 239;
Example 2:
Q2: List the geographic id places where the income above average.
A2: SELECT AVG(NNCOME K) FROM Demog:
R2: SELECT GEOID FROM Demog WHERE INCOME (SELECT AVG(NCOME K) FROM Demog ):
Figure 3: training set. RI and R2 are the corrected SQL queries. More noisy examples can be found in Ap- pendix A.12.
The examples of noisy pairs in the BIRD
Synthesizing data for MSFT. Our MSFT consists of one major task (Tex12SQL) and three minor tasks defined in Section 3.1, namely SL, NC, and CW, to empower LLMs with specialized capabili- ties. To this end, we need to synthesize or construct SFT datasets from D for each specific task.
(1) For the SFT data of Text2SQL task, we utilize the defined function to construct the Prompt and the ground-truth SQL query serves the Response. The SFT data constructed for Tex12SQL is denoted as D.- = (d,, qz), 8; " where N, is the corresponding data size.
(2) For the SFT data of schema linking task, we utilize the ground-truth SQL query to exact the tables and columns the corresponding database as the Response, which is denoted as [(, d,) and f is the parsing function. If there is COUNT (*) statement, we only retain the primary keys of the corresponding table. We represent the dataset as D, (o,(di, q:), (s; d:) )
(3) For the SFT data of noise correction task, the synthesis and construction of the data are con- sistent with that of the dedicated discriminator mentioned above, including some positive exam- plcs and some corresponding negative examples. For convenience, we unify the expression as Dn = (ax (di, qi, si), Apos) ==1
N U Ua qi, s(), Aneg(s:)) N. where N, and N,, are the numbers of positive and negative examples, respectively.
(4) For the SFT data of continuation writing task, we construct the Prompt by truncating the ground-truth SQL query from random position, and the corresponding ground-truth SQL query is used as the Response. Likewise, the SFT data constructed for CW is denoted as D. = (or(di, 4, &)), s; No where No is the corresponding data size and & is the truncated SQL query.
Currently, our multitask supervised fine-tuning (MSFT) dataset is defined as Das = D,UD, UD U D... We will then proceed perform SFT to enable LLMs to handle these tasks explicitly.
F Fine-tuning LLMs through MSFT. Given above MSFT data DM, wherein the input prompt and target response generated are represented x and y for convenience, the supervised fine-tuning for specialized LLMs can be formulated as maximizing the logelikel objective:
T
E (2.y)~Da
log log
(1)
5

---

Published as a conference paper at ICLR 2025
where T is the sequence length of y. After completing the MSFT stage, can obtain specialized LLM that is capable of handling various SQL-related tasks, which allows us to perform multitask collaboration to improve final SQL generation.
3.3 MULTITASK COLLABORATION PROMPTING
To fully utilize these specialized capabilities of LLMs, we develop a Multitask Collaboration Prompting (MCP) method to reduce potential risks in schema linking errors or SQL clause errors in SQL generation as shown in Figure 1-(b). Our MCP consists of three core steps corresponding to the defined task in Section 3.1.
First, to streamline the redundant information in the prompt, we propose an enhanced schema link- ing strategy, which leverages the schema linking capability of LLMs to identify tables and columns relevant to solving the user question, complemented by the pseudo-SQL query (Li et al., 2024d) to simplify the database. Given a database d; and a user question q,, the pseudo-SQL refers to the intermediate SQL query generated in advance using the complete schema information, i.e.
((o,(di, q:), d.). The final simplified database d, can be represented as:
(2) where f is the parsing function to extract the tables and columns from SQL queries through fuzzy matching and ti denotes the operation defined for merging tables and columns. Such a merger oper- ation helps decrease the likelihood of missing potentially related entities (tables columns) and maximize the information relevant to the question, thereby reducing the potential risks in schema linking errors. After obtaining the tables and columns by schema linking, we conduct SQL gener- ation by s (di,(di, 4:)) obtain an intermediate SQL query. Then, explicitly identify the incorrect SQL queries, we utilize LLMs to check them by combing (n, s*) with the exception information e, raised by a SQL executor (SQLer(d, (4, s*)). i.e., MMo,, qt, ee)). Note that e; is empty if the SQL query is executed successfully. If the LLM shows that st cannot answer qi accurately, we take the corrected SQL & and check 8; using the SQL. executor (SQLer 8:)). If executes successfully, replace s" with 5;-
d. M(a,(d., 9:)) w f(M(ar(ds 4;),
Finally, we present a novel strategy stemming from our observations of LLMs on SQL queries after continuation writing. We discovered that continuation writing of given SQL piece by LLMs can enhance the quality of the generated SQL. In view of this finding, we employ LLMs to continue writing truncated SQL queries to refine and improve complex SQL queries. To achieve this, categorize the difficulty of the generated SQL query as follows: Simple: involving only one table, Medium: involving two tables, and Hard: involving more than two tables. For ease of presentation, we define a difficulty evaluation function h(s;,d,) E (1,2,3) where ~ 3 correspond to three hardness levels. For all Hard SQL queries, we conduct continuation writing on the incomplete SQL queries started with 'SELECT' for further refinement. To enhance the clarity of our MCP, we offer a detailed explanation of the process in Appendix A.9.
EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Benchmarks. To evaluate our method, we conduct extensive experiments on five benchmarks to verify the effectiveness of method. These benchmark includes two widely-used cross-domain benchmarks, i..., SPIDER (Yu et al., 2018) and BIRD (Li et al., 2024c), and three robust benchmarks derived from SPIDER, i.e., SPIDER-SYN (Gan al., 202 SPIDER -DK (Gan al., 2021b), and SPIDER- Realistic (Deng et al., 2020), SPIDER consists of 7,000 Text-SQL pairs in the training set, 1,034 pairs in the development set, and 2,147 pairs in the test set, which covers nearly 200 databases and 138 domains. BIRD is a recently proposed benchmark including 9,428, 1,534, and 1,789 pairs in training, development, and test sets, respectively. Compared with SPIDER, BIRD contains more complex databases, more difficult questions, and external knowledge, making it more challenging. For the derived variants, SPIDER-SYE replaces some keywords in the questions in the SPIDER dev set with synonyms. SPIDER-DK introduces some domain knowledge reasoning challenges, while SPIDER-Realinng removes explicit mentions column names in the SPIDER development set. These variants all simulate real-world scenarios for more comprehensive evaluation.

---

Published as a conference paper at ICLR 2025
Table I: Performance comparison on SPIDER and BIRD benchmarks. The results of re-evaluation
using the spen-source code repository are marked with " In groups of open-source LLMs, the best results are highlighted in bold and the second-best results are underlined.
SPIDER
BIRD
Methods
Dev-EX
Dev-TS
Test-EX
Dev-EX
Dev-VES
Prompting with GPT-4
GPT-4 (Achiam et al., 2023)
72.9
64.9
46.4
49.8
DIN-SQL + GPT-4 (Pourreza & Rafici, 2024a)
82.8
74.2
85.3
50.7
58.8
DAIL-SQL GPT-4 (Gao et al., 2024a)
83.5
76.2
86.6
54.8
56.1
MAC-SQL GPT-4 (Wang al., 2023)
86.8
82.8
59.4
66.2
MCS-SQL GPT-4 (Lee et al., 2024)
89.5
89.6
63.4
64.8
Prompting with Open-Source LLMs
Mistral-7b (Jiang et al., 2023)
56.8
47.3
60.1
22.5
27.8
Llama3-8B (Touvron al., 2023)
69.3
58.4
32.1
31.6
Qwen2.5-7B (Yang et al., 2024a)
72.5
64.0
75.9
41.1
42.0
Qwen2.5-14B (Yang et al., 2024a)
76.9
66.3
78.4
48.4
49.2
DIN-SQL + Llama3-8B
48.7
39.3
47.4
20.4
24.6
DIN-SQL + Qwen2.5-7B
72.1
61.2
30.1
32.4
MAC-SQL Llama3-8B
64.3
52.8
65.2
40.7
40.8
MAC-SQL Qwen2.5-7B
71.7
61.9
72.9
46.7
49.8
Ours: MCP AJama3--B
75.0
63.4
72.0
42.7
44.8
Ours: MCP Qwen2.5-7B
78.3
67.2
78.7
49.7
52.8
Ours: MCP Qwen2.5-14B
80.0
67.3
56.3
57.6
Fine-Tumine with Open-Source LLMs
Llama3 8B + SFT (Touvron al., 2023)
82.4
76.2
83.1
53.1
59.0
Qwen2.5-7B SFT (Yang et al., 2024a)
80.9
75.6
82.8
51.4
53.1
DTS- SQL-7B (Pourreza & Rafiei, 2024b)
82.71
78.4
82.81
55.8
60.3
CODES-7B + SFT (Li et al., 2024b)
85.4
80.3
57.2
58.8
CODES-15B SFT (Li et al., 2024b)
84.9
79.4
58.5
56.7
SENSE-7B (Yang et al., 2024b)
83.2
81.7
83.5
51.8
SENSE-13B (Yang al., 2024b)
84.1
83.5
86.6
55.5
Ours: ROUTE Llama3-8B
86.0
80.3
57.3
60.1
Ours: ROUTE Wwwen2.5-7B
83.6
77.5
83.7
55.9
57.4
Ours: ROUTE Qwen2.5-14B
87.3
80.9
87.1
60.9
65.2
Evaluation Metrics. In our experiments, following previous works (Li al., 2024b; Yang et al., experiicn
2024b), we use the execution accuracy (EX) and test-suite accuracy (TS) to evaluate the performance of Texet22QL on SPIDER and its variant benchmarks. More specifically, except for being unable to report TS on the SPIDER-DK and SPIDER test set, report EX and TS on all others. For BIRD, following its official settings, we report EX and an indicator called Valid Efficiency Score (VES) that considers execution efficiency evaluate performance. Note that for all metrics, higher is better.
Implementation Details. We choose the popular LLMs Llama3-88 8BB Instruct (Touvron et al., 2023) and Qwen2.5-7B// 4B-tstruutt (Team, 2024)
investigated models. We the Jlama-Factory framework (Zheng et al., 2024) to conduct SFT and MSFT for reproducibility. We conduct experi- ments 8 1100 GPUs with a batch size of 64 (32 for 14B-sized LLM). LLMs are fine-tuned for two epochs using AdamW with the learning rate of le-5 that decayed to at the end of training by cosine scheduler. During inference, the temperature is set 0.01 ensure reproducibility.
Compared Baselines. Our baselines can be categorized into three groups, i.e. the prompting meth- ods with GPT-4, the prompting methods with open-source LLMs, and fine-tume tnee-tuning-been based methods with open-source LLMs. The first group includes DIN-SQL (Pourreza & Rafici, 2024a), MAC- SQL (Wang al., 2023), DAIL-SQL (Gao et al., 2024a), MCS-SQL (Lee et al., 2024), and the corresponding closed-source LLM GPT-4 (Achiam et al., 2023). For the baselines with open-source LLMs, we choose some popular LLMs and report the zero-shot performance. Besides, we also ap- ply the prompting method, i.e., DIN-SQL, and MAC-SQL, Qwen2.5 (Team, 2024) to explore the robustness. For the last group, it is represented by special- ized LLMs, including DTS-SQU (Pourreza & Rafiei, 2024a), CODES (Li et al., 2024b), SENSE (Yang et al., 2024b), and the base LLMs fine-tuned on the SPIDER and BIRD training sets. To be fair, we re-evaluate the performance of some of the baselines using the open-source code repositories.
Llama3 (Touvron et al., 2023) and

---

Published as a conference paper at ICLR 2025
4.2 COMPARISON RESULTS
Results on SPIDER and BIRD. In Table I, report the performance of our method and baselines on the SPIDER development set, test set, and BIRD development set. Due to time limitations, we unable to provide the results of our ROUTE and MCP on the BIRD test set. From the results, can see that the prompting-based baselines achieve promising performance with the help of GPT-4, however, their effectiveness is obviously limited in the small-sized open-source LLMs. In contrast, our MCP effectively improves the performance of these small-sized LLMs, e.g.. MCP brings more than 5% absolute performance improvement to Qwen2.5-7B on the SPIDER development set. For the fine-tuning group, our ROUTE has achieved the best results in most metrics among the fine- muiin--bssed methods. Especially on the BIRD dataset, our method (14B) surpasses some existing prompting based methods (e.g., DIN-SQL, DAIL-SQL, and MAC-SQL) with an EX score of 60.8, greatly narrowing the gap with existing methods using GPT-4.
Results on SPiDER-variants. Table shows the performance on the benchmarks derived from SPIDER. Except for SPIDER-SYN, our MSFT can slightly improve performance on other variant benchmarks. Despite this, our ROUTE can perform better by inspiring the multi-task capabilities of LLMs with the help of MCP, thus improving performance. In addi- tion, MCP can still have performance gains for the SFT model, demonstrating the necessity of conducting multitask collaboration.
Table 2: Performance on SPIDER-varian bench- marks. The best results are highlighted in bold.
SYN
Realistic EX
DK
Methods
Avg.
EX
TS
TS
EX
Llama3-8B
60.3
47.1
68.5
50.8
58.3
SFT
75.3
68.7
76.8
69.7
72.0
72.5
SFT+ MCP
76.1
69.4
78.0
70.7
73.5
73.5
MSFT
72.1
65.1
77.0
68.1
72.3
70.9
ROUTE
77.4
70.2
80.9
72.6
74.6
75.1
4.3 ABLATION AND ANALYTIC STUDIES
In this section, we first conduct comprehensive ablation study on all benchmarks to explore the impact of each key component, verifying the effectiveness our method. Furthermore, we conduct in-depth analyses to explore the transferability and upper-bound performance of our approach. If not stated, all results are performed on Llama3-8B- -B-tttrttuutttt and evaluated on the SPIDER and BIRD development sets using the EX score.
Table 3: The ablation results (EX) on SPIDER and BIRD development sets.
Table The ablation results (EX) on multi- task collaboration prompting.
No.
SFT MSFT MCP NF SPIDER
BIRD
No.
NC
CW
SPIDER
BIRD
#1
86.0
57.3
V
86.0
57.3
#2
83.6
53.6
56.0
#3
84.5
57.4
#3
83.9
54.7
#4
83.3
53.1
#4
83.8
54,0
#5
82.4
53.1
#5
83.6
$3.6
#6
83.5
56.1
#6
V
75.0
42.7
#7
83.1
52.9
#7
73.3
36.8
#8
83.8
56.0
72.1
38.1
#9
75.0
42.7
#9
71.3
36.4
#10
69.3
32.1
#10
69.3
32.1
Study on ROUTE. Table reports the results of ablation experiments of our ROUTE, where 'V' means that the item adopted and 'NF" means noisy correspondence filtering introduced in Sec- tion 3.2. From the results, we have the following observations. First, our MCP can effectively improve the performance of LLMs before and after MSFT, i.e., #10: 69.3/32.1 to #9: 75.0/42.7 and #2: 83.6/53.6 6 to #1: 86.0/57.3, which demonstrates its effectiveness. Second, applying noisy correspondence filtering significantly improves the performance on SPIDER, i.e., 84.5 vs. 86.0, but slightly degrades the performance hinder the LLM from learning the correct patterns for easy samples. This also suggests that reducing the noise degree and balancing the hard and easy examples in the SFT data is still important for the
BIRD, which indicates that the hard/noisy samples in BIRD
8

---

Published as a conference paper at ICLR 2025
LLM learn the correct SQL-related knowledge. However, in general, the full version of ROUTE achieved the best performance, verifying the importance and effectiveness of each design.
Study MCP. In our ROUTE, MCP can stimulate and give full play to the advantages of mul- titasking collaboration for accurate SQL generation. To this end, we conduct detailed ablation experiments MCP to explore the impact of each task. The experimental results are presented in Table 4, where #1 #1 are for the LLM after MSFT and #6~##1 are for the base LLM. We can that although SL plays a dominant role in performance improvement, it still cannot achieve optimal performance without the full MCP, which shows the complementarity of multiple tasks.
Study on Enhanced Schema Linking. As mentioned above, SL plays leading role in the perfor- mance improvement of MCP, which is attributed to the fact that our SL adopts fusion enhancement strategy, f.e., the combination of the predicted schema linking of LLMs (SLo,) ) and the extracted en- tities from pseudo-SQL. query (SLo,). The experimental results are shown in Table 5, where the first four rows are the results of Llama3-8B after MSFT, and the last four rows are the results of the base model. From the results, SL.. is better than SL as proven by Li et al. (2024d). However, SLr, can easily discard some seemingly irrelevant but important database entities, thereby gener- ating incorrect SQL queries for degraded performance (e.g.. 69.3 to 64.5 set). Therefore, combining SLg, and SLa, can effectively alleviate the issue and achieve a better and more promising performance. In addition, to further understand the performance of our schema linking module, we provide more results and discussions Appendix A.4.
SPIDER development
Study Transferability. To explore the transferability of our ROUTE across different LLMs and different model sizes, we selected several open-source LLMs for experiments, i.e., Gen- eral LLMs: Llama3-8B/70B (Touvron et al., 2023), Qwen2- 7B (Yang al., 2024a), Qwen2.5-7B/722 (Team, 2024); Code Specialized LLMs: CodeLlama-78 (Roziere et al., 2023), Ceepseck-Coder-75 (Guo al., 2024), and Qwen2.5-Coder- 7B (Team, 2024). Due to training costs, we only considered MSFT on the models with a size of around 7B. To explore the transferability of our method sizes, we only apply MCP to them for Text2SQL. From Fig- ure 4, using Text2SQL data will cause the model to lose the ability of other tasks, resulting in poor MCP performance, while MSFT will retain these capabilities for multitask collaboration, thereby generating more accurate SQL. queries. Even for Qwen2.5- 5-Coder which is specially considered for SPIDER and BIRD, our scheme still brings performance improvement. From Table 6, we can see that our MCP is not only appli- cable small-sized LLMs but can also improve the Text2SQL performance of LLMs with sizes of around 70B, which shows its promising transferability on different model sizes. Especially for the challenging dataset BIRD, the absolute improvement empowered by our MCP exceeds 6,5% on average. This shows that our method has good transferability and generalization.
Table 5: The ablation results (EX) on enhanced schema linking.
SL SLo.
SPIDER
BIRD
85.8
56.0
84.1
52.7
85.0
54.5
83.3
53.1
73.3
36.8
LLMs with different model
64.5
30.4
35.4
see that performing SFT single task only
69.3
32.1
SFT
MSFT
SFT Our MCP
Our ROUTE
85.0
83.2
83.7
82.8
84.3
81.3
81,2 81.8 81.7
62.0
82.0
79.4
79.4 79.7
76.0
Qwen2-78
Codellama-78
Deepseek-Code Coder-78
Qwen2.5-Codes
60.0
57.8
54.1
54.6
50.9
50.1
52.2
$1,2
52.1
$1.2 50.7
47.9
48.0
45.9
46.3
42.0
Qwen2-78
CodeLlama-76
Deepseek-Coder-78
Qwen2.5-coder-77
Figure 4: The transferability results on different open-source LLMs on SPIDER (the first row) and BIRD (the second row). See Appendix A.2 for detailed results.

---

Published as a conference paper at ICLR 2025
Study on Upper-bound Performance. In this section, explore the upper bound performance of our MCP and ROUTE to understand the potential of multitask collaboration. As shown in Table 7, define two upper bounds: the first is to schema linking, denoted as U1. The second is to use half of the ground truth SQL. query as a hint refine the wrong challenging SQL by continuation writing, denoted as U2. From the results, ideal schema linking can effectively achieve accurate SQL generation, which shows that schema linking still an effective means to solve Text2SQL at this stage and in the future. Besides, U2 can also significantly improve performance, which shows that continuation writing has the potential to become another figh-precision solution for Text2SQL. By combining them, our MCP and ROUTE show amazing performance on LLMs only with around 7B size.
ground truth simplified database to obtain ideal
Table 7: The upper- bound performance (EX).
Table 6: The performance (EX) of various-sized open-source LLMs.
MCP SPIDER
ROUTE
Methods
BIRD
SPIDER
BIRD
7B
270B
Llama3-8B
75.0
42.7
86.0
57.3
Models
SPIDER BIRD
SPIDER BIRD
U1
79.6
49.8
87.4
69.6
U2
76.8
47.5
87.3
61.2
Llama3
69.3
32.1
77.9
46.9
+ U1 + U2
80.2
53.3
88.8
64.2
Llama3 + MCP
75.0
42.7
79.0
51.8
Qwen2.5-7B
78.3
49.7
83.6
55,9
Qwen2.5
72.5
41.1
81.7
53.3
+ +U1
82,2
58.9
87.3
61.5
Qwen2.5 MCP
78.3
49.7
82.3
57.1
+ U2
78.7
54.7
85.2
60.8
U1 + U2
83.0
60.4
88.2
64.9
LIMITATIONS AND BROADER IMPACTS
Although exploration has achieved promising performance, we have to acknowledge the follow- ing limitations. First, our solution may bring additional reasoning costs. Although some existing efficient inference frameworks can alleviate this (e.g.. VLLM (Kwon et al., 2023)), we still en- courage the exploration of more efficient multitask collaboration modes. Second, from the study on the upper bound performance, there is still large gap between the performance of our method and the upper bound ones, which encourages further study on synthesis data and SFT paradigms of SQL-related tasks understand and mitigate the biases and risks potentially brought by multi- task collaboration. Finally, we provide the ethics and reproducibility statement in Appendix A.13 to avoid the potential impact and risk.
CONCLUSION
In this paper, we study and propose a robust multitask tuning and collaboration method named ROUTE to stimulate the potential of open-source LLM in Text2SQL, narrowing the gap with ex- isting solutions based on closed-source LLMs, such as GPT-4. Our method minimizes the risk of hallucination in SQL generation by explicitly learning multiple SQL-related tasks and conducting multitask collaboration. We apply our approach to recent LLMs to demonstrate its effectiveness and superiority multiple benchmarks. The results show that method has satisfactory transferabil- ity and achieves promising execution accuracy on Text2SQL. In the future, we plan to explore more SQL-relevant tasks, larger LLMs, and more efficient collaboration frameworks for robust Text2SQL..
ACKNOWLEDGMENTE
This work was supported in part by the National Key R&D Program of China under Grant 2024YFB47106044 in part by NSFC under Grant 62472295, 62372315, 62176171 and U21B2040; in part by Sichuan Science and Technology Planning Project under Grant 2024NSFTD0047 and
2024NSFTD0038 in part by System of Systems and Artificial Intelligence Laboratory pioneer fund grant; in part by the Fundamental Research Funds for the Central Universities under Grant CJ202303 and CJ202403; in part by Chengdu Science and Technology Project
0000400000-))) in part by the Sichuan Science and Technology Program (2024NSFTD00499, 2024YFH20089).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Ben Bogin, Matt Gardner, and Jonathan Berant. Representing schema structure with graph neural networks for text-to-sql parsing. arXiv preprint arXiv: 2019.
Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, et al. Dr. spider: diagnostic evaluation bench- mark towards text-to-sql robustness. arXiv preprint arXiv:2301.08881, 2023.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05//28 2023.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.1277, 2020.
Xuemei Dong, Chao Zhang, Yuhang Ge. Yuren Mao, Yunjun Gao. Jinshu Lin, Dongfang Lou, et al. C3: Zero-shot text-to-sql with chatgpt. arXiv preprint arXiv:2307.07306, 2023.
Yingwen Fu, Wenjie Ou. Zhou Yu, and Yue Lin. Miga: unified multi-task generation framework for conversational text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12790-12796 2023.
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward, Jinxia Xie, and Pengsheng Huang. Towards robustness of text-to-sql models against synonym substitution. arXiv preprint arXiv:2106.01063 2021a.
Yujian Gan, Xinyun Chen, and Matthew Purver. Exploring underexplored limitations of cross- domain text-to-sql generalization. arXiv preprint arXiv:2109.05157, 2021b.
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. Proceedings of the VLDB Endowment, 17(5): 132-1145, 2024a.
Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, and Yu Li. Xiyan-sql: A multi-generator ensemble framework for text-to-sql. arXiv preprint arXiv:2411.08599, 2024b. URL https: //axxiv. org/ abs/ org/abs// 2411. 08599.
Zihui Gu, Ju Fan, Nan Tang, Songyue Zhang, Yuxin Zhang. Zui Chen, Lei Cao, Guoliang Li, Sam Madden, and Xiaoyong Du. Interleaving pre-trained language models and large language models for zero-shot n/2sql generation. arXiv preprint arXiv:2306.0888 08891, 2023.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint 2rXiv:2401.141990 2024.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06822. 2023.
George Katsogiannis-Meinet for text-to-sql systems. In Proceedings of the 2021 International Conference on Management of Data, pp. 2846-2851, 2021.
and Georgia Koutrika. A deep dive into deep learning approaches
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
11

---

Published as a conference paper at ICLR 2025
Dongjun Lee, Choongwon Park, Jachyuk Kim, and Heesoo Park. multiple prompts and multiple-choice selection for text-to-sql generation. arXiv preprint arXiv:2405.07467.252 2024.
Mcs-sql: Leveraging
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to sql: Are we fully ready? arXiv preprint arXiv:2406. 01265, 2024a.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13067-13075, 2023a.
Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data, 2(3):1-28, 2024b.
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma. Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13076-1 13084, 2023b.
Jinyang Li, Binyuan Hui, Ge Qu Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024c.
Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions on Machine Learning Research.
Zhishuai Li, Xiang Wang, Jingjing Zhao, Sun Yang, Guoqing Du, Xiaoru Hu, Bin Zhang, Yuxiao Ye, Ziyue Li, Rui Zhao, et al. Pet-sql: A prompt-enhanced two-stage text-to-sql framework with cross-consistency. arXiv preprint arXiv:2403.0973: 2024d.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. A comprehensive evaluation of chatgpt's zero-shot text-to-sql capability. arXiv preprint arXiv: 2023.
Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-sql in the age of well-reasoned language models. arXiv preprint arXiv:2408.07700. 077002 2024.
Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to- sqi with self-correction. Advances in Neural Information Processing Systems, 36, 2024a.
Mohammadreza Pourreza and Davood Rafiei. Dts-sql: Decomposed text-to-sql with small large language models. arXiv preprint arxiv:2402.01117, 2024b.
Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaci, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Serean Arik. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. arXiv preprint arXiv:2410.01943, 2024.
Yang Qin, Dezhong Peng, Xi Peng, Xu Wang, and Peng Hu. Deep evidential learning with noisy cor- respondence for cross-modal retrieval. In Proceedings of the 30th ACM International Conference on Mullimedia, pp. 4998-49566 2022.
Yang Qin, Yuan Sun, Dezhong Peng, Joey Tianyi Zhou, Xi Peng, and Peng Hu. moosl ac- tive complementary learning with self-refining correspondence. Advances in Neural Information Processing Systems, 36, 2023.
Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy- correspondence learning for text-to-image person re-identification. In IEEE International Con- ference Computer Vision and Patlern Recognition (CVPR), 2024.
12

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nitarshan Rajkumar, Raymond Li. and Dzmitry Bahdanau, Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204. 2044099 2022.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /29.5.0...22..... 2023.
Yuan Sun, Yang Qin, Yongxiang Li, Dezhong Peng. Xi Peng, and Peng Hu. Robust multi-view clustering with noisy correspondence. IEEE Transactions on Knowledge and Data Engineering, 36(12):9150-911 2024. doi: 10. 10.119/TKDDE.22244433330
Chang-You You Tai, Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. Exploring chain-of-thought style prompting for text-to-sql. arXiv preprint frXiv:2305.14215. 2023.
Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755, 2024.
Qwen Team. Qwen2.5; A party of foundation models, September 2024. URL https:/:: github.io/blen qwen2.5.....
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian Qian-Wen Zhang. Zhao Yan, and Zhoujun Li. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arX1v:2312.11242, 2023.
Niklas Wretblad and Fredrik Gordh Riseby. Bridging language & data: Optimizing text-to-sql generation in large language models, 2024.
Niklas Wretblad, Fredrik Gordh Riseby, Rahul Biswas, Amin Ahmadi, and Oskar Holmström. Un- derstanding the effects of noise in text-to-sql: An examination of the bird-bench benchmark. arXiv preprint 202.::2442.1224433. 2024.
Chunyang Xiao, Mare Dymetman, and Claire Gardent. Sequence- structured prediction for semantic parsing. Annual meeting of the Association for Computational Linguistics (ACL), pp- 1341-1350, 2016.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 2024a.
Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, and Chang Zhou. Synthesizing text-to- sql data from weak and strong lms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7864 7875, 2024b.
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale and cross-domain semantic parsing and text-to-sql task. arXiv preprini arXiv: aat:::809000000 2018.
dataset for complex
Hanchong Zhang, Ruisheng Cao, Lu Chen, Hongshen Xu, and Kai Yu. Act-sql: In-context learn- ing for text-to-sql with automatically-genet Empirical Methods in Natural Language Processing.
cfain-of-thought. The 2023 Conference on
Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. A survey of table reasoning with large language models. arXiv preprint arXiv:2402.08255 2024.
13

---

Published as a conference paper at ICLR 2025
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient Lin-ttuiigg of 100+ language models. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- 13372.
tics. URL o
A APPENDIX
A. PERFORMANCE COMPARISON WITH DIFFERENT HARDNESS
To explore a more fine- ane-grained performance comparison, we follow previous works (Pourreza & Rafiei, 2024a; Gao et al., 2024a) and report the EX scores development sets of SPIDER and BIRD. From the results shown in Tables and 9, the conclusion of our ROUTE's overall performance consistent with that of its fine-grained performance, which verifies the superiority of our method.
Table 8: The performance (EX) comparison with different hardness on the SPIDER development set. The best results in the fine-tuning group are highlighted in bold.
Methods
Easy
Medium
Hard
Extra
All
comppting-based methods
C3-SQL + ChatGPT (Dong et al., 2023)
92.7
85,2
77.6
62.0
82.0
DIN-SQL + GPT-4 (PourTeza & Rafici, 2024a)
92.3
87.4
76,4
62.7
82.8
DAIL--QLL- + GPT-4 (Gao et al., 2024a)
91.5
89.2
77.0
60.2
83.1
DAIL-SQL + GPT-4 + Self-consistency
91.5
90.1
75.3
62.7
83.6
SuperSQL + GPT-4 (Li ct al., 2024a)
94.4
91.3
83.3
68.7
87.0
MCS-SQL + GPT-4 (Lee et al., 2024)
94.0
93.5
88.5
72.9
89.5
Fine-tuiing-based methods
Graphix-3B PICARD (Li et al., 2023b)
92.3
86.3
73.6
57.2
80.9
RESDSQL-3B (Li et al., 2023a)
94.8
97.7
73.0
56.0
81.8
DTS-SQL (Pourreza & Rafiei, 2024b)
92.7
90.1
74.1
56.6
82.7
RESDSQL-3B NatSQL (Li al., 2023a)
94.4
87.9
77.0
66.3
84.1
ConeS-7B SFT (Li et al., 2024b)
94.8
91.0
75.3
66.9
85.4
Con + SFT (Li et al., 2024b)
95.6
90.4
78.2
61.4
84.9
Ours: ROUTE Llama3-8B
96.0
93.0
75.3
63.3
86.0
Ours: ROUTE Qwen2.5-7B
92.7
89.7
77.0
60.2
83.6
Ours: ROUTE Qwen2.5-14B
94.0
93.0
81.6
68.1
87.3
A.2 THE RESULTS STUDIED ON TRANSFERABILITY
In this appendix, we provide detailed results of a transferability study in Table 10. Furthermore, we compare methods with those using the same base LLMs, i.e., SQL-Llama7B (Wang et al., 2023), DTS-SQL (Pourreza & Rafici, 2024b) and SENSE (Yang et al., 2024b). The results show that method has good scalability various LLMs. whether the base code-specific ones. This is sufficient to verify the superiority and generalization of our ROUTE.
A.3 MORE COMPARISONS WITH RECENT WORKS
In this appendix, we provide more comparisons with recent works in Appendix A.3, including CHASE-SQL (Pourreza et al., 2024), Distillery (Maamari et al., 2024), and CHESS (Talaei et al., 2024). From the results, SPIDER, our ROUTE + Owen2.5-14DP achieves similar performance as CHESS based on Gimini or GPt-4/4o. This suggests that our ROUTE an exceptional choice in both conventional and privatized Text2SQL scenarios. On SPIDER, proprietary by points and CHASS-SQL- + Gemini 1.5 by 12 points. We think this is because the database in BIRD is much complex and contains large number of tables and columns in
ROUTE falls behind CHESS
14

---

Published as a conference paper at ICLR 2025
Table The performance (EX) comparison with different hardness on the BIRD development set. The best results in the fine-tuning group are highlighted in bold.
Methods
Simple
Moderate
Challenging
All
Prompting- baasp methods
ChatGPT Cha (Dong ct al., 2023)
58.9
38.5
31.9
50.2
DAIL-SQL + GPT-4 (Gao et al., 2024a)
62.5
43.2
37.5
54.3
DAIL-SQL + GPT-4 Self-consistency
63.0
45.6
43.1
55.9
SuperSQL GPT-4 (Li al., 2024a)
66.9
46.5
43.8
58.5
MAC-SQL GPT-4 GPT-4 (Wang et al., 2023)
65.7
52.7
40.3
59.4
+ (Lee et al., 2024)
70.4
53.1
51.4
63.4
Fine-tuning-based methods
RESDSQLL3B (L. al., 2023a)
53.5
33.3
16.7
43.9
CODES-7B SFT (Li al., 2024b)
64.6
46.9
40.3
57.0
CODES-15B3 + SFT (Li et al., 2024b)
65.8
48.8
42.4
58.5
Ours: RoUT: Llama3-8B
64.3
49.3
36.8
57.3
Ours: ROUTE + Qwen2.5-7B
63.8
45.4
39.6
55.9
Ours: ROUTE Qwen2.5-14B
67.7
53.1
42.4
60.9
Table 10: The performance (EX) of different open-source LLMs SPIDER and BIRD development sets. V and & are used to mark the methods using the same base LLMs.
Method
SPIDER
BIRD
MAC-SQL SQL-Llama7B (Wang et al., 2023)
76.3
43.9
SENSE-7B (Yang et al., 2024b)
83.2
51.8
DTS-SQL (Pourreza & Rafiei, 2024b)
82.7
55.8
Qwen2-7B + SFT
78.0
45.9
Qwen2-7B + SFT MCP
81.3
50.9
Qwen2-7B + MSFT
79.4
47.9
Qwen2-7B + ROUTE
82.3
54.1
Coda- + SFT
79.4
45.6
CodeI.lama-7B + SFT + MCP
81.1
50.1
CodeLlama-7B + MSFT C CodeLlama-7B + ROUTE
79.7
46.4
83.2
52.2
Deepscek-Coder--55 SFT
81.2
51.2
Deepseek-Coder:-BB + SFT + MCP
81.7
55.5
Deepssek-Coode:-77-- MSFT
81.8
52.1
C Deepseek-Cooder-7B-B ROUTE
83.7
57.0
Qwen2.5 1ren2.55-ooer-71B4 + SFT
82.8
51.2
Qwen2.5 0wen2.5-Coder-77 5CCoe--CCCrrBBBB-- SFT MCP
84.2
54.6
MSFT
82.0
50.7
Qwen2.5-Coder-78 ROUTE
84.3
57.8
single database leading to very long context in input. It widely acknowledged that small-sized LLMs (7B or 14B) are relatively limited reasoning capabilities and handling long texts.
A.4 PERFORMANCE ON SCHEMA LINKING
In this appendix, like (Pourreza & Rafiei, 2024b), report the performance (Recall and Preci- sion) of our schema linking module. The results shown in Table 12 demonstartes several important observations:
15

---

Published as a conference paper at ICLR 2025
Methods
SPIDER-Dev-EX
BIRD-Dev-EX BIRD-DDv-VEES
CHASE-SQU -SLL-----L-- + Gemini 1.5
87.6
73.1
73.0
Distillery GPT-4o
67.2
72.9
CHESS + proprietary (GPT-4)
87.2
65.0
65.4
ROUTE Qwen2.5-14B
87.3
60.8
65.2
Table 11: The comparisons with recent methods on SPIDER and BIRD.
After MSFT, the schema linking capability is significantly improved, especially in terms of precision scores.
A higher Recall score generally leads to improved EX performance due to minor information loss.
While simplifying the database schema necessary, ensuring its completeness is more crucial for achieving enhanced performance.
SPIDER T-P
BIRD T-P
SL..
S
EX
T-R
C-R
C-P
EX
T-R
C-R
C-P
85.8
98.75
94.67
99.24
96.36
56.0
95.21
88.50
96.95
89.93
84.1
97.38
95.71
98.59
96.98
52.7
95.23
90.10
88.21
91.86
85.0
97.01
97.26
98.21
97.99
54.5
91.60
93.14
94.15
94.11
83.3
100.00
18.27
100.00
40.05
53.1
100.00
12.23
100.00
31.64
73.3
97.43
74.99
98.61
90.33
36.8
93.65
73.57
95.78
84.36
64.5
88.35
76.37
94.83
91.46
30.4
83.77
75.38
89.55
86.39
73.1
94.24
91.60
97.12
95.30
35.4
82.15
89.01
88.32
91.63
69.3
100.00
38.47
100.00
18.12
32.1
100.00
12.23
100.00
31.64
Table 12:
The performance of schema linking. T-R/P means the Recall/Precision scores table linking and C-R/P means the Recall/Precision scores on column linking. The best Recall scores are in bold. The first four rows the results of the L.M after MSFT, and the last four the results of the original L.M..
A.5 THE RESULTS ON DR.SPIDER
In this appendix, we conduct experiments on Dr.Spider (Chang et al., 2023) to have clearer and more comprehensive understanding of the advantages of our ROUTE. Dr.Spider includes 17 per- turbation variants that can comprehensively measure the effectiveness and robustness. The specific experimental results are shown in Tables 13 and 14, including the study on MSFT and on each component. In the table, Avg.DB means the average results on DB perturbation test sets, Avg.NLQ means the average results on NLQ perturbation test sets, Avg.SQL means the average results on SQL. perturbation test sets, and Avg.all means the average results all test sets. From the results, the con- clusions are consistent with those on SPIDER and BIRD and each component brought performance improvement, which shows that each task has indispensable contribution to the performance. This further verifies the advantages and robustness of ROUTE.
A.6 THE IMPACT OF SINGLE TASK SFT
In this appendix, we conduct additional experiments to explore the impact of single/triple-task SFT and MSFT on each task. For Text-to-SQL. (TS), we report zero-shot EX results on the SPIDER and BIRD development sets. For Schema Linking (SL), we report the Recall/Precession scores of predicted related tables and columns. For Noise Correction (NC), reports the EX scores of SQL queries refined with Noise Correction on the output SQL queries Llama3-8B. For Continuation Writing (CW), reports the EX scores of all SQL queries obtained by continuation writing half of ground-truth SQL queries. The specific experimental results are shown in following table, where "_' means that L.MMs cannot obtain the output in the expected format due to overfitting. The
16

---

Published as a conference paper at ICLR 2025
Avg. DB
Avg.NIQ
Avg.SQI.
Avg.all
Methods
Pre~Post
Pre~Post
Pre~Post
Pre~Post
Llama3
10.1~55.33
70.6~56.88
69.1~65.5
69.9~~8.8
Llama3 + MCP
75.6~-5..1
77.0~61.7
74.8~72.6
75.8~66.4
Llama3 + SFT
83,4~66.0
83.0~~7.88
79.9~~7.66
82.1~72.2
Llama3 + SFT + MCP
85,4~68.0
85.3~~7.1
84.3~-81.7
85.0~74.9
Llama3 + MSFT
83.8~66.3
82.9~72.5
80.0~777.5
82.2~72.1
Llama3 + ROUTE
86.7~69.6
85.4-75.88
84.5~-819
85.5~~5.88
Table 13: The performance on Dr.Spider benchmark. The best results are highlighted in bold.
Avg.DB Pre-Post
Avg.NLQ Pre-Post
Avg.SQL Pre-Post
Avg.all Pre-Post
No.
SL
NC
CW
#1
86.7~66.66
85.4~~5.88
84.3~~1.9
85.5~~558
#2
86.3~67.9
84.6~75.4
84.4~~8.11
85.1~75.1
#3
84.4~67.7
83.7~73.7
80.3~78.7
82.8~73.3
#4
84.0~66.6
83.0~73.0
80.1~78.0
82.4~~2.6
#5
83.8~66.6
82.9~72.5
80.0~77.5
82.2~72.1
Table 14: The ablation results (EX) on Dr.Spider.
results suggest that tasks not included in MSFT (e.g., No.#2-9) demonstrate lower performance due the overfitting LLMs to other tasks, which not conducive to multi-task collaboration. This highlights the importance of considering SFT across multiple tasks to prevent performance degradation of an L.LM when handling additional tasks. On the contrary, although the performance of the full MSFT on certain tasks, such as SL and CW, is somewhat inferior to that of single triple- task SFT, the full MSFT (No demonstrates significant performance improvements across each task and exhibits better stability, which reduces the risk of overfitting, thus improving the feasibility of multi multi-tasking collaboration.
TS EX EX
SPIDER-SL
BIRD-SL
NC
CW
No. Settings
Table-R/P
Column-R/P Table-R/P Column-R/P
EX
EX
EX
EX
#1
MSFT
83.6 53.6 97.38/95.71 98.59/96.98 90.87/90.22 96.13/90.89
83.4 91.1 73,9
#2
MSFT w/o TS
0.1 16.2 96.58/93.94 98.40/96.32 90.79/88.26 95. 95.95/90.34
45.4 86.5 69.6
#3
MSFT w/o SL
81.8 50.9
(6.447.4133733
#4
MSFT w/o NC
82.8 96.52/94.25 99.00/96.59 90.41/88.85
96.09/90.75
91.7
#5
MSFT w/o CW
81.2 50.3 96.5//33.97 98.65/96.39 90.59/88. 12 96.05/90.64
79.4 81.
#6
SFT with TS
83.1 52.9
85.6
#7
SFT with SL
95.55/92.69 98.91/95.299
87.84/85.11 94.93/89.51
#8
SFT with NC
0.1 8.7
78.9 49.3 48.6 38.6
#9
SFT with CW
68.1 39.0
89.8 70.1
#10
Llama3 w/o SFT
69.3 32.1 88.35/76.37 94.83/91.44 .46 83.77/75.38 89.55/86.39
72.1 1 38.1 57.6
Table 15: The SFT impact of all tasks on each other. The results of MSFT are in bold.
A.7 DETAILS NOISY CORRESPONDENCE FILTERING
In this appendix, we mainly clarify some details in noisy correspondence filtering. To obtain more diversity negative examples, we artificially introduce some errors to the ground-truth SQL queries. Based on some recent works exploring, we focus on five types of errors, schema linking errors, nesting errors, GROUP errors, JOIN errors, and symbol errors. More specifically,
Schema linking error refers to the wrong table and column names in the SQL query. We ran- domly introduce such errors by making typos and synonym substitutions for table or column names.
17

---

Published as a conference paper at ICLR 2025
Nesting errors refer to the need nested or set operations but not using them (e.g., UNION, UNION ALL, INTERSECT and EXCEPT). We destroy the SQL queries by randomly removing the sub-SQLs before after these keywords.
JOIN enrors commonly occur in that the SQL queries using JOIN operation focus on the wrong table or column names. We introduce such errors by randomly replacing the table or column names.
GROUP BY errors commonly occur in that the SQL. queries using GROUP BY operation focus on the wrong column names. We introduce such errors by randomly replacing the column names after GROUP BY.
Symbol errors are some minor errors such as incorrect keywords, missing commas, missing parentheses, and confusing function names (such as COUNT, MAX, MIN, etc.).
To obtain artificially constructed negative examples, select a certain type of error according to certain probability and introduce it into the ground-truth SQL queries. If the SQL query does not meet the type of error introduced, such as no GROUP BY operation, we will randomly select other types of errors to continue to construct negative examples.
A.8 INNOVATION AND DIFFERENCE DISCUSSION
In this appendix, we provide more discussion to further elaborate on the innovations of our ROUTE and highlight how it differs from existing methods. The valuable insights and significant contribu- tions of our work can be summarized as follows:
ROUTE is among the pioneering frameworks under the context of LLMs that explores multi- task tuning and collaborative prompting improve Text2SQL performance.
We have exhaustively introduced and defined three important tasks in generaato,, demon- starting that multi-task tuning and collaborative prompting in Schema Linking, Noise Correc- tion and Continuation Writing significantly improve SQL generation accuracy. The additionally introduced SQL-related tasks are well integrated during both the training and inference phases.
We have achieved sate-of-the-an performance in 7B/14B-sized L.MMs on both the widely- recognized SPIDER and BIRD benchmarks, with verified generalization and transferability across various cross-domains benchmarks and LLMs.
We highlight the key similarities and differences between our ROUTE and other related works as follows:
1. Multi-task Supervised Fine-tuning (MSFT): The method most comparable to our ROUTE approach is MAC-SQL (Wang al., 2023), which introduces multiple task agents and demonstrates the effectiveness of fine-tuning through the use of multi-agent instructions on CodeL.lama-7B (Roziere al., 2023).
(a) First, the defined tasks in MSFT for ROUTE differ from those in MAC-SQL. We have introduced a new continuation writing (CW) task to further refine the challenging SQL queries. As demonstrated in Table 15, CW holds significant potential for SQL generation. On SPIDER development set, exploring CW task able to achieve an impressive EX score of 91.1.
(b) Second, in MAC-SQL, generating instruction data for SFT involves decomposing com- plex questions into multiple sub-questions and constructing corresponding answers. In contrast, our approach, beyond noise correction, allows for the synthesis of SFT data for various tasks using programming functions. This makes our method more practical for large-scale multi-task data synthesis for MSFT.
(c) Third, in terms of performance, our ROUTE is significantly outperforms MAC-SQL based on the open-source LLM of CodeLlama- 7B. The detailed results are presented in Table 10.
2. SQL-Data Synthesis: Our ROUTE involves the synthesis of SQL-related instruction-follooinng data, which shares similarities with the recent work SENSE (Yang et al., 2024b).
(a) First, compared to SENSE, the data synthesis pipeline of ROUTE encompasses not only Text2SQL but also multiply other SQL-related tasks. Our approach focuses on utilizing
18

---

Published as a conference paper at ICLR 2025
existing data synthesize multi-task SFT data, thereby enhancing the capabilities of open- source LLMs to handle various SQL-related tasks. In contrast, SENSE mainly focused on SQL generation task, leveraging strong LLMs to increase the diversity of SQL generation training set and synthesize preference data.
(b) Besides, our ROUTE achieves comparable performance to SENSE on the SPIDER devel- opment and better performance the BIRD development set, shown in Table 10.
3. Multi-tasking Collaboration: To exploit the potential of multi-task capabilities, we propose a multi-task collaborative prompting strategy (MCP) to improve the final SQL generation. The most similar works are DIN-SQL (Pourreza & Rafiei, 2024a) and MAC-SQL. (Wang et al., 2023), which both aim to reduce the complexity of the Text2SQL and improve the final perfor- mance via self-correction.
(a) First, compared to them, MCP efficiently integrates multiple tasks using concise prompts across all tasks, which makes more effective in small-sized LLMs that strug- gle with comprehending complex instructions. As shown in the results of Table 1, the effectiveness of MAC-SQL and DIN-SQL is constrained by the limited capacity of small- sized LLMs to comprehend complex instructions, while our MCP can achieve better and impressive performance.
(b) Besides, while all three methods employ self-correction strategy to enhance the quality of generated SQL queries, MCP introduces novel continuation writing task specif- ically designed to refine challenging SQL queries and improve the performance signifi- cantly.
Considering the comprehensive nature of our work, which encompasses data synthesis, supervised fine-tuning, and multi-task collaborative prompting, with existing work. Nevertheless, our method has offered numerous insights into the Tex12SQ1. and achieved promising results, which we believe are significant contributions to the Text2SQ1. community.
inevitable that there are some similarities
A.9 ALGORITHM OF MCP
In this appendix, to make our MCP clearer, describe the pipeline in detail in Algorithm 1.
Algorithm The algorithm MCP
Input:
The database d, user question q, and LLM M;
// Conduct schema linking.
1: Obtain simplified database d via Equation (2);
// SQL generation.
2: Generate intermediate SQL query via M(or(dd
// Conduct noise correction.
3: Check the SQL query s* via M(an e)))
4: Obtain the the correct SQL if M shows that s* is inaccurate.
5: if M shows s is inaccurate and SQLer (d, 8) is True then
6: s" =
7: end if
// Refine wrong or hard SQL queries by continuation writing.
8: if SQLer(d, s²) is False or hu(s", " .d >> then
9:
Construct the truncated SQL query å based on s*;
10:
Continue writing: ø = (d, q, 8));
if (QLerrdd, 8) is True then
12:
13: end if
14: end if
Output: The final SQL query s*
19

---

Published as a conference paper at ICLR 2025
A.10 STATISTICS OF MSFT DATA
Our MSFT data includes the main SFT data for Text2SQL and the SFT data for three other tasks. The former consists of the SPIDER and BIRD training sets after noisy correspondence filtering (898 pairs removed), with total of 15,530 pairs. The SFT data for other tasks are randomly selected from the filtered dataset and 10,000 data pairs are constructed. That is say, our MSFT data has total of 45,530 data pairs.
A.11 PROMPT TEMPLATES
In this appendix, we provide the prompts all tasks used in our pipeline as shown Figures to 8. Among these prompts, only the BIRD dataset provides question hints. Mcanwhile, other infor- mation, such as few-shot examples and execution exceptions, only provides for non-SFT models to guide the output. The prompt used for noisy correspondence filtering is presented in Figure which is similar to the NC's instructions (Figure 6) but does not include the terms of execution exception avoid LLMs focusing only on the wrong pattern instead of the discriminative pattern.
Given the following database schema and question, your task is to write valid SQL query whose execution results can accurately answer the question.
/* Database schema */
CREATE TABLE customers (CustomerlD NUMBER, Segment TEXT, Currency TEXT, PRIMARY KEY(CustomerID, Segment, Currency)); CREATE TABLE transactions, 1k (CustomerlD NUMBER, Amount NUMBER, Price NUMBER, PRIMARY KEY(CUstomeriD, Amount, Price));
/" Sample of each table */ customers: [3, 'SME, 'EUR'), (5, 'LAM', 'EUR'), (6, 'SME, 'EUR')] transactions 1 [(11543, 28, 672.64), (46707, 18, 430.72), (46707, 1, 121.99)]
/" Question *
Who the top spending customer and how much the average price per single item purchased by this customer? What currency being used?
/" Question hint +
verage price per single item price / amount
Answer the question by SQL query only with no explanation:
Figure 5: The prompt of Text-to-SQL.
20

---

Published as a conference paper at ICLR 2025
determine whether the execution results of SQL query can answer the given question according to the following database schema. If the execution results cannot correctly answer the question, please give me the correct SQL query.
Examples *
...few-hoot examples omitted...
/" Database schema */
CREATE TABLE customers (CustomerlD NUMBER, Segment TEXT, Currency TEXT, PRIMARY KEY(CustomerID, Segment, Currency)); CREATE TABLE transactions 1k (CustomeriD NUMBER, Amount NUMBER, Price NUMBER, PRIMARY KEY(CustomerID, Amount, Price));
/" Sample rows of each table *
customers: [(3, 'SME, 'EUR'), (5, 'LAM', 'EUR'), (6, 'SME, 'EUR') transactions 1k: [(31543, 28, 672.64), (46707, 18, 430.72), (46707, 1, 121.99)]
/" Question //
Who is the top spending customer and how much is the average price per single item purchased by this customer? What currency being used?
Question hint /
verage price per single item price / amount
/" SQL query * SELECT T2. Cu, SUM(T2.Price A2.Aunttt), T1. T1.Curr FROM customers AS T1 INNER JOIN transactions 1k AS ON T1.CustomerID Custoroo WHERE T2.CustomeriD (SELECT
CustomerID FROM yearmonth ORDER BY Consumption DESC LIMIT GROUP BY T2.CustomerID, T1.Currency
/* Execution exception / ..information omitted...
Output:
Figure 6: The prompt template of noise correction.
Given the foll following database schema and question, your task extract the tables and columns relevant solving the question.
Examples *
...few-hott examples es omitted...
/* Database schema +
CREATE TABLE customers (CustomerID NUMBER, Segment TEXT, Currency TEXT, PRIMARY KEY(CUstomerlD. Segment Currency)): CREATE TABLE transactions 1k (CustomerlD NUMBER, Amount NUMBER, Price NUMBER, PRIMARY KEY[CustomeriD, Amount, Price));
/* Sample rows of each table * customers: [(3, 'SME' 'EUR'), (S, 'LAM', 'EUR"), (6, 'SME' 'EUR')] transactions 1k: [(31543, 28, 672.64), (46707, 18, 430.72), (46707, 1, 121.99)]
/* Question *
Who is the top spending customer and how much is the average price per single item purchased by this customer? What currency was being used?
Question hint /
verage price per single item price/ amount
Output:
Figure The prompt template of schema linking.
21

---

Published as a conference paper at ICLR 2025
Given the following database schema and question, your task is to write an incomplete SQL query into a complete SQL query whose execution results can correctly answer the question.
/" Examples *
..fee-shot examples omitted...
/" Database schema */
CREATE TABLE customers (CustomerlD NUMBER, Segment TEXT, Currency TEXT, PRIMARY KEY(CustomerID, Segment, Currency)); CREATE TABLE transactions 1k (CustomerlD NUMBER, Amount NUMBER, Price NUMBER, PRIMARY KEY(CustomerID, Amount, Price));
/* Sample rows of each table * customers: [3, 'SME, 'EUR', (5, 'LAM', 'EUR"), (6, 'SME' 'EUR')] transactions 1 [(31543, 28, 672.64), (46707, 18, 430.72), (46707, 1, 121.99)]
1e Question *
Who the top spending customer and how much is the average price per single item purchased this customer? What currency being used?
/" Question hint //
verage price per single item price / amount
/" The incomplete SQL query "sal
SELECT T2.CustomeriD, SUM(T2.Price T2.Amount), T1.Curry FROM customers AS INNER JOIN transactions 1k ON
Output:
Figure 8: The prompt template of continuation writing.
task determine whether execution results of SQL query can answer given question according to the following database schema. If the execution results cannot correctly answer the question, please give me the correct SQL query.
/* Database schema *
CREATE TABLE customers (CustomerID NUMBER, Segment TEXT, Currency TEXT, PRIMARY KEY(CUstomerID. Segment, Currency)): CREATE TABLE transactions 1k (CustomerlD NUMBER, Amount NUMBER, Price NUMBER, PRIMARY KEY(CustomerID, Amount, Price));
/" Sample rows of each table *
customers: [(3, 'SME, 'EUR'), (5, 'LAM', 'EUR'), (6, 'SME, 'EUR')] transactions 1 [(11443, 28, 672.64), (46707, 18, 430.72), (46707, 1, 121.99))
Question */
Who is the top spending customer and how much is the average price per single item purchased by this customer? What currency being used?
Question hint
verage price per single item price amount
/a SQL query // SELECT T2. Cus SUM(T2.Price T2.Amount), T1.Curyyycy FROM customers AS T1 INNER JOIN transactions 1K CustomerID FROM yearmonth ORDER BY Consumption DESC LIMIT 1) GROUP BY T2.CustomerID, T1.Currency
ON T1.CustomeriD T2.CustomerID WHERE T2.CustomeriD (SELECT
Output:
Figure 9: The prompt of STF data for noisy correspondence filtering.
22

---

Published as a conference paper at ICLR 2025
A.12 QUALITATIVE EXAMPLES OF NOISY PAIRS
In Listings I and 2, we provide more identified noisy pairs in SPIDER and BIRD training sets. To show the error visually, we mainly provide noisy examples with obvious semantic inconsistencies.
Q1: /=How many followers does each user have?*/
Al: SELECT count (*) FROM follows;
R1: SELECT count (4), name FROM uscr_profiles AS T1 JOIN follows AS T2
ON Tl.uid T2.f1 GROUP Tl.uid;
Q2: /=Find the number of followers for each user.*/
A2: SELECT count (*) FROM follows GROUP BY fl
R2:
SELECT count
FROM follows GROUP BY fl;
Q3: /+What is the party that has the largest number of representatives?*
A3:
SELECT Party FROM representative GROUP BY Party ORDER BY COUNT
DESC LIMIT
R3:
SELECT Party, COUNT (+) FROM representative GROUP BY Party ORDER BY COUNT DESC LIMIT 1;
Q4: /*Find the number of kids staying in the rooms reserved by person called ROY SWEAZ.*/
A4: SELECT kids FROM Reservations WHERE FirstName "ROY" AND LastName "SWEAZY";
R4:
SELECT sum Kids) FROM Rooms AS T1 JOTN Reservat ons AS T2 ON mt Roomi - 12. Room WHERE T2. ratName "ROY" AND 12. LastName "SWRAZ"
Q5:
/*Which manufacturer has the most number of shops? List its name and year of opening.*/
A5:
SELECT open_year, name FROM manufacturer ORDER BY nur_of_shops DESC LIMIT 1;
R5:
SELECT name, yean_year FROM manufacturer ORDER BY hur_of_shop DESC LIMIT 1;
Listing I: Identified noisy pairs in SPIDER training set.
Q1: /*What is the number of inhabitants and income of geographic identifier 239?./
A1: SELECT INHABITANTS_K FROM Demog WHERE GEOID 239;
R1: SELECT INHABITANTS K, INCOME FROM Demog WHERE GECID 239;
Q2: /*List the geographic id of places where the income is above average.
x/
A2: SELECT AVG (INCOME_R] FROM Demog;
R2: SELECT GEOID FROM Demog WHERE INCOME_K (SELECT AVG (INCOME FROM
Demog) ;
Q3: /*Average length of the rivers flowing into the Donau River.*/
A3:
SELECT FROM river WHERE Name 'Donau' SELECT avg (Length) FROM river WHERE Name 'Donau';
R3:
Q4: /+What are the corresponding classes for the "very large bike"
attribute? */ SELECT ATT_CLASS_ID FROM ATT_CLASSES WHERE ATT_CLASS 'very_large' SELECT ATT_CLASS_ID FROM ATT_CLASSES WHERE ATT_CLASS very_large bike';
A4:
R4:
Q5: /*Which Shakespeare story with character 324 has description this friend Caesar' ?+/
A5: SELECT T1. Title FROM works AS T1 INNER JOIN chapters AS T2 ON Ti.id T2. 12..wok_ii id INNER JOIN paragraphs AS т3 ON T2.1d = I3, chapter_ic INNER JOIN characters AS T4 ON T3. charaaacterric 14.1d WHERE T2.16 3 AND 12. www.Deescriiii.
"friend "friend_to_camm ta__ttearrree
23

---

Published as a conference paper at ICLR 2025
R5: SELECT T1.Title FROM works AS INNER JOIN chapters AS ON Ti.id T2. work_id INNER JOIN pazagraphs т3 ON T2.1d cww.phapiiecccc INNER JOIN characters AS T4 ON 13.character_id T4.id WHERE T3. character_id = 324 AND T4 De.ccprription = "this_frfi ftt___fffeood__ds__e_e__eee
Listing 2: Identified noisy pairs in BIRD training set.
A.13 ETHICS AND REPRODUCIBILITY STATEMENT
This paper aims to explore the possibility of improving Text2SQL performance by the multitask collaboration based on large language models. For this purpose, we use some open-source LLMs our investigated models propose solutions, which may pose potential commercial risks. We pledge that we will only conduct academic research on these LLMs to verify the effectiveness of our proposed solution and will not use them for other purposes.
In addition, to ensure the reproducibility of our research, we have made several efforts to ensure that our solution is convineing. First, we provide details Section 3.2 clarify the data construction pipeline used for MSFT. In addition, the used prompt templates are provided in Appendix A.11 ensure reproducibility. During the training stage, we utilized unified fine-tuning framework, ie., Llama-Factory, and detailed the settings of core parameters in Section 4.1. During the infer- ence stage, we recommend setting a very low temperature of 0.01 to ensure the reproducibility of LLMs. Finally, have released the code and synthetic data at here for reproducibility, thus further advancing the Text2SQL community.
24


Reviews:
[{"review_id": "xqoPjlKYPS", "reviewer": "Reviewer_2daL", "summary": "This paper addresses the limitations of current Text-to-SQL approaches that rely heavily on in-context learning using closed-source LLMs, such as GPT-4, which can cause privacy issues. To overcome these issues, the authors propose ROUTE, a comprehensive solution to enhance open-source LLMs' Text2SQL capabilities. ROUTE utilizes a multitask supervised fine-tuning approach incorporating tasks like text-to-SQL, schema linking, noise correction, and continuation writing to broaden the model's SQL generation skills and reduce the risk of overfitting. Additionally, a Multitask Collaboration Prompting (MCP) strategy is employed during inference to decompose the SQL generation process into simpler sub-tasks, reducing hallucinations and improving performance.", "questions": "1). For the open-source LLMs and super large databases such as some of the databases in BIRD benchmark, how these large schema are fitted into the prompt of the open-source models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed method significantly improves the performance of open-source LLMs and outperforms all existing methods trained on open-source LLMs.\n\n2) The proposed MCP approach not only enhances the performance of models trained with MSFT but also improves other models.\n\n3) The novel MSFT method substantially boosts model performance compared to standard SFT.", "weaknesses": "1) Although this paper focuses more on open-source LLMs, some recent approaches, such as CHASE-SQL, Distillery, and CHESS, are not included as benchmarks in their experiments.\n\n2) The proposed approach is a multi-step pipeline that can be prone to error propagation. To better understand the performance of the schema linking module and ensure it is not introducing errors into the pipeline, it would be beneficial to report the precision and recall of the schema linking module, as done in CHESS and DTS-SQL.\n\n3) The performance gap with the close-source LLMs is still large, roughly 13% on BIRD development set, which makes the applicability of this approach limited to the scenarios where privacy and local LLMs is essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "INlt8SOUPS", "reviewer": "Reviewer_cCGH", "summary": "The paper introduces “ROUTE,” a novel approach to (1) finetune open-source large language models (LLMs) for Text-to-SQL through multi-task supervised fine-tuning (MSFT) and (2) leverage multitask collaboration prompting (MCP) for SQL generation during inference. The MSFT tasks include Text-to-SQL, Schema Linking, Noise Correction, and Continuation Writing. The proposed method aims to reduce hallucinations and enhance Text-to-SQL robustness, demonstrated by improved performance on two benchmarks: Spider and BIRD.", "questions": "For Noise Correction, is it able to handle a large table as the execution result?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a multitask learning approach that leverages several text-to-SQL related tasks. Noise Correction is designed to assess whether the execution result of a SQL query correctly answers the question, reducing hallucinations when paired with multi-turn generation.\n2. ROUTE demonstrates competitive accuracy, outperforming some closed-source methods on benchmarks, thus showcasing the effectiveness of multitask training over single-task fine-tuning.\n3. The authors provide comprehensive experiments using multiple LLMs as base models, demonstrating that ROUTE is generalizable across various LLMs.", "weaknesses": "1. The paper lacks an ablation study on the contribution of each task in MSFT. For instance, the loss from continuation writing is likely already included in text-to-SQL learning after the first token of the SQL prediction. It is unclear how each task directly benefits SQL generation and other inference components.\n2. Although Noise Correction helps improve performance, it relies on execution results within the model, which may be difficult to apply to queries with large outputs, such as selecting an entire column.\n3. While ROUTE demonstrates strong performance on Spider variants compared to baselines, it remains unclear whether these gains are due to improved robustness or general text-to-SQL performance. It would also be valuable to understand how each component contributes to robustness specifically. Dr. Spider [1] is a more comprehensive perturbation dataset with relative robustness evaluation, which could be useful for evaluating ROUTE’s improvement more clearly.\n[1] https://arxiv.org/pdf/2301.08881", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8F3kfR7dB", "reviewer": "Reviewer_TcaB", "summary": "The paper introduces ROUTE, a method for enhancing text-to-SQL capabilities in open-source language models. The approach addresses limitations in current methods that rely heavily on closed-source large language models (LLMs), such as GPT-4, for text-to-SQL tasks. ROUTE leverages Multitask Supervised Fine-Tuning (MSFT) and Multitask Collaboration Prompting (MCP) to improve SQL generation performance by incorporating tasks like schema linking, noise correction, and continuation writing. These tasks enable a collaborative prompting approach that reduces hallucinations in SQL generation. Extensive experimentation on multiple benchmarks with open-source LLMs shows that ROUTE significantly improves SQL generation accuracy and outperforms recent methods using fine-tuning and prompting approaches.", "questions": "1. The noise correction process assumes access to well-curated data and high-quality schema information, which might not be available for all databases or domains. Without rigorous data preparation, the model may struggle with hallucinations, as noise correction and schema linking effectiveness are diminished when data quality is compromised.\n2.  In low-resource settings where high-quality SQL annotations or database schema information might be scarce, could ROUTE be enhanced by incorporating weak supervision, unsupervised learning, or semi-supervised data to fill gaps?\n3. Given that database schemas often change over time in production, can ROUTE adapt to new tables or columns without needing extensive retraining, or would these require ongoing fine-tuning?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper incorporates multiple tasks to enhance text-to-SQL capabilities, making the LLM more versatile and capable of handling complex SQL generation scenarios.\n2. The paper evaluates ROUTE on several well-known benchmarks and compares its performance with other prompting and fine-tuning methods, demonstrating its effectiveness in real-world applications.", "weaknesses": "1. The authors mention that \"Most training-based methods only incorporate the ⟨Question, SQL⟩ pairs for SFT, resulting in degraded performance in other tasks, such as schema linking.\" However, our approach usually incorporates a ⟨Question, Schema, SQL⟩ tuple for SFT. Additionally, a reduction in schema linking performance cannot be seen as a limitation of existing methods. If a specific task is not included in training, optimal results for that task are not expected. Therefore, this should not be considered a limitation; instead, one could state that training with schema linking can achieve better outcomes.\n2. The authors state that \"Training LLMs on a single SFT task poses a significant risk of overfitting, which may diminish the model's capability to understand instructions.\" However, overfitting is not further addressed in the subsequent sections. Could the authors clarify what overfitting entails in the context of SQL tasks, and explain how multi-task training specifically mitigates this risk? Additionally, training on more data and achieving good results may also suggest a potential overfitting scenario.\n3. The authors mention that \"This strategy leverages collaboration across several SQL-related tasks to reduce hallucinations during SQL generation.\" However, the term \"SQL hallucinations\" is not defined, nor is there any discussion in the experimental section explaining how hallucinations are reduced. This claimed advantage, therefore, remains unclear.\n4. If Schema Linking, Noise Correction, and Continuation Writing are considered important, could the authors provide the relative improvement metrics for these tasks?\n5. There are inconsistencies in writing style, such as using both \"text-to-SQL\" and \"Text-to-SQL\" interchangeably. Ensuring uniform terminology would improve the clarity and professionalism of the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UCvfEVXIp1", "reviewer": "Reviewer_Ej7x", "summary": "The paper proposes ROUTE, a method that involves (1) multi-task training and (2) multi-stage prompting to improve LLMs’ performance on text-to-SQL parsing. Compared to an extensive list of baselines, the proposed ROUTE method demonstrates better parsing accuracy on two established datasets, Spider and BIRD, and three other perturbed variants of Spider.", "questions": "1. What is the “pseudo-SQL” used to perform schema linking? How is it implemented? This term only appears twice in the paper without any further elaboration.\n\n2. The use of “hallucination” may not be appropriate here in the context of text-to-SQL parsing. Are the authors simply trying to say incorrect column matching and entity linking?\n\n3. The manuscript would benefit from another round of proof-read to correct typos and standardize term usage, including those mentioned above and some other examples as follows:\n- “in-contextual learning” -> “in-context learning” (line 39)\n- “promoting-based methods” -> “prompting-based methods” (lines 200, 373)\n- “shema linking” -> “schema linking” (line 228)\n - “SQLer$(d_i, \\tilde{s}^*)$” -> “SQLer$(d_i, s^*)$” (line 283)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The performance of ROUTE is strong. On two well-established text-to-SQL datasets, Spider and Bird, ROUTE effectively improves the performance of two latest open-weight LLMs, Llama 3.1 and Qwen 2.5, and achieves comparable performance to GPT-4. The performance improvement also holds on three perturbed Spider variants, indicating the robustness of ROUTE.\n2. The experiments to evaluate ROUTE are comprehensive. The authors gathered an extensive list of baselines and compared them with ROUTE (or the multi-stage prompting step in ROUTE). Additional experiments and ablation studies of the method further supports some design choices of ROUTE and demonstrates its stable performance improvement across different models and datasets.\n3. The paper is easy-to-follow, and the writing is mostly clear.", "weaknesses": "1. It is not very clear what kind of novel contribution this paper is making. The tasks themselves for multi-task training and multi-stage prompting have all been studied in related work, and some of them are rebranded under new terms. To name some examples, “noise correction” is essentially training and prompting LLMs to self-debug [1][2], and the “continuation writing” is simply a subset of text-to-SQL generation by the autoregressive nature of LLMs and has been one of the prompting paradigm for text-to-SQL parsing with LLMs [3][4]. At the framework level, there are also existing papers compiling different tasks to improve LMs’ text-to-SQL performance via multi-task training [5]. Thus, it is opaque how the proposed method combines these existing ideas in a novel way.\n\n2. The noisy correspondence filtering step to pre-process the training data is not fully elaborated, and the contribution of this step to ROUTE is minimal according to the ablation study (#1 vs #3 and #6 vs #8 in Table 3). Training details and quality of the noise filtering model is not discussed, e.g. through an intrinsic evaluation of how accurately it can discriminate noisy examples. The difference between ROUTE’s data synthesis procedure and that of SENSE is not clear. The method to “artificially and randomly introduce errors” (lines 225-230) is also not documented. Overall, this part of the method is not clearly explained, and its contribution in ROUTE is not obvious.\n\n[1] https://arxiv.org/abs/2304.05128 \n\n[2] https://arxiv.org/abs/2312.11242\n\n[3] https://arxiv.org/abs/2204.00498 \n\n[4] https://arxiv.org/abs/2303.13547 \n\n[5] https://arxiv.org/abs/2212.09278", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)