Structure-Guided Large Language Models for Text-to-SQL Generation
Authors: Qinggang Zhang
Hao Chen
Junnan Dong
Shengyuan Chen
Feiran Huang
Xiao Huang
Venue: ICML 2025 poster (2025)
Keywords: Text-to-SQL
large language model
structure learning
Primary area: deep_learning->large_language_models

Abstract: Recent advancements in large language models (LLMs) have shown promise in bridging the gap between natural language queries and database management systems, enabling users to interact with databases without the background of SQL. However, LLMs often struggle to fully exploit and comprehend the user intention and complex structures of databases. Decomposition-based methods have been proposed to enhance the performance of LLMs on complex tasks, but decomposing SQL generation into subtasks is non-trivial due to the declarative structure of SQL syntax and the intricate connections between query concepts and database elements. In this paper, we propose a novel Structure GUided text-to-SQL framework ( SGU-SQL) that incorporates syntax-based prompting to enhance the SQL generation capabilities of LLMs. Specifically, SGU-SQL establishes structure-aware links between user queries and database schema and recursively decomposes the complex generation task using syntax-based prompting to guide LLMs in incrementally constructing target SQLs. Extensive experiments on two benchmark datasets demonstrate that SGU-SQL consistently outperforms state-of-the-art text-to-SQL baselines.

Structure-Guided
Large Language Models for Text-to-SQL Generation
Qinggang Zhang'
Hao Chen Junnan Dong'
Shengyuan Chen Feiran Huang Xiao Huang'
Abstract
to interpret intricate natural language queries and construct SQL queries with precise syntax structure, all while linking with the correct tables and columns in the database. A wide range of research has been proposed to address these issues, including intermediate query languages and skeleton query generation (Wang et al. 2019 Li et al. 2023a b).
Recent advancements in large language models (LLMs) have shown promise in bridging the gap between natural language queries and database management systems, enabling users interact with databases without the background of SQL.. However, LLMs often struggle to comprehend complex database structures and accurately inter- pret user intentions. Decomposition- -hased meth- ods have been proposed to enhance the perfor- mance of LLMs on complex tasks, but decompos- ing SQL generation into subtasks is non-trivial due to the declarative structure of SQL syntax and the intricate connections between query concepts and datahase elements. In this paper, we pro- pose a novel Structure GUided text-to-SQL frame- work (SGU-SQL) that incorporates syntax-base prompting to enhance the SQL. generation capabil- ities of LLMs. Specifically, SGU-SQL establishes structure-aware links between user queries and database schema and decomposes the complex generation task using syntax-based prompting to enable more accurate LLM-based SQL genera- tion. Extensive experiments on two benchmark datasets demonstrate that SGU-SQL consistently outperforms state-of-the-art eeLt models.
Recently, this field has seen significant progress with the emergence of Large Language Models (LLMM)) like GPT se- ries (Radford et al. 2018 Achiam et al., 2023 Zhang et al.) 2024 2025). Training on a wide array of corpora, LLMs ex- hibit exceptional ability in understanding and producing text that closely mimics human communication. Researchers have started exploring the potential of LLMs for text-to- SQL by leveraging their extensive knowledge reserves and superior generation capabilities (Rajkumar et al. 2022 Gao
et al. 2024 Hong et al.) 2024a). These approaches often involve prompt engineering to guide proprietary LLMs in
SQL generation (Chang & Fosler-Lussier 2023 Pourreza & Rafiei 2023) or fine-tuning open-source LLMs on text-
to-SQL datasets (Gao et al. 2024 Yuan et al. 2025).
Despite their advancements, LLM-based text-to-SQL. mod- els encounter several limitations that impede their success- ful application in practice: O Ambiguous User Intent. Accurately interpreting the user's intents in natural lan- guage remains a significant challenge for LLM-based mod- els. Natural language is inherently ambiguous and context- dependent, making it difficult for models to discern precise requirements. For example, a query like "Show me last quarter's sales performance" requires the model to infer specific details such relevant tables, metries defin- ing "performance" and the exact time frame for "last quarter". Additionally, nuanced language involving im- plied conditions or comparisons, such as "better than average" or "most recent" can lead to misinterpretations, resulting in queries that do not fully align with the user's intent. @ Sophisticated Database Architecture. Map- ping natural language terms to specific database columns and tables is another critical area where LLM-based mod- els struggle. Databases often have complex schemas with interrelated tables and non-intuitive naming conventions. For instance, a user referring to "customer purchases" might imply multiple tables like "Customers", "Orders" and "OrderDetails" The model must accurately identify and relate these tables, which is challenging without com-
1. Introduction
Text-to-SQL (Hong et al. 2024ch is a challenging task that aims bridge the gap between natural language queries and database management systems, enabling users to interact with databases without knowing SQL syntax. In the past few years, this task has been incrementally evolving due to the complexity of SQL syntax and the intricate connections between user queries and database elements. Models need
'Department of Computing, The Hong Kong Polytechnic Uni- versity, Hung Hom, Hong Kong SAR. China *City University of Macau, Macau SAR, China "College of Information Science and Technology, Jinan University, GZ, China. Correspondence to: Shengyuan Chen <shengyuan.co www.eguuan.cheeeeeee
chen @ connect.polyu..te.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Structure-Guided Large Language Models for Text-to-SQL Generation
prehensive schema awareness. Moreover, similar column names across different tables can cause confusion, leading to incorrect selections and incomplete queries, especially in large or poorly documented databases. @ Complex Syn- tax Structure of SQL.. Generating syntactically accurate and logically coherent SQL queries is a challenging task. SQL requires precise clause arrangement, correct operator usage, and adherence to grammatical rules. LLMs may produce queries with syntax errors, such missing com- mas, incorrect JOIN conditions, or misplaced keywords. Constructing complex queries involving nested subqueries, aggregate functions, or window operations demands high precision, which is typically beyond the current capabilities of LLMS. Recently, decomposition- methods have been proposed to enhance the performance of LLMs on complex tasks. However, decomposing the complicated linked structure into smaller, manageable components for step-by-step SQL. generation requires effective strategies. Traditional approaches often struggle with complex queries due to the declarative structure of SQL and the intricate connections between user queries and database elements.
2. Problem Statement
Let D be a database schema consisting of a set of tables T = (Ti T2..... .......,TyT. Tn). where each table T; has a set of columns C, = {Ci1, Co...., Cim The database schema D can be represented as tuple (T,C), where C = Ui= C.. Using the above notations, we describe our problem below.
Definition 1. Structure Learning for TexL-to-SQL: Given a natural language query D and a database schema C, the task of graph learning for Text-to-SQL. aims to generate a graph- representation 9 that captures the structural and semantic relationships between the query and the schema, and to learn mapping function f 9g - Ga, where Gg is the structural user queries, and Ga is the corresponding database contents linked to the query 9g-
Definition 2. Text-to-SQL Generation: Given a natural language query Q and database schema D, the task of Text-to-SQL generation aims to translate Q into a corre- sponding SQL query S that accurately retrieves the desired information from the database.
In this paper, we propose novel Structure Guided text-to- SQL framework (SGU-SQL). SGU-SQL addresses the above issues by leveraging the structural information in queries and databases through cructure-aware linking and syntax- based decomposition, providing additional guidance to the LLM for better SQL generation. Specifically, SGU-SQL rep- resents user queries and databases into unified and structured graphs and employs a tailored structure- learning model to establish a connection between the user queries and the databases. The linked structure is then decomposed into sub-syntax trees, guiding the LLMs to generate the SQL query incrementally. Our main contributions are summa- rized as follows:
3. The Framework of SGU-SQL
In this section, we will introduce the key components of SGU-SQL in detail. We leverage the implicit structural in- formation in both queries and databases from three aspects: (i) A graph-based structure construction for both user query and database understanding: (ii) A tailored structure linking method is proposed to map the natural language query to the relevant database elements. (iit) Structure- prompting to LLMs for accurate SQL generation, which decomposes the complex generation into sub-tasks and guides L.LMs to generate the SQL query incrementally, adhering to the syn- tax structure. The overall framework is shown in Figure [
We identify the limitations of LLM--ased TSLL models and introduce SGU-SQL, which breaks down the complex generation task in syntax-aware manner. This ensures that the generated queries maintain both semantic accuracy (correctly capturing user intentions) and syntac- tic correctness (following proper SQL. structure).
3.1. Revisiting User Query and Database via Graph
Bridging the gap between textual queries and the struc- tured database poses several challenges. Firstly, construct- ing an accurate structure that captures the relationships be- tween query terms and database entities is non-trivial task. Secondly, linking the query to the appropriate tables and columns in the database is challenging, especially when there is ambiguity or a lack of explicit connections. In this paper, we build a comprehensive query-schema graph designed to structure the query concept, the schema, and pre- defined relations between the query phrases and the tables or columns present within the schema. The graph contains three key structures: (i) Query Structure (R₂):: Encodes dependencies between tokens in the question, derived from its syntactic parse. (ii) Database Structure (R,): Repre- sents intrinsic relationships within the database schema, like foreign keys. (iii) Linking Structure (R)): Aligns query entities with the columns tables in the database.
SGU-SQL proposes graph-based structure construction to comprehend user query and database structure and then link query and database with dual-graph encoding.
SGU-SQL introduces tailored structure-decome -eccuured gen- eration strategies to decompose queries with syntax trees and then incrementally generate accurate SQL with LLM.
Experiments on two benchmarks verify that SGU-SQL outperforms otate-of-the-ar baselines, including 11 fine- tuning models, 7 structure learning models, and 14 in- context learning models.
2

---

Structure-Guide Large Language Models for Text-to-SQL Generation
Lber
Depondenry
Whet had both Mosters
tochellrs acrolle???
Synaatik Dependency
SELECT
Query Groph
FILTER
User Query
Database Schema
Result
Decompasing Tasks
Bodke coume of INTEGER
Start
Jasks
Column
PoReeCt Ceourn
Table
Sohama
Schemp Graph
Sub-Tosks
Dual Graph Encoding
Syntax based Decomposing
Prompts
Figure I: The overall framework of SGU-SQL.
3.1.1. USER QUERY UNDERSTANDING AND REPRESENTATION
where P(q: g) is the probability of interpretation given the original query
A query graph can be depicted as G. (V,, R2), where V, denotes the node set that characterizes the keywords specified in the question, and Rg signifies the relationships among these keywords. To differentiate the relationship between various words, we establish three separate link categories, including Forward-Syntax, Backward- Birkward-Syntar Synttax and Synntax relations to encapsulate the particular syn- tactic connections among words in the vernacular question.
Natural language queries may contain multiple mentions of the same entity, which need to be resolved construct an accurate graph representation. Let M be the set of entities mentioned in the query and F be the set of unique entities. The coreference resolution can be formulated as finding mapping function ø M + E that maps each mention to its corresponding entity:
((m) argmax_eee P m),
(3)
a) Query Parsing: Syntactic parsing can help resolve struc- tural ambiguities in the query by providing a hierarchical representation of the sentence structure. Specifically, we first define context-free grammar Gg for the query lan- guage:
where P(e m) is the probability of entity given the mentioned entity m.
c) Query Graph Construction: Once the syntax tree T, is ob- tained, we can construct the graph structure G, = (Vg Egy) representing the user query. The vertices V, = Vg is the set of query concepts and terms and edges Eg are defined as follows:
(1) where N₂ is finite set of non non-terminal symbols represent- ing query concepts. E.g is finite set of terminal symbols representing query terms. Pa is a finite set of production rules that map non-terminals to sequences of terminals and non-terminals. S, Ng is the start symbol.
G₁ = (Na; Egg Pq. Sq);
Eg = E.g U (1))0y) Us; Ui E Vg A relation(e;, Uz). (4)
The edges E, in the graph structure include both the syntac- tic relationships from the syntax tree and additional edges based on semantic relationships between query concept- s/terms. The resulting graph structure Gg captures both the syntactic structure of the user query and the semantic relationships between query concepts/terms.
The production rules Pg define the syntactic structure of the query language.
Parsing user query Q using the grammar G2 yields syntax tree T2 (V..E.) Eq), where Vg is the set of vertices representing query concepts. F.y R2 is the set of edges representing syntactic relationships between the query con- cepts.
3.2. DATABASE UNDERSTANDING AND REPRESENTATION
b) Coreference resolution: Natural language queries often contain ambiguities, such as polysemy (words with multiple meanings) and syntactic ambiguity (multiple possible syntax trees). Let Q be the set of all possible interpretations of query g. The ambiguity challenge can be formulated as selecting the most likely interpretation q from Q:
To generate accurate SQL queries, text-to-SQL systems also need to have a comprehensive understanding of the database structure, including table names, column names, and relationships between or across various tables/columns. Representing and encoding the database in a way that can be effectively utilized by the eext-to-SQL model is a challeng- ing task. In this paper, we introduce a schema graph to rep- resent database structure. Specifically, let D be database
q = arg max P(gs 9):
3

---

Structure-Guidel Large Language Models for Text-to-SQL Generation
consisting of a set of tables T = Ti, T2,....... Tn- Each table T; ' has set of columns C; (C11, Cizy..... Cim We define a database schema graph Ga = (Va, Ra)) to represent the structure of the database schema, where S denotes the set of nodes representing tables and columns, and Ra is the set of edges representing the relationships between them.
the structure of the database schema, with vertices repre- senting tables and columns and edges representing their re- lationships. By combining the syntax tree with the schema graph through the mapping function o, we establish a link between the query concepts/terms and the corresponding schema elements. This mapping allows us to identify which tables and columns in the database are relevant to the user query, enabling more accurate and targeted querying.
a) Node Representation: Each table T; E T represented as node VT; E S in the schema graph. Similarly, each column Cij E C, of table T; is represented as node tC,,, E S. The set of nodes S in the schema graph is defined as:
Specifically, given the constructed query and database graphs, we value the adjacency information during the matching process and propose to automatically build the connection between the query structure and schema at the node level. Specifically, we design a tailored structure-basce linking framework. Both query and schema structures are first encoded through a Relational Graph Attention Network (RGAT) (Busbridge et al. 2019) for initial node representa- tions. The representation learning process is guided by the message propagation within the self-structure. We formalize the procedure of structure-aware question-schema structure linking as follows:
= vr. TUNC C. T.T.T.T. (5)
b) Edge Representation: The relationships between tables and columns in the database schema are represented as edges in the schema graph. We define the following three types of edges:
Table-Colume Edges: For each column Cij E C; of table Ti, we add an edge ET;; Ctj} E Rs connecting the table
node öT, to the column node U.C, This edge represents the relationship between a table and its columns.
G/ Agg Gq);
(9)
G Agg(G Ga),
(10)
E(T,, Ciy) = {UT, PC,y, "has").
(6)
where the structure-aware aggregation function Agg(.) is employed to gather information from both the achema-graph Ga and the query-graph Gg and transfer it to the adjacent graph.
Primary-Key Edges: If a column Cij E C; is the primary key column of table T,, we add an edge E(Coy. TT) E Ra connecting the corresponding column nodes VC., and the table VT, The primary-key relations in the schema graph provide information about the structure and integrity con- straints of the database.
Let (h()m² represent set of node embeddings in the query graph Gú and let (ht }リ= denote a set of node embeddings in the subgraph 9% that extracted from the schema graph Gg. In partieular, we first employ global-average pooling on the node embedding hy of the query structure Gg to derive the global query structure embedding hg. Following this, to encapsulate globally pertinent information, the key node embedding h is updated subsequently:
E Cit) = (ur, "C,,, "primary.key") (7)
Foreign-Key Edges: If a column City E C, of table 7; is a foreign key referencing a primary key column Cj E C2 of table Tg. we add an edge E{Cr,, Cu} E Ra connecting the corresponding column nodes UC;; and v.a.. This edge represents the foreign key relationship between the columns.
hg m
hg
(11)
aj 0 (hg SSTWWhh) h1)
(12)
h = Σ EIEN, an W + a, Wih H11-a)
(13)
E(Crj, Ck) (uo.y V. "foreign_kep key").
(8)
(14)
3.1.3. STRUCTURE LINKING WITH DUAL GRAPH ENCODING
where WG, Wg. We represent trainable parameters, and illustrates a sigmoid function. While aj denotes the rel- evance score situated between the j-th key node and the global query structure.
The syntax tree T, obtained from parsing the user query Q captures the syntactic structure of the query. It represents the hierarchical relationships between query concepts and terms, which is crucial for understanding the intent behind the query. By incorporating the syntax tree into the query graph Gg. we preserve the syntactic structure of the query and its inherent meaning. The schema graph G. represents
For each node in the query structure Gy, it is necessary to find a corresponding matching node in the database Ga- The proposed solution mainly consists of three steps. First, a set of most relevant candidate nodes {si, $2, sx } is identified through string matching in the set of tables and
4

---

Structure-Guided Large Language Models for Text-to-SQL Generation
columns V. Second, for each candidate node 8, an enclos- ing subgraph G(a, s) is constructed. As shown in Figure [ G(a, s) includes the query graph Gg. adjacent nodes of 8k, and an edge connecting and Sk. Lastly, we adopt struc- ture learning model (- to learn the graph-level repre- sentation of Gia, 5x) that captures the compatibility between natural language concepts and database elements.
first employ the query parsing described in Section to build the syntax tree achieve linguistic understand- ing of the natural language query and then adopt node mapper to match nodes in the linguistic syntax tree to SQL operations (Kate) 2008). Following this, the original query can be divided into several subtasks according to the SQL operations distributed on the syntax tree.
RGAAT(((ass))
(15)
3.2.2. SUBTASK DECOMPOSITION
Given the context-free syntax tree T, decompose the generation task into subtasks based on the syntactic structure of the query. Each non-terminal node n E N in the tree represents subtask that needs to be solved to generate the corresponding part of the SQL query. The decomposition process f N + S maps each non-terminal node to its corresponding SQL component.
The matching score of the candidate pair sk) is then measured by the degree of compatibility:
((s)))) = o > (c h
(16)
Based on positive samples (a, s) and negative samples (a, sk), where 8 * s, the structure learning model RGAT(-)
is iteratively trained:
3.2.3. SQL GENERATION
To generate the SQL component 8n for non-terminal node n E N, we employ LLM M that takes the natural lan- guage query Q and the subtask context c, as input and produces the corresponding SQL component:
S(ai,)) 8)
CR min
log S(a,, s) + Es, Es S(a), 8k) aFEo,
Σ
(17)
Sn = =M(Q c.).
(18)
This contrastive training objective encourages the model to maximize scores for correct matches while minimizing scores for incorrect ones. Through this process, the match- ing scores evolve into reliable indicators that guide the se- lection of correct database elements during SQL generation.
The subtask context en captures the relevant information from the context free syntax tree T that is needed to gener- ate the SQL component for node nt. It can include the parent node, sibling nodes, and other relevant contextual informa- tion. The final SQL query S is obtained by combining the SQL components generated for all the non- non-terminal nodes in the context-free syntax tree T, starting from the root node no: S Sno By decomposing the text-to-SQL generation task into subtasks based on the syntax structure of the user query, we can leverage the hierarchical information captured by the context-free syntax tree generate more accurate and structured SQL queries.
Incorporating pre-defined relations: After we got the accurate linking from the structure learning model, we further incorporated several additional relations to supple- ment effective connections between the user query and the set
database schema. The mapping function ø relies of pre-defined relations R. between query concepts/terms and schema elements. These relations capture the semantic connections between the query and the database schema. By incorporating these relations into the query-schema graph construction, we ensure that the final graph not only captures the syntactic structure of the query but also incorporates the semantic relationships between the query and the schema.
4. Experiments
This section empirically evaluates the proposed SGU-SQL. Our experiment motivated by the following questions: Q1 How does our proposed SGU-SQL perform in comparison with the strongest baselines? Q2 Could our proposed Gram enhance other I.MMs by substituting the original framework with the decomposing-basee prompt? Q3 Is our proposed structure prompting effective when handling queries of dif- ferent complexity? Q4 Which type of queries are prone to occur errors? And what is the underlying reason?
3.2. Structure-Decomposeed Prompting with Syntax Tree
3.2.1. DECOMPOSING QUERY WITH SYNTAX TREE
The performance of LLMs on complex tasks can be im- methods. However, decomposing a SQL query into subtasks is challenging due to its declarative structure and the intricate connections between query concepts. To this end, in this section, we introduce a context-free syntax tree to break down the text- to-SQL generation task into smaller subtasks according to the syntax structure of the user query. Specifically, we
proved by using decomposing-basee
4.1. Experiment Setup
Datasets We assess the performance of text-to-SQL SQL models using two renowned datasets, Spider (Yu et al., 2019) and
5

---

Structure-Guided Large Language Models for Text-to-SQL Generation
Table 1: The Execution Accuracy of text-to-SQL models on SPIDER. The best and second-best results in each column highlighted in bold font and underlined. V and x represent that the case is applicable and not applicable, respectively.
Text-to-SQI. Method
Backbone LM/LLM
Finetuning
Structure Information
Prompt
SPIDER
Strategy
Easy
Medium
Hard
Extra
Overall
SFT
0.5775=0.0106
0.352110.0130
0.201010.0089
0.066710.0115
0.3353a0.0125
Baichuan2 7B
LoRA
X
0.8714:0.0073
0.6305=0.0069
0.448900.0063
0.295840.0084
0.6035a0.0079
QLoRA
X
0.891920.0057
0.6367=0.000
0.488520.0053
0.330620.0079
0.624220.0061
Baichuan2
SFT
X
0.5805=0.0099
0.4133+0.008
0.2644=0.0067
0.1875=0.0077
0,3927=0.008
Baichuan2-13B
LoRA
X
0.9024=0.0075
0.7015=0.0000
0,5688#0.0883
0.3915=0.0071
0.677620.0080
QLoRA
0.8951=0.0103
0.67466000122.
0.580920.0115
0.343420.0109
0.659220.0114
LoRA
0.8868=0.0016
0.6410 =0.0041
0.4892+0.0030
0.3311=0.0017
0.6259=0.0022
LlaMA2-7B
QLoRA
0.8472+0.0025
0.6234+0.000
0.4658+0.0021
0.3309H0.0022
0.6083+0.0035
LlaMA2
LlaMA2-138
LoRA
0.906620.0033
0.7292+0.00
0.5517#0.0029
0.3430+0.0055
0.680940.0030
QLoRA
x
0.91 10+000004
0.7004+0
0.5523+0.0022
0.3190+0.006]
0.664880.0005
SFT
X
0.4110:0.0093
02293+0.0075.
0.990610.0081
0.0725+0.0090
0.2414=0.0108
LlaMA2-708
LoRA
x
0.9151=0.0069
0.7323+0.000 0080
0.5575+0.0049
0.3921++000035
0.686900.0040
SFT
X
0.2136=0.0155
0.1769=0.010
0.0921=0.0169
0.0363=0.00
0.1487=0.0163
CodeLlama-7B
LoRA
X
0.9228=0.01
0.7562+0.0134
0.5863=0.0090
0.3485+0.0126
0.701820.0108
QLoRA
0.9115
0.750610.0143
0.598240.0120
0.331020.00885
0.696120.0104
CodeLlama
SFT
0.6980=0.0115
0.6015#0.0122
0.4073=0.0109
0.2708+0.0145
0.5288=0.0140
CodeLlama-135
LoRA
0.941420.0088
0.788510.000
0.684240.0081
0.404120.0066
0.746210.0092
QLoRA
0.940240.0053
7744450000066
0.626340.0085
0.3915a0.0061
0.7270a0.0085
SFT
X
0.7223+0.0143
0.624510.0122
0.443220.0131
0.3028a0.0147
0.5675a0.0144
CodeLlamaa-70B
LoRA
x
0.9621#0.0053
0.8122:0.0066
0.716720.0055
0.432410.0069
0.771020.0061
SFT
X
x
0.3956=0.0155
0.2561=0.0133
0. 0.1384+0.0137
0.0427=0.0169
0,2356=0.0140
Qwen-7B
LoRA
X
0.8546#0.000
0.6876+0.00m 0089
0.5743+0.0076
0.3340 0065
0.6519#0.0075
QLoRA
0.91 0.9110=0.0045
0.6747=0. 0001
0.5750=0.0076
0.3436+0.0055
0.6623=0.0069
Qwen
SFT
0.8713=0.0100
0.6323+0.00 0144
0.3686=0.0133
0.1810=0.0120
0.5735+0.0135
Qwen- 14B
LoRA
X
0,894620.011 01110
0.7021
0.551720.0125
0.3669#0.0118
0.6625=0.0121
QLoAA
0.9185=0.0075
0.7439H0
0.597640.0081
0.458340.0083
0.7010=0.0090
SFT
X
0.831340.0100
0.634510.0077
0.488610.0065
0.277240.0123
0.6033a0.0110
Qwen-72B
LoRA
X
0.9269#0.0075
0.7563+0.0059
0.6215+0.0083
0.3673+0.0136
0.7127+0.0094
x
X
0,804410.0107
0.6395.0.000
0.557320.0124
0.403610.0101
0.6271
RAT-SQL
Lage
SFT
0.8643#0.0119
0.736740.0145
0.6210H0.0093
0.4279H0.01
0.6955e0.0124
x
x
0.8633=0.0097
0.6952+0.0066
06154+0.0099
0.4106e0.00
0.676820.0109
LGESQL
BERT-Large
SFT
0.9150=0.0103
0,7647+0.006
0.6673+0.0107
0.4888#0.000
0.7421=0.0096
T5-Large T5-3B
SFT
0.899340.0075
0.787410.00 0068
0.598010.0102
0.440110.0083
0.726340.00 20.977
Graphx-T55
SFT
0.9193=0..0338
0.8164:0
0.615720.0053
0.500620.0081
0.756240.0065
TS-Base
SFT
0.919020.0047
0.836910.000 0.001
0.684120.0070
0.518340.0065
0.779710.0073
RESDSQL
TS Large
SFT
0.9355#0.0040
08843300.005
0.7241=0.0070
0.5361+0.0045
0.800820.0063
TS-3B
SFT
0.9476+0.0081
0.8767+0.0104
0.7299#0.0120
0.5602+0.0094
0.8182+0.0100
DTS-SQL
DeepSeck-7B
SFT
x
0.9274=0.009
0.9013+0.0077
0.7414=0.0090
0.5663+0.0103
0.8269=0.0094
CodeS
CodeLlama-133
SFT
x
0.927420.0084
0.878920.00 0002
0.706920.0079
0.590420.0038
0.8150=0.0070
C'-SQL
GPT-3.5
x
x
0.9136=0.0000
0.8402+0.0094
0.7731=0.0064
0.6153=0.0088
0.8108=0.0095
DIN-SQL
GPT-4
x
x
0.923420.0055
0.874410.000
0.764410.0091
0.626510.0103
0.827910.0098
DAIL-SQL
GPT-4
x
x
0.9153=0.0103
0.8924x0.0125
0.7701=0.0098
0.602420.0107
0.8308=0.0110
EPI EPI-SQL
GPT4
x
x
0.9310#0.0121
0.905340.000
0.817820.0108
0.618910.0097
0.851120.0114
SuperSQL.
GPT-4
x
x
0:9435+0.0074
0.9126+0
0.833340.0062
0.686740.0055
0.868220.0068
PURPLE
GPT4
x
x
0.940440.0086
0.920620.0041
0.8268:0.0055
0.6715#0.0080
0.8670a0.0072
SGU-SQL
GPT-4
V
0.935240.0061
0.9190=0.0043
0.8437+0.0005
0.721360.0067
0.8795=0.0063
6

---

Structure-Guided Large Language Models for Text-to-SQL Generation
BIRD Li al. 2023c). Spider, a cross-domain text-to-SQL dataset, comprises 8659 instances in the training split and 1034 instances in the development split, spanning across 200 databases. Each instance comprises a natural language question related to a specific database and its corresponding SQL query. For evaluation purposes, we utilize the Spider- dev development split since the test split has not been re- leased. On the other hand, BIRD (Blg Bench for large-scale Database Grounded text-to-SQL. Evaluation) is another pio- neering cross-domain dataset that focuses exploring the impact of extensive database contents on text-to-SQI. pars- ing. BIRD features over 12,751 unique question-SQL pairs, encompassing 95 large databases with a total size of 33.4 GB. encompasses more than 37 professional domains.
based methods and 11 SOTA incontext-leaminng meth- ods over two benchmark datasets. As shown in Table T and 2 SGU-SQL achieves superior performance across both datasets, outperforming all baselines in terms of execution accuracy and exact match accuracy.
Obs.2. In-context learning- method is better than the methods of the other two categories. Among the three cate- gories of methods, in-context learning-based methods con- sistently demonstrate superior performance. This suggests that leveraging in-context learning mechanisms is crucial for enhancing the understanding and generation of SQL queries from natural language inputs. Specifically, the in-context baased
learning- based methods, i..., DIN-SQL and DAIL-SQL in our comparison set achieve higher accuracy rates and re- quire less computational overhead compared fine-tuning and pructurr-learnnng based methods. Additionally, the in-context learning-base based methods exhibit better generaliza- tion across different datasets, indicating their robustness and adaptability.
Baselines To valid SGU-SQL, we compare it with several state-of-art baselines. Following the taxonomy in Sec- tion D we divide baselines into three categories: (i) Fine-
tuning:
T5-base (Raffel et al.) 2020). T5-large (Raffel
let al.
2020 (ii) Arructure-learnngg
RAT-SQL (Wang
iet al.
2019). RASAT (Qi et al. 2022). SQL (Hui et al.
2022)
.RESDSQL eLi al.) 2023a),GG GRAPHIX (Li et al.
4.3. Ablation Staudy: Q2
2023bb;
; and (tii) nncontext-learningg
PaLM-2 (Anil al.
The effect of prompting strategy In this part, we conduct comprehensive experiments to investigate the effectiveness of our proposed prompting strategy. Specifically, we compare the structure decomposing strategy used in our SGU-SQL with other prompting strategies like CoT (Wei et al. 2022) and few-shot prompting. As shown in Table and can have the following observations.
2023),
CodeX (Chen al. 202T), GPT-4 (OpenAl 2023
C3-GPT (Dong et al. 2023). ). DIN-SQL (Pourreza & Rafici
2023). DAIL-SQL Gao et al. 2023), EPI-SQL
(Liu &
Tan 2024), SuperSQL (Li et al. (2024)), E-SQL
Cafer-
oglu & Ulusoy| 2024). MAC-SQL (Wang et al.
2024).
PURPLE (Ren et al. (2024), CHESS TTlaci et al.
2024).
CHASE-SQL (Pourreza et al. 2024).
Evaluation Metrics We evaluate our models using three key metrics: Exact-Set-Match Accuracy (EM Acc), Exe- cution Accuracy (Exec Acc), and Valid Efficiency Score (VES). EM Acc compares each predicted clause to the vali- dated query, but may produce false results due to value omission. Exec Ace compares execution results of predicted and confirmed SQL queries, offering more comprehensive assessment by acknowledging multiple valid SQL solutions for a single question. VES measures the efficiency of gen- erated SQLs that produce correct result sets, discounting those that fail to retrieve accurate values. This metric com- bines execution efficiency and accuracy to provide a holistic performance evaluation.
Obs.3. Our structure-based decomposing significantly outperforms other simple prompting strategies. Our method demonstrates superior performance across all tested LLMss Specifically, compared CoT, our approach achieves average improvement of 5.03%, while outperforming few-shot prompting by 4.98% on average.
Obs.4. Our structure-based decomposing significantly outperforms other advanced prompting strategies. The key distinction of our approach is that it dynamically decom- poses queries based on their syntax structure, rather than either using fixed decomposition patterns (like DIN-SQL) or purely relying on LLM's s black-box understanding (like ACT-SQL, MAC-SQL). This syntax-aware decomposition strategy proves more effective for handling complex SQL generation tasks.
4.2. Main Results: Q1
To answer Q1, we perform extensive experiments to com- pare SGU-SQL with the strongest baselines, to the best of our knowledge. The comparison results on Spider and Bird are placed in Tables [and We summarize our experimental observations as follows.
Obs.5. Simple decomposing--aseer methods are ineffective in the text-to-SQL task. While decomposing complex tasks into subtasks like CoT, can enhance model performance in many natural language understanding tasks, it proves to be ineffective the text-to-SQL task. As shown in
Obs.1. Our proposed SGU-SQL significantly outperforms SOTA text-to-SQL baselines. We compare SGU-SQL (ours) with 11 fine-tuning based methods, 7 structure-leaminn-
7

---

Structure-Guine Guided Large Language Models for Text-to-SQL Generation
Table 2: The Execution Accuracy and Exact Match Accuracy of text-to-SQL models on SPIDER and BIRD. The best and second-best results in each column are highlighted in bold font and underlined. NaN denotes that the result is not available.
Datasct
Spider
BIRD
Metric
EX Acc
EM Acc
VES
EX Acc
EM Acc
VES
Baichuan2-7B
0.6035
0.5793
0.6082
0.1719
0.0547
0.2097
Baichuan2-13P
0.6776
0.6078
0.6545
0.1766
0.0455
0.2126
LlaMA2-7B
0.6083
0.5816
0.5795
0.1675
0.0469
0.1670
LlaMA2-13B
0.6809
0.6400
0.6712
0.1993
0.0743
0.1739
Finctuning- based
LlaMA2-70B
0,6869
0.6555
0.6779
0.2414
0.0778
0.1987
CodeLlama-78
0.7018
0.6431
0.7357
0.2370
0.1283
0.2504
Codee 13B
0.7462
0.7056
0.7391
0.2944
0.2551
0.3004
CodeLlama-77 70B
0,7710
0.7139
0.7463
0.3287
0,2557
0,3428
Qwen- 7
0.6519
0.6106
0.6625
0.1709
0.0439
0.1915
Qwen- 0.6625
0.6238
0.6757
0.2286
0.0645
0.2396
Qwen-72B 0.7127
0.6812
0.7082
0.2392
0.0894
0.2488
Structure Learning
RAT-SQL
0,6955
0.6597
0.6734
0.2639
0.2431
0,2431
BRIDGE 0.6928
0.7053
0.6893
0,2459
0.2068
0.2574
LGESQL
0.7421
0.7251
0.7067
0.2837
0.2493
0.2889
S'SQL
0.7643
0.7385
0.7539
0.2960
0,2649
0.3143
RESDSQL.
0.8182
0.7580
0.8226
0.33
0.3174
0.3286
Graphix-T5 -55
0.7562
0.7463
0.7643
0,2984
0.2538
0.3062
METASQL
0.7695
0.7288
0.7498
0.3180
0.3011
0.3225
GPT-3.5
0.7394
0.5327
0.7457
0.3562
0.304
0.3415
GPT-4
0.7665
0,5892
0.7390
0.4633
0.4255
0.4794
PaLM-2
0.6985
0.4438
0.7148
0,2735
0.2543
0.3061
In- nn-Context Learning
CodeX
0.7167
0.4905
0.7011
0.3438
0,3019
0.3496
C*-GPT
0.8108
0.7036
0.8009
0,5020
0.4143
0.5077
DIN-SQL
0.8279
0.7187
0.8173
0.5072
0,4398
0.5879
DAIL-SQL.
0,8308
0.7443
0.8317
0.5434
0,4581
0.5576
DTS-SQL
0.8269
0.7260
0.8163
0.5581
0.4825
0.6038
CodeS
0.8150
0.7069
0.8092
0,5714
0.4893
0.6120
SuperSQL
0,8682
0.7589
0.8410
0.5860
0,4745
0.6067
MAC-SQL 0.8635
0.7545
0.8541
0.5759
0.4906
0.5872
SGU-SQL 0.8795
0.7826
0.8652
0.6180
0.5144
0.6393
Tables and 8 applying COT on PaIM-2 even leads to performance decrease 1.08% compared the naive few-shot prompting. This is attributed to the complex syntax of SQL, and the intricate correspondence between query terms in user queries and database data units. Conversely, we formally define the meta-operations in SQL and propose a decomposing strategy according to the syntax tree to separate the query into subtasks. This boosts the LLMs' comprehension of linked queries generate accurate SQLs.
the decomposing-based prompts. Specifically, we replace GPT-4 used SGU-SQL with other representative generative LLMs, including PaLM-2 (Anil et al.) 2023), CodeX (Chen et al. 2021). ChatGPT and GPT-4 (OpenAl 2023 as alter- natives. Specifically, we used the model 'chat-bison-001 provided by GoogleAI as the implementation of PaLM-2, and 'ChatGPTTttrboo' and 'gpt-4' as the implementations of ChatGPT and GPT-4, respectively. The text-to-SQI. task is conducted under the few-shot setting with the query from the development of Spider as input. As shown in Figure we have the following observations.
The generalization ability of prompts
Obs.6. The performances of the original LLMs improved significantly by integrating the prompt learned from our SGU-SQL. Specifically, PaLM-2 improved by 4%, CodeX by 3%, ChatGPT by 5%, and GPT-4 by almost 11%. The substantial performance gains indicate the robustness and
To further verify the generalization ability of our proposed prompting strategy, in this part, we conduct comprehensive experiments to investigate whether SGU-SQL could enhance other LLMs by substituting their original framework with
8

---

Structure-Guidel Large Language Models for Text-to-SQL Generation
reasons for the typical failure of LLMs in Text-to-SQL. tasks. Compared the baseline model, we achieved reduction of approximately 33.5% in errors and made progress in the schema- -inemialinnkig and join statement components where traditional models often falter. In this section, we will first discuss the failure cases of our model in comparison with baseline models, and then discuss the reasons for the typical failure for LLMs in text-sql tasks. We checked the errors in the generated SQL answers and classified them into six categories, shown in Figure 2 followed by the major classification by (Pourreza & Rafiei 2023 Compared to the baseline model, we achieved reduction of approximately 33.5% errors and made progress in the schema-linking and join statement components where traditional models often falter. Errors in the schema-linking segment decreased by around 38%, primarily attributed to the utilization of Precise Query Matching, wherein graph neural networks were employed to learn and match the datahase schema. This underscores the efficacy of Structure Linking.
Schemeiedeng
21 (6P%)144
Scherra- seerg
23
Ctrih
kitl haturis
kiil haluns
18 (0S6E2)
heglol
Wolld
22 22,0056411
Jon
Goundh
OUSSISUU-SCLH GGPTT)
C3-PPT
Figure 2: Error Analysis of GPT-4 + SGU-SQL and C3-GPT on the Dev Set: A Comparison of 125 and 188 Failures.
generalization ability of our proposed prompting strategy. Furthermore, the consistent improvements across different LLMs highlight the versatility and applicability of our ap- proach in enhancing the capabilities of existing language models.
5. Conclusion
Obs.7. LLMs with stronger reasoning abilities exhibit greater improvement. We observe that LLMs with stronger reasoning abilities benefit more from integrating the prompts learned from SGU-SQL. Specifically, GPT-4, which is known for its advanced reasoning capabilities, shows a more sub- stantial performance improvement compared to PaLM-2, CodeX, and ChatGPT. This suggests that our prompting strategy is particularly effective in enhancing the perfor- mance of LLMs that require more complex reasoning tasks.
Recent advancements in large language models (LLMs) have shown promise in improving the accuracy of text-to- SQL generation. However, existing models typically in- put queries and database schemas into LLMs to perform semantic-structure matching and generate structured SQL., while often overlook the structural information inherent in user queries and databases, which could significantly enhance the generation of accurate SQL queries. This over- sight can result in the production of inaccurate or unex- ecutable SQL queries. To fully exploit the structure, we propose the Atructure-to-QQ framework (SGU-SQL), which leverages the inherent structure information to improve the SQL generation of LLMs. Specifically, SGU-SQL links user queries and databases in a structure-enhanced manner. then decomposes complicated linked structures with syn- tax trees guide the LLM to generate the SQL step by step. Extensive experiments two benchmarks demon- strate that SGU-SQL consistently outperforms state-of-the-art SQL generation baselines.
4.4. Model Analysis
Difficulty analysis Q3: In this part, first analyze the performance of our proposed method on queries with differ- ent levels of difficulty. Our analysis focused on evaluating the performance of our proposed method across queries of provides a comparative assessment of our method against state-of-the-art (SOTA) prompting methods on the Spider development set. Our findings reveal that our method consistently outperforms competing methods across all difficulty levels. Notably, we observe the most substantial improvements in the extra hard and hard classes, where other prompting models struggle. Additionally, our method also shows a slight improvement in the easy class, which suggests that our method is robust and effective across queries of different difficulty levels, highlighting its potential for practical applications in natural language understanding and query generation tasks.
varying difficulty levels. Table |
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Error analysis Q4:
We checked the errors in the generated SQL answers and classified them into six categories, as shown in Figure 2 following the classification by (Pourrezal & Rafici 2023). We discuss the failure cases of our model in comparison with baseline models and then discuss the
A Acknowledgement
The work in this paper was fully supported by grant from the Research Grants Council of the Hong Kong Special Ad- ministrative Region, China (Project No. PolyU 25208322).

---

Structure-Guided Large Language Models for Text-to-SQL Generation
References
Chen, S., Zhang, Q. Dong, J., Hua, W., Li, Q., and Huang, X. Entity alignment with noisy annotations from large lan- guage models. arXiv preprint arXiv:2405. 16806, 2024b,
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models self-debug. In International Conference on Learning Representations (ICLR), 2024c.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. et al. Palm 2 technical report. arXiv preprint arXiv:2305. 2023.
Choi, D., Shin, M. C., Kim, E., and Shin, D. R. Ryansql: Re- cursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. Computational Linguistics, 2021.
Bi B., Liu, S., Mei, L., Wang. Y., Ji, P., and Cheng, X. De- coding by contrasting knowledge: Enhancing llss' confi- dence edited facts. arXiv preprint arXiv:2405.1166 2024.
Devlin, J., Chang, M.- W.,W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019.
Bi, B., Liu, S., Wang, Y., Xu, Y., Fang, J., Mei, L., and Cheng, X. Parameters context: Fine-grained control of knowledge reliance in language models. arXiv preprint arXiv:2503. 158888 2025.
Dong, J., Zhang. Q., Zhou, C., Chen, H., Zha, D., and
Huang, X. Cost-efficient knowledge-baser question an- swering with large language models. arXiv preprint a4X5: 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D.,
Dhariwal, P., Neelakantan. A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Horbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances Neurul Information Processing Systems (NeurIPS), 2020.
Dong, X., Zhang, C., Ge, Y., Mao, Y., Gao, Y., Chen, Lin, J., and Lou, D. C3: Zero-shot text-to-sql with chatgpt, 2023.
Dou, L., Gao, Y., Liu, X., Pan, M.. Wang, D., Che, W., Zhan, D., Kan, M.-Y., and Lou, J.-G. Towards knowledge- intensive text-to-SQL semantic parsing with formulaic knowledge. Empirical Methods in Natural Language Processing (EMNLP), 2022.
Busbridge, D., Sherburn, D., Cavallo, P., and Hammerla, N. Y. Relational graph attention networks. arXiv preprint arXiv: 2019.
Fang, J., Wang, Y., Wang, R., Yao, 7., Wang, K., Zhang, A., Wang. X., and Chua, T.-S. Safemlrm: Demystifying safety in multi- large reasoning models. arXiv preprint arXiv:2504. 08813, 2025.
Caferoğlu, H. A. and Ulusoy, Ö. E-sql: Direct schema linking via question enrichment in text-to-sql. arXiv preprint 4rXiv:2409.1673
2024.
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. Text-to-sql empowered by large lan- guage models: A benchmark evaluation. arXiv preprint arXhv:2308. 15363, 2023.
Cao, R., Chen, L., Chen, Z., Zhao, Y., Zhu, S., and Yu, K. Lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations. arXiv preprint arXiv:2/06.01093, 2021.
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, Text-to-sql empowered by large language mod- els: benchmark evaluation. In International Conference on Very Large Data Bases (VLDB), 2024.
Chang, S. and
How to prompt LLMs for text-to-SQL: A study in zero-shot, single-domain, and cross-domain settings. In NeurlPS 2023 Second Table Representation Learning Workshop (NeurIPS), 2023.
Guo, C., Tian, Z., Tang, J., Wang, P., Wen, Z., Yang, K., and Wang. T. case- based reasoning framework for adaptive prompting in cross doms-n text-to-sql. arXiv preprint arXiv:2304. 1tt.23344.3331,. 2023.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint ar 1rxiv:2107.03377 2021.
Guo, J., Zhan, Z., Gao, Y., Xiao, Y., Lou, J.-G., Liu, T., and Zhang, D. Towards complex text-to-sql in cross-domain database with intermediate representation. arXiv preprint arXiv: /005 08205, 2019.
Chen, S., Zhang, Q., Dong, J., Hua, w., Cao, J., and Huang, X. Neuro-symbolic entity alignment via variational infer- ence. arXiv preprint 2xXiv:24/0.044555 2024a.
10

---

Structure-Guided Large Language Models for Text-to-SQL Generation
Hochreiter, S. and Schmidhuber, Long short-term memory. Neural Computation, 1997.
database interface? a big bench for large-scale database grounded text-to-sqls. arXiv preprint arXiv:2305.03/ 2023c.
Hong, Z., Yuan, Z.,, Chen, H., Zhang, Q. Huang, F., and Huang, X. Knowledge-to-sql. Enhancing sql generation with data expert Ilm. arXiv preprint arXiv:2402.1155 2024a.
Liu, A., Hu, X., Wen, L., and Yu, P. S. A comprehensive evaluation of chatgpt's zero-shot text-to- sql capability, arXiv preprint arXiv:2303.13547 2023.
Hong, Z.. Yuan, Z., Chen, H.. Zhang, Q. Huang, F., and Huang, X. Knowledge-tow Enhancing sql generation with data expert Ilm. arXiv preprint urXiv:2402.11577, 2024b.
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L.,, and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint arXiv: 0et::22110060000 06804, 2021.
Liu, X. and Tan, Z. Epi-sql: Enhancing text-to-sql trans- lation with error-prevention instructions. arXiv preprint arXiv:2404. 2024.
Hong, Z., Yuan, Z., Zhang, Q. Chen, H., Dong, J., Huang, F. and Huang, X. Next-generation database inter- faces: A survey of llm-based text-to-sql. arXiv preprint DxXiv:4406008426. 2024c.
Mahmud, T., Hasan, A., Ahmed, M., and Chak, T. H. A rule based approach for nlp based query processing. In International Conference Electrical Information and Communication Technologies (EICT), 2015. 2015.
Hui, B., Geng, R., Wang, L.,, Qin, B., Li, B., Sun, J., and Li, Y. Sisql: Injecting syntax to question-schema interaction graph encoder for text-to-sql parsers, 2022.
Müller, M. and Sennrich, R. Understanding the proper- ties of minimum Bayes risk decoding neural machine translation. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACI-IJCNI.)), 2021.
Kate, R. Transforming meaning representation grammars to improve semantic parsing. In CoNLL 2008: Proceed- ings of the Twelfth Conference on Computational Natural Language Learning, pp. 33-40, 2008
Lei, F., Chen, J., Ye, Y.. Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., Yin, P., et al. Spider 2.0: Evaluat- ing language models on real-world enterprise text-to-sql workflows. arXiv preprint arXhv:2411.07760 2024.
Nan, L., Zhao, Y., Zou, W., Ri, N., Tae, J., Zhang, E., Cohan, A., and Radev, D. Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. arXiv preprint 2rXiv:2305.125533 2023.
Li, B., Luo, Y., Chai, C., Li, G., and Tang, N. The dawn of natural language to sql: Are we fully ready? arXiv preprint 4rXiv:2406. 01265, 2024a.
Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-t,, Wang, S. I., and Lin, X. V. Lever: Learning to verify language- to-code generation with execution. In International Con- ference on Machine Learning (ICML), 2023.
Li, F. and Jagadish, H. V. Constructing an interactive natural language interface for relational databases. In Interna- tional Conference on Very Large Data Bases (VLDB), 2014.
OpenAL. Gpt-4 technical report, 2023.
Pourreza, M. and Rafiei, D. DIN-SQL: Decomposed in-context learning of text-to-QLL with self-correction. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Li, H., Zhang, J., Li, C., and Chen, H. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 13067-13075, 2023a.
Pourreza, M., Li, H., Sun, R., Chung, Y., Talaei, S., Kakkar, G. T., Gan, Y., Saberi, A., Ozcan, F., and Arik. S. O. Chase-sql: Multi-path reasoning and preference opti- mized candidate selection in text-to-sql. arXiv preprint arXiv:2410.01944 2024.
Li, H., Zhang, J.. Liu, H., Fan, Zhang, X., Zhu, J.. Wei, R., Pan, H., Li, C., and Chen, H. Codes: Towards building open-source language models for text-to-sql. In Confer- ence on Management of Data (SIGMOD). 2024b.
Qi, J., Tang, J., He, Z., Wan, X., Cheng, Y., Zhou, C., Wang, X., Zhang, Q., and Lin, Z.. Rasat: Integrating relational structures into pretrained seq2seq model for text-to-sql. arXiv preprint arXiv:2205.0698. 2022.
Li, Hui, B., Cheng, R., Qin, B., Ma, C.,, Huo, N., Huang, F, Du, W., Si, L., and Li, Y. Graphix-t5: Mixing pre- trained transformers with graph-aware layers for text-to- sql parsing. arXiv preprint arXiv:2301.077
2023b.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. OpenAl blog, 2018
Li, J., Hui, B., Qu, G., Li, B., Yang, J., Li, B., Wang, B., Qin, B., Cao, R., Geng, R., et al. Can mlm already serve as

---

Structure-Guided Large Language Models for Text-to-SQL Generation
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511 2020..
Wang, C., Liu, Y., Li, B., Zhang, D., Li, Z., and Fang, J. Safety in large reasoning models: A survey. arXiv preprint arXiv:2504.17704, 2025.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 01824-24837, 2022.
Rajkumar, N., Li, R., and Bahdanau, D. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.0 2004092004004498, 2022.
Ren, T., Fan, Y., He, Z., Huang, R., Dai, J., Huang, C., Jing, Y., Zhang, K., Yang. Y., and Wang, X. S. Purple: Making large language model better sql writer. arXiv preprint arXiv:2403.200/ 2024.
Xu, F., Wu, Z., Sun, Q., Ren, S., Yuan, F., Yuan, S., Lin, Q., Qiao, Y., and Liu, J. Symbol-llm: Towards foundational symbol-centric interface for large language models. arXiv preprint arXiv:2311 1.00928, 2024.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y. Liu, J., Remez, T., Rapin, J., al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Yang, C., Wang, X., Jiang, J., Zhang, Q., and Huang. X. Evaluating world models with Ilm for decision making. arXiv preprint arXiv:2411.088 1.08794, 2024.
Yin, P., Neubig, G., Yih, W.-t,, and Riedel, S. Tabert: Pre- training for joint understanding of textual and tabular data. arXiv preprint 2xXix:0005083114 2020.
Scholak, T., Schucher, N., and Bahdanau, Picard: Parsing incrementally for constrained auto-regressive decoding from language models. arXiv preprint arXiv:2109.05093 2021.
Yu, T., Zhang, R., Yang, K., Yasumaga, M., Wang. D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text- to-SQI. task. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., and Wang. S. Natural language to code translation with execution. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
Sun, R., Arik, S. O., Nakhost, H., Dai, H., Sinha, R., Yin, P., and Pfister, T. Sql-palm: Improved large lan- guage modeladaptation for text-to-sql. arXiv preprint arXiv:2306.000733 2023.
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma. J., Li, I., Yao, Q. Roman, S., Zhang, Z., and Radev, D. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text- to-sql task, 2019.
Talaci, S., Pourreza, M., Chang, Y.-.,, Mirhoseini, A., and Saberi, A. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405. 16755, 2024.
Yu, T. Wu, C.-S., Lin, X. V., bailin wang, Tan, Y. C., Yang, X., Radev, D., richard socher, and Xiong, C. Grappa: Grammar-augmenttdd pre-training for table semantic pars- ing. In International Conference on Learning Represen- tations (ICLR), 2021.
Trummer, I. Codexdb: Synthesizing code for query pro- cessing from natural language instructions using gpt-3 codex. Proceedings of the VLDB Endowment, 15(11): 2921-2928 2022.
Yuan, Z., Chen, H., Hong. Z., Zhang, Q., Huang, F., and Huang, X. Knapsack optimization-base schema link- ing for llm-based text-to-sql generation. arXiv preprint arXiv: 2..2. /291 2025.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. Atten- tion all you need. In Advances Neural Information Processing Systems (NeurIPS), 2017.
Zhang, Q. Dong, J., Chen, H., Zha, D., Yu, Z.,, and Huang, X. Knowgpt:
Wang, B., Shin, R., Liu, X., Polozov, O., and Richardson, M. Rat-sql: Relation-awe aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv: /91 19111.04000 1,0492,, 2019.
Knowledge graph based prompting for large language models. Advances in Neural Information Processing Systems, 37:6052-6080, 2024.
Zhang, Q., Chen, S., Bei, Y., Yuan, Z.,, Zhou, H., Hong, Z., Dong, J., Chen, H., Chang, Y., and Huang, X. A survey of graph retrieval-augmenter generation for customized large language models. arXiv preprint arXiv:2501.1395 2025.
Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.- Yin, D., Sun, X., et al. Mac-sql: A multi-agent collaborative framework for text-to-sql. arXiv preprint arXiv:2312. 1242, 2024.
12

---

Structure-Guided Large Language Models for Text-to-SQL Generation
Zhong, V., Xiong, C., and Socher, R. Seq2sql: Generating structured queries from natural language using reinforce- ment learning. arXiv preprint arXiv: 00103, 2017.
Zhou, C., Du, J., Zhou, H., Chen, H., Huang. F., and Huang, X. Text-attributed graph learning with coupled augmenta- tions. In Proceedings of the st International Conference Computational Linguistics, pp. 10865-10876, 2025.
Thou, H., Zhou, S., Chen, H., Liu, N., Yang, F., and Huang, X. Enhancing explainable rating prediction through anno- tated macro concepts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11736-11748 2024.
Zhuang. A., Zhang, G., Zheng, T., Du, X., Wang, J., Ren, W., Huang, S. W., Fu, J., Yue, X., and Chen, W. Structim: Towards building generalist models for structured knowl- edge grounding. arXiv preprint arXiv:2402. 16671, 2024.
13

---

Structure-Guided Large Language Models for Text-to-SQL Generation
A. Ablation Study
In this section, we have conducted detailed experiments to validate the effectiveness of each component in SGU-SQL.
A.1. The Effect of Structure Learning
As shown in Table 3 we have the following observations: (i) Removing the query graph representation leads to significant performance drops (-3.45% on Spider-dev, -2.87% on BIRD-dev), demonstrating that our proposed query graph is crucial for understanding the intent behind the query. (ti) The ablation of the database graph results in performance decreases of 14% on Spider-dev and -3.54% on BIRD-dev. The larger performance drop on BIRD (-3.54%) vs Spider (-2.14%) indicates that graph-based database representation is particularly important for complex, realistic databases. (fit) When removing structure-aware linking, we observe substantial performance degradation (-5.33% on Spider-dev, -6.49% on BIRD-dev), representing the second-largest impact among all components. The more significant drop on BIRD emphasizes that our linking mechanism particularly crucial for complex queries and databases, effectively bridging the semantic gap between natural language and database components while maintaining structural integrity.
Table Ablation study on different components of SGU-SQL.
Variant
Full Model w/o query graph w/o database graph w/o structure linking w/o decomposition
SPIDER-dev
87.95
84,50 (-3.45)
85.81 (-2.14)
82.62 (-5,33)
82.35 (-5.60)
BIRD-dev
61.80
58.93 (-2.87)
58.26 (-3.54)
55.31 (-6.49)
53.78 (-8.02)
A.2. The Effect of Syntax--asee Decomposition
To verify the effectiveness of our syntax- based decomposition strategy, conducted additional experiments to compare SGU-SQL with other advanced decomposition- based methods, including DIN-SQL, ACT-SQL and MAC-SQL.
Table 4: Performance comparison between SGU-SQL. and advanced decomposition methods.
Text-to-SQL DIN-SQL ACTSSLL MAC-SQL SGU-SQL
SIPDER-dev 82.79
82,90
86.35
87.95
As shown Tables 3 and 4 (1) the ablation of our decomposition strategy leads to the most significant performance decrease (-5.60% on Spider-dev, -8.02% on BIRD-dev). These results validate our approach of breaking down complex queries into manageable components while preserving structural relationships, especially beneficial for real-world applications involving complex dabase structure and intricate query patterns. (ii) Our SGU-SQL achieves 87.95% execution accuracy on SPIDER-dev, outperforming all these methods. The key distinction of our approach is that it dynamically decomposes queries based on their syntax structure, rather than either using fixed decomposition patterns (like DIN-SQL) or purely relying on LLM's black-box understanding (like ACT-SQL, MAC-SQL). This syntax-aware decomposition strategy proves more effective for handling complex SQL generation tasks.
A.3. The Effect of Backbone LLMs
For a thorough evaluation of SGU-SQL's performance, we conduct additional experiments on BIRD dev with different LLMs as backbones. Specifically, we compared SGU-SQL against two categories of methods: (i) Open-source models with available paper and codes: MAC-SQL, SQ,, E-SQL and CHESS; and (ii) Undisclosed methods that have demonstrated strong performance: PURPLE, Distillery and CHASE-SQL.
As shown in 3 our SGU-SQL achieves competitive performance across different LLM backbones. Specifically, have the following observations: Using GPT-4 as the backhone, SGU-SQL achieves the best performance compared to other models using the same backbone. With GPT-4o, SGU-SQL achieves 69. 69.28% in terms of execution accuracy, outperforming several strong baselines: PURPLE (68. 12%). CHESS (68.31%) E-SQL (65.58%) and Distillery (67.21%) The only model showing higher performance is CHASE- SQL (released in October 2024), which uses Gemini 1.5 Pro as its backbone. Notably, CHASE-SQL incorporates query fixer module that leverages database execution feedback guide L.LMs to
14

---

Structure-Guided Large Language Models for Text-to-SQL Generation
iteratively refine generated queries. In contrast, our model generates SQL queries in single pass without utilizing any execution feedback.
B. Model Analysis
B.1. Performance on more Challenging Dataset
To further verify the effectiveness of our model, we conduct additional experiments on more challenging datasets, like Spider 2.0-Snow and Spider 2.0-Lite (Lei et al. 2024). As shown in Table while the performances are relatively low across all models, SGU-SQL consistently demonstrates better capability handling complex SQL generation tasks in both single and multi-database scenarios.
B.2. Efficiency analysis
To assess our approach thoroughly, we conducted the efficiency analysis on the BIRD dataset, large-scale benchmark in text-to-SQL research with 12,751 unique question-SQL pairs across 95 databases (33.4 GB total). Given that the queries in this dataset categorized into 3 difficulty levels: simple, moderate, and challenging, we specifically tested our model on the challenging set of the BIRD dataset and compared its performance with DIN-SQL and MAC-SQL.
As shown in Table our model demonstrates superior performance while maintaining competitive computational efficiency. Specifically, our model requires less time for both training and inference. This superior efficiency can be attributed to our graph- architecture. While baseline methods avoid the overhead of graph construction, they heavily rely on prompt-base based modules that require multiple calls LLMs like GPT-4. These API calls introduce substantial latency that accumulates during both the training and inference phases. In contrast, our graph-based approach, despite its initial graph construction overhead, achieves faster end-to-end processing by minimizing dependence on time-consuming API calls.
B.3. Difficulty analysis Q3
In this part, first analyze the performance of our proposed method on queries with different levels of difficulty. Our analysis focused on evaluating the performance of our proposed method across queries of varying difficulty levels. Table I provides comparative assessment of our method against state-of-the-an (SOTA) prompting methods on the Spider development set. Our findings reveal that our method consistently outperforms competing methods across all difficulty levels. Notably, we observe the most substantial improvements in the extra hard and hard classes, where other prompting models struggle. Additionally, our method also shows a slight improvement in the easy class, which suggests that our method is robust and effective across queries of different difficulty levels, highlighting its potential for practical applications in natural language understanding and query generation tasks.
B.4. Error Analysis Q4
We checked the errors the generated SQL answers and classified them into six categories, as shown in Figure following the classification by (Pourreza & Rafiei 2023). We discuss the failure cases of our model in comparison with baseline models and then discuss the reasons for the typical failure of L.MSs in Text-to-SQI. tasks. Compared to the baseline model, we achieved reduction of approximately 33.5% in errors and made progress in the schema-linking and join statement components where traditional models often falter. In this section, we will first discuss the failure cases of our model in comparison with baseline models, and then discuss the reasons for the typical failure for LLMSs in text-sql tasks. As shown in Figure 2 we checked the errors in the generated SQL answers and classified them into six categories.
Compared the baseline model, we achieved reduction of approximately 33.5% in errors and made progress in the
Table 5: Performance comparison on BIRD dev with different LLMs as backbones.
Backbone
MAC-SQL
PURPLE
E-SQL
CHESS
Distillery CHASE-SQL SGU-SQL
GPT-4
59.59
60.71
58.95
61.37
61.80
GPT-4o
65.05
68.12
65.58
68.31
67.21
69.28
Gemini-15 1.5 Pro
73.14
15

---

Structure-Guided Large Language Models for Text-to-SQL Generation
schema-linking and join statement components where traditional models often falter. Errors in the schema-linking segment decreased by around 38%, primarily attributed the utilization of Precise Query Matching, wherein graph neural networks were employed to learn and match the database schema. This underscores the efficacy of Structure Linking, In the sections prone to errors, such as Group-by and Join, our errors decreased by 35%, indicating that our syntax tree decomposing enables the model to more accurately utilize corresponding SQL. Meta-operations to mine the intentions behind queries, thus further enhancing the accuracy in identifying the targeted tables or columns for manipulation.
To further analyze the reasons for errors in the baseline model, we conducted a comprehensive case study by comparing the results of the baseline model with those of our model, as shown in Figure
Subtask Decomposing LLMs often do not adequately break down the task into its essential steps for reasoning. For example, in Case 1, the primary subtask of linking flight data to specific cities was ignored. The question did not adequately break down the task into its essential components without further guidance from LLMs. In Case 3, the query did not decompose the task into two separate subtasks to identify semesters with Masters and Bachelors enrollments independently which also leads to wrong returned answers.
Intention Understanding LLMs sometimes misunderstand the core intention of the question. In Case 2, LLMs fail to identify the intention that the question is trying to find all countries where English is spoken, regardless of its official status which leads to errors. concentrated on the official language status, which did not align with the broader objective of considering songlish-spaaking countries in general. In Case 1, the query was centered around airport codes (SourceAirport), misinterpreting the intention identify the busiest city, not just the airport. In Case 3, LLM misinterprets the intention of finding how many likes Kyle has received. It erroneously assumes the task is to count how many likes Kyle has given, not received.
Data Schema Linking Since LLMs get data schema information with plain text inputs, it might be challenging to reason the right linking strategy to solve the problem correctly. It needs to understand the referenced tables and columns in the question which are often being mentioned in an inexplicitly way, then matching with the database schema. In contrast, our tailored GNN model can handle this situation well. In Case 1, the initial query failed to incorporate the airport's 's table, which was essential for linking airport codes to their respective cities. In Case 3, the query did not effectively link degree program types (Masters, Bachelors) to semesters in databases in a way that would allow for the inclusive identification of valid semesters. There was also misalignment in linking: student_id id from the Likes table was incorrectly associated with the id in Highschooler table. It should link liked_ id from Likes to id in Highschooler to align with the task's s objective.
C. Preliminaries
C.1. Structure Learning for Text-to-SQL.
Definition 1. Structure Learning for Text-to-SQI: Given natural language query and a database schema Q, the task of graph learning for Text-to-SQ) SQL aims generate a graph-based representation that captures the structural and semantic relationships between the query and the schema, and to learn a mapping function G, - Ga. where G, the structural user queries, and Ga is the corresponding database contents linked to the query Gq.
Let G = (V, E) denote the graph representation, where V is the set of nodes and E is the set of edges. The nodes v E V/ represent the entities and components in the query and schema, such as tables, columns, and query tokens. The edges e E E represent the relationships and dependencies between the nodes. The graph learning task involves two main components,
Table 6: Execution accuracy for baseline methods on Spider 2.0-Snow and Spider 2.0-Lite.
Datasets
DAIL-SQL+GPT-4 CHESS+GPT-40 SGU-SQL+GPT-40
Spider 2.0-Snow
2.20
1.28
4.39
Spider 2.0-Lite
5.68
3.84
6.40
16

---

Structure-Guided Large Language Models for Text-to-SQL Generation
NL Query:
What is the code of airport that has the highest number of flights?
Given the Following database schema: //
CREATE TABLE 'flights' Airline INTEGER,
Prompts from
Baseline Model
/+ Anszer the follouing: What the code of airporte/; that has the highest number flights? */; Let's think step by step.
Results from Baseline Model's Prompt
SELECT Sourcekirport, COUNT(+) AS NumberOfFlights FROM flights CROUP BY SourceAirport
ORDER BY NumberOtFlights DESC LIMIT 1;
SELECT T1 T1.AirportCode FROM AIRPORTS AS T1 JOIN FLIGHTS OR Airporott 12.Sourcehirport
r1.AirportCode T2. 12.Destrirport
Gold SQL
GROUP A1.Arrportoode
ORDER count(*) DESC LIMIT 1;
(a) Case Airports Database Question
NL Query:
How many people live in countries that do not speak English?
/* Given the following database schema: +/
CREATE TABLE countrylanguage ( CountryCode c3ar(3) NOT NULL DEFAULT ",
Prompts from
Baseline Model
FOREIGH KEY (CountryCode) REFERENCES country (Code)); /* Ansuer the following: What the code of airport that has the highest nunber of flights? */ Let's think step by step.
SELECT SUM(c. Populatot))))on) FROM country WHERE c. Code NOT IN
Results from Baseline Model's Prompt
SELECT cl. Countryoodd FROM countrylanguage el WHERE La.gugge "English' AND cl. 'T' );
SELECT sum(Population) FROM country WHERE Name
SELECT T1. Name FROM country AS T1
Gold SQL
JOIN countrylanguage AS T2
OX Code = T2. Countrydeeodd
WHERE T2. Language "English" );
(b) Case 2: Languages Database Question
NL Query:
Return the number of likes that the high schooler named Kyle has.
/+ Given the following database schema: CREATE TABLE Likes( student_id int,
Prompts from Baseline Model
FOREIGN KEY (CountryCode) REFERENCES country (Code)) /+ Ansver the following: Return the number of likes that the high schooler named Kyle has. +/
Let's think step by
SELECT count(*) FROM Likes JOIN Highschooler ON L.kees.iked_.5 Highschooler. WHERE Highschooler name name 'Kyle');
Results from Baseline Model's
Prompt
SELECT c*unt(*) FROM Likes ROM country AS JOIN Highschooler
Gold SQL
Ti.dtudent__d T2. WHERE
"Kyle";
(c) Case Social Network Database Question
Figure 3: NL query from Spider and the corresponding results from different prompting approaches
17

---

Structure-Guidel Large Language Models for Text-to-SQL Generation
including graph construction and graph representation learning.
C.1.1. GRAPH CONSTRUCTION
The first step is to construct the graph 9 from the query Q and schema D. This involves extracting relevant entities and relationships from the input and organizing them into a graph structure. The graph construction process can be formally defined as:
Construct(Q. D);
(19)
where Construct(-) (-) is method that maps the query and schema to the graph representation.
C. 1.1.2. GRAPH REPRESENTATION LEARNING
Once the graph is constructed, the next step is to learn meaningful representations the nodes and edges in the graph. This typically achieved using Graph Neural Networks (GNNs), which propagate information across the graph structure to capture the structural and semantic relationships. The representation learning process can be formally defined as:
h (1+1) GNN(h (h. : (
(20)
where h is the representation of node at layer I, (v) is the set of neighboring nodes of v, and GNN(-) is the graph neural network function that updates the node representations based their neighbors. The learned graph representations then used to generate the corresponding SQL query S by applying decoding function f to the graph:
S = (())
(21)
The objective of graph learning for Text-to-SQL is optimize the parameters of the graph construction and representation learning components, well the decoding function, to generate accurate and executable SQL queries from natural language queries and database schemas.
C.2. Text-to-SQL. Generation with LLMs
We now formally define the problem of text-to- tt- generation. Let 'D be database schema consisting of set of tables T T2 ... T. J. where each table T, has set of columns C, {C11, Cr...... C.m The database schema D can be represented as a tuple (T,C) C), where C = U C; is the set of all columns across all tables.
Given a natural language query Q and a database schema D, the task of Text-to-SQL generation aims to translate Q into a corresponding SQL query S that accurately retrieves the desired information from the database. Let M be the LLM that maps the natural language query Q and the database schema DD to the target SQL query S, the main objective can be formulated follows:
M: (Q, S.
(22)
The objective LLM-based text-to-SQL generation to learn the optimal parameters or prompts 0 that minimize the difference between the generated SQL query M(Q, D)0) and the ground truth SQL query S:
0r arg min ((M(Q,
(23)
where L is a loss function that measures the discrepancy between generated and ground truth SQLs.
Table 7: Efficiency analysis the 'Challenging' set of BIRD.
Text-to-SQL
Backbone LLM
Training Time Inference Time Performance
DIN-SQL
GPT-4
4.69
0.39
36.7%
MAC-SQL
GPT-4
4.98 h
0.36 h
39.3%
SGU-SQL
GPT-4
3.47
0.22
42.1%
18

---

Structure-Guided Large Language Models for Text-to-SQL Generation
D. Related Work
Text-to-SLL has witnessed significant evolution over the past few years (Hong et al. 2024c). Early researchers focused on well-designed rules, which were later superseded by deep learning- based techniques. More recently, the integration of pre-trained language models (PLMs) and large language models (LLMs) (Wang al. [025 Fang et al. (025) has further advanced state-of- the-art text-to-SQL generation. This section traces the developmental trajectory of text-to-SQL methods, highlighting the key milestones and innovations that have shaped the field.
D.1. Traditional Text-to-SQL Methods
Text-to-SQL has witnessed significant advancements in recent years. Early research heavily relied on well-designed rules and templates (Li & Jagadish 2014] Mahmud et al.) 2015: Yu et al., 2021). which were suitable for simple database scenarios. How- ever, the increasing complexity of database structure and the high labor costs associated with rule-based methods have made such approaches impractical. The advent of deep neural networks, such Scquence-to-sequee
CP74-55UU-0L
Churge
Changet
models and incodrrddcoodd structures like LSTMs (Hochreiter & Schmidhuber] (997) and Transform- ers (Vaswani et al. 2017). has revolutionized the field of text-to- SQL (Guo et al., 2019 Choi et al. 2021). They automatically learn a mapping from user queries to corresponding SQL queries. Typically, RYANSQL. (Choi et al. 202T) introduced intermediate representations and sketch-based slot filling to handle complex questions and improve cross-domain generalization. More recently, pre-trained language models (PLMs) with strong semantic parsing capabilities have become the new paradigm of text-to-SQL systems. The initial adoption of PLMs in Text-to-SQL primarily focused fine-tuning off-the-shelf models, such as BERT (Devlin et al. 2019) and RoBERTa, on standard text-to-SQL. datasets (Yu et al.) 2018) Zhong et al. (017).
10% 60%
70% 80% 90% 100%
Figure 4: Ablation Study of SQU-SQL on SPIDER.
Incremental research on PLM-based optimization, such as table content encoding (Guo et al. 2019 Yin et al. 2020
Dou
et al. 2022). and schema information incorporation (Li et al. 2023a). has further advanced this field.
D.2. LLM-based Text-to-SQL Models
Large language models (LLMs), such as GPT series (Radford et al. 2018 Brown et al. 2020 Achiam et al. 2023).
have gained significant attention in recent years due to their capability to generate coherent and fluent text (Zhou
jet al., 2025 2024] [hen et al., 2024b Dong et al.] 2024: Bi et al.
2024 2025 Yang et al. 2024: Chen et al. 2024a).
Researchers have started exploring the potential of LLMs for text-to-SQL. by leveraging their extensive knowledge
reserves and superior generation capabilities (Rajkumar et al. 2022
Gao et al. 2024). These approaches often involve
fine-tuning the open-source LLMs on text-to-SQL datasets (Anil et al. 2023 Hong et al. (024b) or prompt engineering to
guide the closed-source LLMs in generatit (Chang & Fosler-L.ussier
2023 Pourreza & Ratei 2023
Gao et al. (022)).
1.2.1. FINE-TUNING LLMS FOR TEXT-TO-SQI.
Recently, the emergence of large language models (LLMs) has markedly altered the landscape for text-to-SQL tasks, LLMs, with their capacity for understanding and generating human-like text, present a robust solution for text-to-SQL applications (Liu et al. (023). The development of LLMs typically encompasses pre-training followed by fine-tuning, Research has concentrated on fine-tuning with domain-specific data and optimization techniques to enhance base models for
Table 8: Ablation Study: Performance comparison of different prompting strategies on the development set of Spider
Prompting strategy
PaLM-2
CodeX
ChatGPT
GPT-4
Few-shot Prompting
0.6985
0.7167
0.7394
0.7665
+ CoT Prompting
0.6873
0.7198
0.7552
0,7834
SGU-SQL
0.7395
0.7418
0.7846
0,8795
19

---

Structure-Guided Large Language Models for Text-to-SQL Generation
coding tasks, including text-to-SQL. This process enables models master programming language syntax and database
schema intricacies (Raffel et al. 2020 Roziere al. 2023). Through training on tailored datasets of annotated SQL queries. LLMs acquire the syntax and structure necessary for generating compliant SQL code [Trmmeer 2022 Sun et al. 2023) Furthermore, PICARD (Scholak et al. (021) introduced decoding mechanism for LLMs that ensures the generation of valid sequences by discarding inadmissible tokens at each step, employing incremental parsing to guarantee the validity of SQL queries produced by autoregressive language models. More recently data-augmented fine-tuning techniques have emerged as a promising approach to improve text-to-SQL generation models. By focusing on enhancing the quality and diversity of the training data during supervised fine-tuning, these methods enable models to better capture the complexities of translating natural language queries into SQL statements. For example, Symbol-LLM (Xu et al.) 2024) proposes a two-stage approach, consisting of an injection stage and an infusion stage, for data-augmented instruction tuning. This method effectively incorporates additional data to improve the L.M's ability to follow instructions. Similarly, CodeS (Li) et al., 2024b) leverages ChatGPT to generate bi-directional training data, augmenting the model's training dataset and enhancing its code generation capabilities. Additionally, StructLM (Zhuang et al., (024)) introduces a training paradigm that involves multiple structured knowledge tasks, aiming to improve the model's overall performance across a wide range of applications. These approaches demonstrate the potential of data augmentation and multi-task learning in boosting the performance of LLMs.
D.2.2. IN-CONTEXT LEARNING FOR TEXT-TO-SQL
In-context learning enhances LLM performance by providing detailed task instruction, background knowledge, and contextual examples during inference, thereby improving performance for specific tasks. This approach has seen innovative applications in text-to-SQL, with strategies aimed at optimizing prompt contents and formats based on user queries and database structures. Typically, C3-SQL (Dong et al.) (023) designed a zero-shot prompting framework for ChatGPT with clear prompting for effective input format and tailored hints for calibration and consistency checking during the query generation. KATE (Liu et al. 202T) first investigated the impact of few-shot examples on GPT-3's performance. (Nan et al. 2023) further conducted systematic investigation into different demonstration selection methods and optimal instruction formats for prompting LLMs in the text-to-SQI. task, whereas DESEM (Guo et al. 2023) developed a domain- specific vocabulary masking technique, called similarity assessment, highlighting the relevance of SQL-specific terms. DIN-SQL (Pourreza& & Rafiei 2023 introduced a decomposed framework, categorizing user queries by complexity and breaking down the generation task into sub-problems and feeding the solutions of those sub-problems into LLMS to improve the generation performance of complex SQL. queries. DAIL-SQLI. (Gao et al. 2024) further enhanced the performance by incorporating suitable formatting of the database schema and selecting examples based on skeleton similarities. Some recent work improves the in-context learning framework by incorporating execution feedback through second-round prompting for regeneration. For example, MRC-EXEC (Shi et al. (022) introduced a natural language to code translation framework with execution, which executes each sampled SQL query and selects the example with the minimal execution result-based Bayes risk (Müller& & Sennrich| 2021). LEVER (Ni et al. 2023) proposed an approach to verify NL2Code with execution, utilizing a generation and execution module to collect sampled SQL set and their execution results, respectively, then using a learned verifier to output the probability of the correctness. Similarly, the SELF-DEBUGGING (Chen et al., (024c) framework is presented to teach LLMs to debug their predicted SQL via few-shot demonstrations. The model can refine its mistakes by investigating the exccution results and explaining the generated SQL in natural language without human intervention.
D.3. Structure Learning for Text-to-SQL
Structure learning-based models, particularly those utilizing Graph Neural Networks (GNNs), have emerged as powerful approach to modeling the complex relationships between user queries and database schemas in text-to-SQL generation. By organizing information into graph structures and leveraging GNNs to learn rich structural representations, these methods enhance the semantic understanding and generalization abillity of text-to-SQL. models. Specifically, RATSQL (Wang et al., 2019) employs a graph-based structure to delineate relationships within database schemas and queries, treating the schema as a graph of tables and columns connected by relational edges. LGESQL (Cao et al.] 202[) introduced an edge-centric graph model derived from conventional node-centric graphs, to capture diverse structural topologies. S2SQL. (Hui et al. 2022). integrates syntactic dependency information into a question-schema interaction graph, focusing on primary relationships to mitigate overfitting while emphasizing essential graph structures. Graphix-T5 (Li et al. 2023b) explored the integration of GNN layers into the large language model T5 (Raffel et al., 2020), aiming to leverage both semantic and structural information from PLMs and GNNs, respectively. RESDSQL (Li et al., (023a) designed a vanking-enhancee encoder to rank and filter the schema items for skeleton-aware schema linking and the skeleton parsing.
20

---

Structure-Guided Large Language Models for Text-to-SQL Generation
E. Future Work
Discussing potential extensions is crucial for the research community. Following your suggestion, we have identified several promising future research directions from the following there perspectives.
E.1. Technical Extensions
E.I.I. STRUCTURE- AWARE FEW-SHOT EXAMPLE SELECTION
While our framework emphasizes the significance of the decomposition strategy, we recognize that the performance of LLM-based text-to-SQL can be further enhanced through tailored few-shot example selection. Current approaches to few-shot example selection primarily rely on keyword matching and semantic similarity between user queries. These surface-level matching approaches often fail identify the most effective examples because they consider only query semantics while ignoring the underlying SQL structural complexity.
One promising solution is to incorporate syntax structure information into the few-shot example selection process. This structure-aware approach would consider both semantic relevance and SQL. structura patterns, enabling better matching of complex query requirements with appropriate examples.
E.2. Exploring More Challenging Scenarios
E.2.1. TEMPLATE-BANE BASSD SYNTHETIC DATA GENERATION FOR TEXT-TO-SQL TRAINING
Adapting a text-to-SLL. model to a new database, like a company's proprietary database, requires developers to manually create extensive training data. This process requires: (t) Writing natural language questions about the database. (ti) Creating the corresponding correct SQL. queries. (iii) Validating the accuracy of both questions and SQL. queries. This manual data collection process is not only time-consuming but also requires expertise in both SQL and the specific database domain, making it significant bottleneck for the practical deployment of text-to-SQL. systems.
Generating synthetic training data based on template- camplate-hased approach. This method aims to eliminate the need for manual data collection by systematically generating training examples using predefined syntax templates and database schema information. The generation process operates in three coordinated stages: template selection based on database schema, schema integration by populating templates with actual table and column names, and natural language query generation.
E.2.2. INTERACTION WITH DYNAMIC DATABASE
While current text-to-SQL. methods, including our model, primarily focus static databases, real-world databases inherently dynamic. To develop truly comprehensive database management system, it is essential to extend the Text-to-SOL. framework to support full CRUD operations, Create, Read, Update, and Delete, enabling seamless and complete interaction with databases.
E.3. Broader Applications
The structure-guided approach could be extended to other domains requiring structured output generation.
E. TEXP-TO-CYPHER
Text-to-SQL converts natural language queries into SQL queries to interact with relational databases, while cext-to-Cypher translates natural language into Cypher queries for graph database operations. Considering that data in graph databases stored nodes (entities) and edges (relationships) in the format of graphs, our SGU-SQL could be seamlessly applied on Text-to-Cypher.
E.3.2. API PLANNING
API planning aims to generate sequence of API calls to accomplish a given goal or user request. Each API is essentially a function with input parameters and return values. Each function can be treated as a table, where input parameters and return values are equivalent to columns in the table. Based on the data flow, we can build a graph to describe the dependencies between different APIs, transforming the API planning task into a problem similar to Text-to-SLL, as the dependency graph analogous the schema graph in teXt-to-SQL.
21


Reviews:
[{"review_id": "ogKlCr1DPf", "reviewer": "Reviewer_hLFK", "summary": "This paper proposes a novel methodology to enhance the schema linking and complex SQL generation of LLMs for the text-to-SQL domain. Current LLM-based text-to-SQL methods face several challenges like ambiguous user intent, sophisticated database schema which often lacks proper documentations, and complex syntax structure of the SQL queries. To address's these challenges this work suggests SGU-SQL which represents the user query and the database structure into a unified graph and use a structure-learning model to find the links between the user question and the database schema effectively improving the schema linking. Finally the linked schema is divided into sub-syntax trees that are used to generate the final SQL query incrementally, breaking the complex SQL generation into multiple steps.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hLFK,\n\nThanks a lot for your detailed feedback. We really appreciate your time and effort in pointing out the potential concerns related to our paper, and also, thanks a lot for the opportunity to clarify the technical details and contribution of our framework.\n\nTo avoid any potential confusion, we first offer the following clarification:\n\n- **Our model**, like most text-to-SQL methods, is model-agnostic, meaning that it **can be integrated with any LLM as the backbone model**.\n- In Tables 1 and 2, we report the main results using GPT-3.5 and GPT-4 for cost-effectiveness considerations.\n- We compare with the top-performing methods (CHASE-SQL, CHESS, Distillery) using **Gemini-1.5 Pro and GPT-4o** as the backbone LLM in **Table 5** of the Appendix.\n\nBelow are our responses in detail.\n\n---\n\n1. **Detailed comparison with XiYan-SQL.** Thanks for your insightful comments. XiYan-SQL is the top research-based method (3rd on the BIRD leaderboard, behind two industrial solutions), but its **source code and backbone LLM** remain **undisclosed**.\n\nHowever, the authors released the pre-train version (XiYanSQL-QwenCoder)  on Hugging Face, enabling a direct comparison by using Qwen2.5-Coder as the backbone LLM.\n\nTable 2: Comparison with XiYan-SQL using Qwen2.5-Coder as the backbone LLM.\n\n| **Model** | +Qwen2.5-Coder-7B | +Qwen2.5-Coder-14B | +Qwen2.5-Coder-32B |\n| --- | --- | --- | --- |\n| XiYan-SQL(DDL) | 56.58 | 60.37 | 63.04 |\n| XiYan-SQL(M-Schema) | 59.78 | 63.10 | 67.01 |\n| SGU-SQL | 60.24 | 63.75 | 68.12 |\n\nAs shown in Table 2,  our **SGU-SQL** outperforms this competitor across all model sizes, suggesting the effectiveness and robustness of our framework.\n\n2. **Compared with other SOTA methods on BIRD.** Thanks for your insightful suggestions. For a thorough evaluation of SGU-SQL's performance, we added top-performing models from the BIRD leaderboard as baselines. From the top 10 methods in the BIRD leaderboard, we include XiYan-SQL (3rd), CHASE-SQL (4th), OpenSearch-SQL (6th), Distillery (7th),  CHESS (8th), and PURPLE (10th) in our comparisons. We exclude the remaining 4 methods (AskData, Contextual-SQL, ExSL, Insights AI) since they are all industrial solutions **without any released instructions** (papers and technical reports) or **accessible code**.\n\nTable 1: Performance comparison on BIRD dev with different LLMs as backbones.\n\n| Execution Accuracy | MAC-SQL | PURPLE | E-SQL | CHESS | Distillery | CHASE-SQL | SGU-SQL (Ours) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| GPT-4 | 59.59 | 60.71 | 58.95 | 61.37 | - | - | 61.80 |\n| GPT-4o | 65.05 | 68.12 | 65.58 | 68.31 | 67.21 | - | 69.28 |\n| Gemini-1.5 Pro | - | - | - | - | - | 73.14 | - |\n\n*Note that PURPLE, Distillery, and CHASE-SQL are closed-source models. We will update their results on GPT-4 and GPT-4o once their implementations become publicly available.*\n\nAs shown in Table 1, we have the following observations:\n\n- **SGU-SQL+GPT-4** achieves the best performance compared to the other baselines using GPT-4 as the backbone.\n- **SGU-SQL+GPT-4o** achieves $\\textcolor{maroon}{69.28\\\\%}$, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%). (We didn’t compare our model with CHASE-SQL+GPT-4o since CHASE-SQL is still closed-source and unable to integrate other LLMs. While XiYan-SQL is also closed-source, only their Qwencoder series model has been released.)\n- When using G**emini 1.5 Pro** as the backbone, SGU-SQL achieves highly competitive results  ($\\textcolor{maroon}{72.76\\\\%}$, with gemini-1.5-pro and $\\textcolor{maroon}{72.93\\\\%}$ with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).\n- With **Claude 3.5 Sonnet** as the backbone, SGU-SQL ($\\textcolor{maroon}{70.36\\\\%}$) slightly outperforms CHASE-SQL (69.53%). This improvement suggests that our method may better leverage Claude's capabilities through its structured decomposition approach.\n\nTo summarize, our approach demonstrates robust and competitive performance across different base LLMs.\n\n---\n\n---\n3. **The effect of schema linking.** Thank you for the constructive comments. Following your suggestion, we compare our graph-based schema linking with previous models and report the results in the following table.\n\nTable 3. Schema linking on BIRD.\n\n| Metrics | CodeS | CHESS | SGU-SQL |\n| --- | --- | --- | --- |\n| Precision | 92.40 | 93.12 | 95.19 |\n| Recall | 79.69 | 81.33 | 85.60 |\n\nAs shown in Table 3, our model achieves the best performance across different linking strategies, which further verifies the effectiveness of our proposed structure-aware linking mechanism.\n\n---\n\n4. **Typos in Definition 1.** Thanks for your careful review. We will fix the typos in line62-63 by changing the statement “Given a natural language query D and a database schema Q” into “Given a natural language query Q and a database schema D.”", "claims_and_evidence": "1) The claims about surpassing SOTA performance on both Spider and BIRD is not quite accurate as methods such as XiYan-SQL and CHASE-SQL achieve much higher performances on both of these benchmarks.\n\n2) The decomposition approach proposed in this work is only compared with few-shot ICL, CoT, DIN-SQL, ACT-SQL, and MAC-SQL, which are all relatively older decomposition approaches. Methods like Divide-and-conquer prompting suggested in CHASE-SQL are more advanced that should be considered as well.", "methods_and_evaluation_criteria": "1) Benchmarks and evaluation criteria used in this paper is fair, but the problem is the use of older versions of the LLMs such as GPT-4, PaLM, and text-bison model. Some of these models are no longer used for text-to-SQL pipelines and considering more advanced models like Gemini-2.0-flash or GPT-4o is essential for fair comparison.", "theoretical_claims": "In the problem formulation section, and specifically definition 1 is wrong. In this section, it is mentioned \"given a natural language query D\", I think here instead of D authors should use Q?", "experimental_designs_or_analyses": "I checked all experiments and mentioned some of the issues in the above comments.\nAdditionally, I think it would be beneficial to compare the schema linking method proposed in this work with some of the previous works such as the ones used in CODES and CHESS paper in terms of recall and precision.", "relation_to_broader_scientific_literature": "The proposed approach for schema linking seems promising to mitigate some of the challenges for complex queries and database schemas.", "essential_references_not_discussed": "XiYian-SQL paper is not mentioned.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JyTEWBLoYa", "reviewer": "Reviewer_wUD6", "summary": "This paper addresses the challenge of generating precise SQL queries from natural language, particularly when handling ambiguous user intents, complex database schemas, and SQL’s rigid syntax. The authors propose SGU-SQL, a framework that enhances Text-to-SQL generation by modeling structural relationships between entities in user questions and database tables. Key innovations include a graph-based representation to align ambiguous natural language entities with database components and a syntax-guided decomposition strategy that breaks complex questions into sub-questions to guide LLMs in incrementally constructing target SQLs. Experiments on two benchmarks verify that SGU-SQL outperforms state-of-the-art baselines, including 11 fine-tuning models, 7 structure learning models, and 14 in-context learning models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wUD6,\n\nWe are deeply grateful for your recognition of our work and also appreciate your time and effort in providing insightful suggestions that can help further polish our paper. Below are detailed responses to your comments and suggestions:\n\n---\n\n1. **The effect of the base LLMs**. Our model, like most text-to-SQL methods, is model-agnostic, meaning that it can be integrated with any LLM as the backbone model. To verify the effect of the base LLMs, we added additional experiments using **GPT-4**, **GPT-4o**, and **Gemini-1.5 Pro** as backbones.\n\nTable 1: Performance comparison on BIRD dev with different LLMs as backbones.\n\n| Execution Accuracy | MAC-SQL | PURPLE | E-SQL | CHESS | Distillery | CHASE-SQL | SGU-SQL (Ours) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| GPT-4 | 59.59 | 60.71 | 58.95 | 61.37 | - | - | 61.80 |\n| GPT-4o | 65.05 | 68.12 | 65.58 | 68.31 | 67.21 | - | 69.28 |\n| Gemini-1.5 Pro | - | - | - | - | - | 73.14 | 72.93 |\n\n*Note that PURPLE, Distillery, and CHASE-SQL are closed-source models. We will update their results on GPT-4 and GPT-4o once their implementations become publicly available.*\n\nAs shown in Table 1, our SGU-SQL achieves competitive performance across different LLM backbones. Specifically, we have the following observations:\n\n- Using **GPT-4** as the backbone, **SGU-SQL achieves the best performance** compared to other models using the same backbone.\n- With **GPT-4o**, SGU-SQL achieves 69.28% in terms of execution accuracy, **outperforming several strong baselines**: PURPLE (68.12%), CHESS (68.31%), E-SQL (65.58%) and Distillery (67.21%).\n- The only model showing higher performance is CHASE-SQL, which uses Gemini 1.5 Pro as its backbone. Notably, **CHASE-SQL** incorporates a query fixer module that **leverages database execution feedback to guide LLMs to iteratively refine generated queries**. In contrast, **our model generates SQL queries in a single pass** without utilizing any execution feedback.\n\n---\n\n2. **Justification on the backbone GNN model**. Thanks for your insightful comments. To verify the effectiveness of the backbone model, i.e., RGAT, we replace it with other alternatives, including **RGCN** [1] and **CompGCN** [2].\n\nTable 2: Alation study on backbone GNN models.\n\n| Execution Accuracy(EX) | SPIDER | BIRD |\n| --- | --- | --- |\n| Full Model (RAGT) | 87.95 | 61.80 |\n| w/o structure-aware linking | 82.62 | 55.31 |\n| with RGCN | 86.37 | 60.92 |\n| with CompGCN | 86.09 | 60.25 |\n\nAs shown in Table 3, our RGAT-based approach outperforms alternative architectures across all evaluations. Besides that, **removing structure-aware linking causes a dramatic performance drop** - accuracy decreases by 5.33% on SPIDER-dev and 6.49% on BIRD-dev. These substantial reductions highlight the critical role of our structure-aware linking strategy.\n\n*[1] Modeling Relational Data with Graph Convolutional Networks.*\n\n*[2] Composition-based Multi-Relational Graph Convolutional Networks.*\n\n---\n\n3. **Paper structure**. Thanks a lot for your valuable suggestion. We will reorganize the paper and move the key experiments into the main content of the paper to enhance clarity.", "claims_and_evidence": "The paper makes two key claims: (1) graph-based schema linking improves SQL accuracy by resolving ambiguities, and (2) syntax-guided prompting outperforms traditional methods like Few-Shot and Chain-of-Thought through syntax-aware decomposition. These claims are supported by rigorous benchmark comparisons and ablation studies.", "methods_and_evaluation_criteria": "The proposed method is innovative and well-designed, combining graph-based schema linking and syntax-aware decomposition to handle ambiguous user queries, complex database schemas, and SQL’s rigid syntax. Evaluation is thorough, using Spider and BIRD benchmarks with metrics like Execution Accuracy (EX), Exact Match Accuracy (EM), and Valid Efficiency Score (VES). Comparisons against 32 baselines across fine-tuning, structure-aware, and in-context learning paradigms highlight the framework’s robustness.", "theoretical_claims": "The decomposition strategy and structure linking are motivated and verified empirically with few theoretical claims introduced in this paper.", "experimental_designs_or_analyses": "The experiments are comprehensive, comparing SGU-SQL with 32 state-of-the-art models and including ablation studies and error analysis. While, critical findings, such as error-type distributions, are relegated to the appendix, which slightly weakens the narrative flow. Integrating these results into the main text would enhance clarity.", "relation_to_broader_scientific_literature": "SGU-SQL builds on prior LLM-based Text-to-SQL methods (e.g., DIN-SQL) by introducing graph-based schema linking and syntax-guided decomposition. This combination addresses limitations in structural alignment and complex query handling.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper identifies the critical challenges in leveraging LLMs for SQL generation. It highlights that LLMs often face significant difficulties in comprehending complex database schemas, particularly when dealing with intricate relationships between tables, columns, and constraints. Additionally, the paper emphasizes that LLMs frequently struggle to accurately interpret user queries, especially when the queries involve nuanced semantics or require precise SQL syntax.\n2.\tThe overall idea is clear and novel. The graph-based schema linking enhances SQL accuracy by effectively resolving ambiguities, and syntax-guided prompting surpasses traditional prompting strategies by leveraging syntax-aware query decomposition. The combination of these two techniques demonstrates a novel and well-thought-out solution that ensures the generated SQL queries are not only semantically correct but also syntactically precise.\n3.\tThe methodology section is well-organized. The authors provide clear mathematical formulations for key components of their approach, such as the graph-based schema linking mechanism and the syntax-guided prompting strategy. \n4.\tExtensive empirical results on widely recognized benchmarks like Spider and BIRD verifies SGU-SQL outperforms state-of-the-art baselines, including 11 finetuning models, 7 structure learning models, and 14 in-context learning models. \n5.\tThe evaluation is comprehensive, encompassing ablation study, error analysis and experiments on both open-source and proprietary LLMs, ensuring an objective assessment of the method's effectiveness and robustness across different backbone LLMs. \n\nWeaknesses:\n\n1.\tWhile the authors evaluate their framework across multiple backbone LLMs, a more systematic and detailed comparison with baseline methods using different backbone LLMs would further clarify the framework’s generalizability.\n2.\tThe paper utilizes RGAT as the backbone model for graph-based structure linking. A more thorough discussion comparing RGAT with other graph neural network architectures would make the method more clear and easier to follow.\n3.\tKey findings, such as ablation and error analysis, are placed in the appendix. Prioritizing these results in the main body would enhance clarity and strengthen the narrative coherence.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1.\tHow does the framework perform across different backbone LLMs, and are there specific LLMs for which it is particularly well-suited?\n2.\tCan the authors provide a more detailed justification for choosing RGAT over other GNN variants?\n3.\tCould critical findings from the appendix, such as ablation study and error analysis, be integrated into the main text to enhance clarity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xmT9hLUBxz", "reviewer": "Reviewer_5QmZ", "summary": "The author has proposed Structure Guided text-to-SQL framework. At a high level, it i) represent user query as a graph, vertex is key word and edge is relationship, ii) use schema graph to represent database schema, iii) linking with dual graph encoding (with Relational Graph Attention Network), and iv) apply syntax tree based guidance to decompose the generation task.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5QmZ,\n\nThank you for your expertise and insightful comments. Below are detailed responses to your comments and suggestions:\n\n---\n\n1. **Performance on BIRD**. Thanks for your insightful comments. For a thorough evaluation of SGU-SQL's performance, we added top-performing models from the BIRD leaderboard as baselines. From the top 10 methods in the BIRD leaderboard, we include CHASE-SQL (4th), OpenSearch-SQL (6th), Distillery (7th),  CHESS (8th), and  PURPLE (10th) in our comparisons. We exclude the remaining methods (AskData, Contextual-SQL, ExSL, Insights AI) since they are all industrial solutions **without any released instructions** (papers and technical reports) or **accessible code**.\n\nTable 1: Performance comparison on BIRD dev with different LLMs as backbones.\n\n| Backbone LLM | MAC-SQL | PURPLE | E-SQL | CHESS | Distillery | CHASE-SQL | SGU-SQL (Ours) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| **+GPT-4** | 59.59 | 60.71 | 58.95 | 61.37 | - | - | **61.80** |\n| **+GPT-4o** | 65.05 | 68.12 | 65.58 | 68.31 | 67.21 | - | **69.28** |\n| **+Gemini-1.5 Pro** | - | - | - | - | - | **73.14** | 72.93 |\n| **+Claude 3.5 Sonnet** | - | - | - | - | - | 69.53 | **70.36** |\n\n*Due to time and API budget limits, we have currently only evaluated our model's performance with Gemimi 1.5 Pro and Claude 3.5. We plan to conduct more comprehensive experiments with other baselines using these advanced LLMs in future work.*\n\nAs shown in Table 1, we have the following observations:\n\n- **SGU-SQL+GPT-4** achieves the best performance compared to the other baselines using GPT-4 as the backbone.\n- **SGU-SQL+GPT-4o** achieves $\\textcolor{maroon}{69.28\\\\%}$, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%). \n- When using G**emini 1.5 Pro** as the backbone, SGU-SQL achieves highly competitive results  ($\\textcolor{maroon}{72.76\\\\%}$, with gemini-1.5-pro and $\\textcolor{maroon}{72.93\\\\%}$ with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).\n- With **Claude 3.5 Sonnet** as the backbone, SGU-SQL ($\\textcolor{maroon}{70.36\\\\%}$) slightly outperforms CHASE-SQL (69.53%).\n\nTo summarize, our approach demonstrates robust and competitive performance across different base LLMs.\n\n---\n2. **The importance of schema linking**.  We thank the reviewer for raising this important point. While we commend Distillery-SQL’s novel schema-free paradigm leveraging iterative refinement and execution feedback, we respectfully contend that explicit schema linking remains indispensable for real-world text-to-SQL systems, particularly for three reasons:\n\n- While Distillery-SQL achieves strong results without dedicated schema linking, **its reliance on iterative query refinement** (via augmentation/selection/correction) introduces substantial computational overhead. For instance, their pipeline requires multiple LLM calls with database execution feedback, which incurs significant latency and infrastructure costs. In contrast, schema linking modules enable single-pass query generation while maintaining desirable performance.\n- Inaccurate schema linking degrades LLM-based SQL generation while **accurate schema linking still improves the model performance**. Current top-performing models (XiYAN-SQL, CAHSE-SQL, etc.) universally incorporate schema linking, achieving superior performance on benchmarks like BIRD and Spider (+5-8% over schema-free baselines). This aligns with our findings: explicit schema linking improves robustness, particularly for long-tail schemas and compositional queries.\n\n---\n\n3. **Compared to CHASE-SQL**. Thanks for your insightful comments. Following your suggestion, we compare our model with CHASE-SQL by integrating different backbone LLMs.\n\n| **Model** | +GPT-4o | +Gemini-1.5 Pro | +Claude 3.5 Sonnet |\n| --- | --- | --- | --- |\n| CHASE-SQL | - | 73.14 | 69.53 |\n| SGU-SQL | 69.28 | 72.93 | 70.36 |\n\nNotably, **CHASE-SQL** incorporates a query fixer module that **leverages database execution feedback to guide LLMs to refine generated queries iteratively**. In contrast, **our model generates SQL queries in a single pass** without utilizing any execution feedback. As shown in the table, our model shows more desirable performance than CHASE-SQL.  It is because that traditional methods attempt to generate entire SQL queries in one step or rely on simple decomposition strategies, SGU-SQL breaks down the complex generation task in a syntax-aware manner. This ensures that the generated queries maintain both semantic accuracy (correctly capturing user intentions) and syntactic correctness (following proper SQL structure).\n\n---\n\n4. **The structure of Table 1**. Thanks a lot for your insightful comments. Following your suggestion, we will make Table 1 more concise by removing some less important baselines.", "claims_and_evidence": "Some of the claims are not well supported.\n\nFor example, please add more discussion about [1], which achieve high execution accuracy for BIRD and argues that schema linking is not important \" if the schema fits within the context length\".\n\nIs the proposed decomposition strategy better than CHASE-SQL [2]? I didn't find results of CHASE-SQL in table 1.\n\n\n[1] Maamari, Karime, et al. \"The death of schema linking? text-to-sql in the age of well-reasoned language models.\" arXiv preprint arXiv:2408.07702 (2024).\n\n[2] Pourreza, Mohammadreza, et al. \"Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql.\" arXiv preprint arXiv:2410.01943 (2024).", "methods_and_evaluation_criteria": "Spider and Bird are important benchmarks for text-to-SQL solutions.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "Text-to-SQL is an important problem with significant practical importance.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "S1. The proposed method for schema linking is sound.\n\nS2. The evaluation section compared with many different models and techniques.\n\n\nW1. The performance of the model on Bird benchmark is 61.8% (Table 2), which falls in the range of 25-32 on Bird leaderbaord. \n\n\nW2. Need more discussion and support on why is schema linking is important, given Distillery-SQL argues otherwise. Distillery-SQL ranks at 7th place on Bird benchmark. \n\nW3. Need better demonstration on why is the proposed task decomposition is better than that of Chase-SQL.", "other_comments_or_suggestions": "Replace the number of models in Table 1. Maybe only include the best model in fine-tuned and structure learning category.", "questions_for_authors": "Please see W1-W3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D1JcL9nesX", "reviewer": "Reviewer_jn4B", "summary": "This paper introduces SGU-SQL, a structure-guided framework for text-to-SQL generation using large language models (LLMs). By leveraging syntax trees and database schema graphs, SGU-SQL recursively decomposes queries into subtasks guided by SQL syntax, enabling incremental and accurate SQL generation. Experiments on Spider and BIRD benchmarks demonstrate its superiority over state-of-the-art baselines, particularly in handling complex queries. The framework addresses key challenges such as schema linking, syntax errors, and structural ambiguity through graph-based representations and syntax-aware decomposition.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jn4B,\nThank you for your recognition of our work and for providing such thorough and insightful feedback. Your comments and suggestions are invaluable in helping us improve the quality and clarity of our work.\n\n---\n\n1. **Insufficient details for error analysis.** Thank you for your thorough review. In this paper, we perform an error analysis to evaluate our model’s performance. Specifically, we classify errors into two primary categories:\n- **Schema-linking errors**: Incorrect matching of tables or columns in the database schema.\n- **Syntactic errors**: Invalid SQL syntax, including misuse or omission of key SQL clauses (e.g., `JOIN`, `GROUP BY`, nested queries and others).\n\nAs illustrated in **Figure 2** of our submission, we provided a detailed analysis of the error distribution. Compared to the baseline model, our approach significantly improves **schema-linking accuracy** and **syntactic correctness**.\n\n---\n2. **The definition of VES metric is not clear**. The definition of VES metric is not clear. Sorry for the confusion caused. Valid Efficiency Score (VES) is defined to measure the efficiency of valid SQL queries, which was first defined in BIRD bechmark. \n\nA valid SQL query is a predicted SQL whose executed results exactly match the ground truth results. Specifically, VES evaluates both the efficiency and accuracy of predicted SQL queries.\nFor a text dataset with $N$ examples, VES is computed by:  $\\text{VES} = \\frac{1}{N}\\sum_{n=1}^{N}\\textbf{I}(V_n, \\hat{V}_n) \\cdot \\textbf{R}(Y_n, \\hat{Y}_n),$ where $\\hat{Y}_n$ and $\\hat{V}_n$ are the predicted SQL query and its executed results and $Y_n$ and $V_n$ are the ground truth SQL query and its corresponding executed results, respectively. $\\text{I}(V_n, \\hat{V}_n)$ is an indicator function, where: $\n\\textbf{I}(V_n, \\hat{V}_n) = 1 if V_n = \\hat{V}_n.\n$ Then, $\\textbf{R}(Y_n, \\hat{Y}_n) = \\sqrt{E(Y_n)/E(\\hat{Y}_n)}$ denotes the relative execution efficiency of the predicted SQL query in comparison to ground-truth query, where $E(\\cdot)$ is the execution time of each SQL in the database.\n\n---\n3. **The examples of the syntax tree are not clear.** Thanks a lot for your careful review. Following your suggestion, we have updated **Figure 5** to include more complex examples, particularly those involving nested queries. We will add the revised figure in our new manuscript since we are not allowed to include external links in this version.\n\n---\n4. **Personalization gaps.** We sincerely appreciate your insightful suggestion regarding personalized decomposition strategies tailored to different user intents and database structures. This is indeed a promising research direction that could significantly enhance real-world applicability. Moving forward, we plan to extend our framework by incorporating adaptive decomposition mechanisms to further improve system performance.\n\n---\n5. **Efficiency trade-offs**. Thank you for making this valuable suggestion. To assess our approach thoroughly, we conducted the efficiency analysis on the BIRD dataset (33.4 GB total). Given that the queries in this dataset are categorized into 3 difficulty levels: simple, moderate, and challenging, we specifically tested our model on **the challenging set of the BIRD** dataset and compared its performance with DIN-SQL and MAC-SQL. \n\nTable 1: Efficiency analysis on the ''Challenging'' set of BIRD.\n\n| **Model** | **Training Time** | **Inference Time** | **Performance** |\n| --- | --- | --- | --- |\n| DIN-SQL | 4.69 h | 0.39 h | 36.7% |\n| MAC-SQL | 4.98 h | 0.36 h | 39.3% |\n| SGU-SQL | 3.47 h | 0.22 h | 42.1% |\n\nAs shown in Table 4, **our model demonstrates superior performance while maintaining competitive computational efficiency.**  This superior efficiency can be attributed to our graph-based architecture. While baseline methods avoid the overhead of graph construction, **they heavily rely on prompt-based modules** that require multiple calls to LLMs like GPT-4. These **API calls introduce substantial latency** that accumulates during both the training and inference phases. In contrast, our graph-based approach, despite its initial graph construction overhead, achieves faster end-to-end processing by minimizing dependence on time-consuming API calls.\n\n---\n6.  **The performance of lightweight LLMs.** Thanks for the valuable comments. Following your suggestion, we add the QwenCoder series model as the backbone LLM.\n\nTable 2: Performance on BIRD with Qwen2.5-Coder as the backbone LLM.\n\n| **Model** | +Qwen2.5-Coder-7B | +Qwen2.5-Coder-14B | +Qwen2.5-Coder-32B |\n| --- | --- | --- | --- |\n| XiYan-SQL(DDL) | 56.58 | 60.37 | 63.04 |\n| XiYan-SQL(M-Schema) | 59.78 | 63.10 | 67.01 |\n| SGU-SQL | 60.24 | 64.75 | 68.12 |\n\nXiYanSQL-QwenCoder series model is the SOTA method that uses lightweight Qwen2.5-Coder as the backbone. As shown in Table 2,  our **SGU-SQL** outperforms this competitor across all model sizes, suggesting the effectiveness and robustness of our framework.", "claims_and_evidence": "The paper claims that SGU-SQL significantly reduces errors in complex queries (such as Schema links and Join statements), and demonstrates performance improvements on Spider and BIRD datasets through experimental data (such as Table 1). These data support the main argument, but there are some problems: Insufficient error classification details: Although it is mentioned that errors are reduced by 33.5% (Appendix case analysis), the distribution and quantitative standards of specific error types (such as syntax errors, logical errors) are not clearly stated, which may lead to doubts about the credibility of the conclusion.", "methods_and_evaluation_criteria": "Graph structure construction and dual graph encoding: It is reasonable to use RGAT to handle the graph structure of queries and databases, but it does not elaborate on how to solve the ambiguity problem in graph alignment (for example, how to select the optimal solution when multiple candidate nodes match).\nSyntax tree decomposition strategy: It is innovative to decompose SQL generation into subtasks based on syntax trees, but the specific processing mechanism for nested subqueries or complex aggregate functions is not discussed.\nEvaluation Metrics: The selection of EM Acc, Exec Acc, and VES is comprehensive, but the calculation method of VES is not clearly defined (for example, how to balance efficiency and accuracy)", "theoretical_claims": "The paper does not provide a rigorous theoretical proof and mainly relies on experimental verification. For example, does structural decomposition necessarily improve the generation effect? Are there any theoretical boundaries (e.g., too fine a decomposition granularity may lead to context loss)? These questions are not explored at the theoretical level.", "experimental_designs_or_analyses": "Limited Baseline Comparisons: While SGU-SQL outperforms listed baselines. If possible, please compare with newer LLM-based text-to-SQL methods (e.g., GPT-4 variant 4o, DeepSeek-R1).", "relation_to_broader_scientific_literature": "The paper makes a good connection between traditional methods (such as RAT-SQL), PLM-based methods (such as T5), and LLM paradigms (such as GPT-4)", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1.Innovative Methodology: The integration of syntax trees and schema graphs to guide LLM-based SQL generation is novel and addresses critical limitations of existing methods, such as schema linking errors and structural ambiguity.\n2.Comprehensive Evaluation: Extensive experiments on two benchmark datasets (Spider and BIRD) validate SGU-SQL’s effectiveness, with significant improvements in execution accuracy, especially for complex queries.\n3.Practical Insights: The ablation studies and error analysis provide valuable insights into the contributions of each component and highlight the framework’s robustness across query difficulty levels.\nWeakness:\n1.Personalization Gaps: The framework does not explore personalized decomposition strategies for different user intents or database structures, which could further enhance performance in real-world scenarios.\n2.Efficiency Trade-offs: The graph construction and syntax decomposition steps introduce computational overhead. While efficiency analysis is included, the trade-off between accuracy and latency in large-scale applications is underexplored.", "other_comments_or_suggestions": "None.", "questions_for_authors": "The framework uses GPT-4 as the backbone LLM. Would performance degrade significantly with smaller, open-source models (e.g., CodeLlama-7B)? Are there strategies to mitigate this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes SGU-SQL, a structure-guided framework designed to improve the performance of Large Language Models (LLMs) on complex text-to-SQL tasks. The core idea is to address challenges like ambiguous user intent and complex database schemas through two main components: 1) a structure-aware schema linking module that represents the user query and database schema as graphs and uses a Relational Graph Attention Network (RGAT) to identify relevant schema elements, and 2) a syntax-guided decomposition strategy that breaks down the SQL generation process into smaller sub-tasks based on SQL syntax trees, guiding the LLM to construct the query incrementally. Initial reviews were mixed, acknowledging the novelty of combining structure-aware linking and syntax-guided decomposition, but raising significant concerns. Key issues included the accuracy of claims regarding state-of-the-art (SOTA) performance (missing comparisons to top leaderboard methods like XiYan-SQL and CHASE-SQL), the justification for the importance of schema linking (given schema-free methods like Distillery-SQL), the limited comparison of the decomposition strategy to recent methods, and the use of older LLM backbones in primary experiments.

In response, the authors provided a substantial rebuttal with extensive new experimental results. They added direct comparisons against top BIRD leaderboard competitors (including XiYan-SQL on Qwen models, CHASE-SQL, Distillery, CHESS, etc.) using various modern LLM backbones (GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet), demonstrating that SGU-SQL achieves competitive or state-of-the-art performance across different models, notably outperforming CHASE-SQL on Claude 3.5 Sonnet and XiYan-SQL on Qwen models without relying on execution feedback like CHASE-SQL. They defended the importance of schema linking by arguing for its efficiency benefits over iterative refinement methods like Distillery and provided new experimental results showing their schema linking outperforms prior methods (CODES, CHESS). They also added ablations on the GNN component (justifying RGAT) and addressed clarity issues regarding evaluation metrics and error analysis. This rebuttal effort successfully convinced one reviewer (wUD6) to raise their score from Weak Accept to Accept. However, the two most critical reviewers (hLFK and 5QmZ), who had raised the concerns about SOTA comparisons and schema linking, did not acknowledge or respond to the rebuttal, despite the authors providing direct evidence addressing their points.
Decision: Accept (poster)