Structure-Guided Large Language Models for Text-to-SQL Generation

Authors: Qinggang Zhang, Hao Chen, Junnan Dong, Wentao Li, Feiran Huang, Xiao Huang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-to-sql, large language model, structure learning

Abstract:
Recent advancements in large language models (LLMs) have shown promise in bridging the gap between natural language queries and database management systems, enabling users to interact with databases without the background of SQL. However, LLMs often struggle to fully exploit and comprehend the user intention and complex structures of databases. Decomposition-based methods have been proposed to enhance the performance of LLMs on complex tasks, but decomposing SQL generation into subtasks is non-trivial due to the declarative structure of SQL syntax and the intricate connections between query concepts and database elements. In this paper, we propose a novel $\textbf{S}$tructure $\textbf{GU}$ided text-to-$\textbf{SQL}$ framework ($\textbf{SGU-SQL}$) that incorporates syntax-based prompting to enhance the SQL generation capabilities of LLMs. Specifically, SGU-SQL establishes structure-aware links between user queries and database schema and recursively decomposes the complex generation task using syntax-based prompting to guide LLMs in incrementally constructing target SQLs. Extensive experiments on two benchmark datasets demonstrate that SGU-SQL consistently outperforms state-of-the-art text-to-SQL baselines. These results highlight the importance of incorporating structural syntax information for effective text-to-SQL generation and pave the way for more robust and reliable interfaces to databases in the era of artificial intelligence.

Decision: reject
Reviewer scores: 5, 10, 8, 3

Reviews and discussion:
<thread>
creation date:
1730564610261

thread_type:
official_review

<thread_object>
creator: reviewer_esja

<content>
summary:
Recent text-to-SQL methods using LLMs encounter challenges in accurately interpreting user intent and navigating the complex structures of databases. Existing decomposition-based approaches to improve LLM performance on text-to-SQL tasks often struggle with SQL's declarative syntax and the intricate links between queries and database elements. This paper introduces Structure Guided text-to-SQL (SGU-SQL), a novel framework that integrates syntax-based prompting to enhance LLM-driven SQL generation. SGU-SQL incorporates structure-aware linking between user queries and database schemas and decomposes complex SQL generation tasks into manageable subtasks through syntax-based guidance.

strengths:
1) The experiments provide a comprehensive comparison with most of the previous works, showcasing that their approach outperforms all included baselines, although some recent approaches were not covered.

2) The novel prompting strategy significantly enhanced model performance across different LLM families, achieving substantial improvements over naive few-shot prompting and chain-of-thought (CoT) methods.

3) The paper presents an effective strategy for linking the structural elements of user queries to the database schema, facilitating more accurate SQL generation.

weaknesses:
The primary limitation of this approach lies in the experimental section, where some of the most recent state-of-the-art methods, such as CHESS, Distillery, and CHASE-SQL, are not included. This omission is significant, as approaches like Distillery suggest that LLMs already demonstrate strong performance in handling complex schemas without the need for schema linking or query decomposition, challenging the findings of this paper. Additionally, as shown by CHASE-SQL, LLMs can achieve approximately 80% performance (compared to the 62% reported in this paper on BIRD) through enhanced sampling strategies, indicating that the main challenge in the text-to-SQL domain now lies in verifying the outputs of LLMs.

Furthermore, crucial tables, such as Table 2 and Table 3, have been relegated to the appendix. These tables contain valuable data and should be included in the main body of the paper for better accessibility and understanding.

questions:
NA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704676292

thread_type:
official_review

<thread_object>
creator: reviewer_vrvu

<content>
summary:
The paper focuses on tackling the challenge of generating robust and reliable SQL for a given natural language question. The authors propose SGU-SQL which is a novel framework leveraging the structural relationship between entities in user questions and the database tables for solving Text to SQL tasks using LLMs. There are three main challenges that the authors focus on:
1. Ambiguous user intents which lead to LLM's making uninformed decisions about SQL generation
2. Complex database schemas which may lead the LLM to confuse between different columns or tables
3. Complexity of SQL - LLMs are trained mainly for text generation whereas SQL is complex like programming language where only well-defined syntax must be used.

The authors tackle these issues by creating this framework where entities in natural language questions and the database are represented using graphs. These entities from questions to database are linked using a structure-linking model forming connections. Then, finally a prompt based method decomposes the complex question into subquestions and tackle these subquestions separately while aggregating them later. This method has been experimented with GPT-4 and a comparison has been reported using several models including SOTA from the leaderboard. Overall, the framework is proved to be successful while also achieving SOTA results on

strengths:
1. The paper focuses on one of the critical issues with LLMs for SQL generation which requires the LLMs to tackle them in a unique way where reliability and robustness are a strong focus.
2. The framework is novel, intuitive and well designed to bridge the gap between an LLM which is tuned effectively for text generation and the SQL generation task which requires a structured understanding where the user queries can be ambiguous and answers can be complex.
3. The paper demonstrates strong results on the most popular SQL datasets SPIDER and BIRD which certifies the potential of the approach.
4. Well articulated work and not complex to follow from beginning to end. Conducted experiments including ablation studies are designed with thorough understanding.
5. The real world applicability and potential of this framework can be very high and effective when tackling large production grade databases in industrial applications.

weaknesses:
1. The framework is novel and intuitive but because of SPIDER not being the best representation of production grade SQL, it would also beneficial to probably test them using SPIDER-2. BIRD is somewhat closer to a real-world SQL but SPIDER-2 has large-scale and diverse databases which evaluates the proposal on a much better dataset revealing how the proposal works on large scale datasets.
2. There's not much information provided on the future direction of this work which might hurt the significance/potential of the work a little. I'm mainly looking for how can this work be extended as it can provides opportunities for other researchers trying to explore in the same direction.
3. Specific discussion on the model used for structure linking seem insufficient. Like what training methodology and architecture choices.
4. When we discuss about constructing graphs and linking nodes, it may be important to learn about the complexities at a high level.

questions:
1. In a real world, the databases are changed very often and there may be challenges where the proposed framework might not work same as proposed. In such cases, what components of the framework can still be relevant and what components must be changed is particularly industrial practitioners care about. Atleast including it in future directions might suffice.
2. How does this framework vary for dialects other than SQL? Not a super important question to clarify for the purpose of this conference, but having information about this can be helpful for practitioners.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730738099336

thread_type:
official_review

<thread_object>
creator: reviewer_b8gr

<content>
summary:
+ Identify the limitations of LLM-based Text-to-SQL models and introduce SGU-SQL, which
leverages structural syntax information to improve SQL generation capabilities of LLMs.
+ SGU-SQL proposes graph-based structure construction to comprehend user query and database
structure and then link query and database structure with dual-graph encoding.
+ SGU-SQL introduces tailored structure-decomposed generation strategies to decompose queries with
syntax trees and then incrementally generate accurate SQL with LLM.
+ Experiments on two benchmarks verify that SGU-SQL outperforms state-of-the-art baselines, including 11 fine-tuning models, 7 structure learning models, and 11 in-context learning models

strengths:
+ The article is quite neat and meticulous, from giving mathematical formulas and reasoning why to do so and also the evaluation steps, ablation study, the article has introduced a completely new method in text2sql which is syntax-based prompting, avoiding traditional classical methods such as FewShot and CoT.
+ The method of the author group is quite new and very well implemented, the evaluation of the method is also on both open-source and closed source to provide an objective view.
+ The novelty of the method lies in the ability to build a good enough graph structure to link from user queries and information in the database, thereby helping to reduce the input of LLM to some extent, the selected information is carefully filtered to help the generated query be of better quality.

weaknesses:
Overall I found the article quite good and have no comments on weaknesses.

questions:
+ Have the authors compared the current method with the traditional CoT or FewShot methods?
+ What Lora rank does the author team use for fine-tuning? Have they tested it on multiple ranks?
+ Nhóm tác giả có thể giải thích thêm về cách training GNN, các thử nghiệm và kết quả đánh giá với GNN của bạn?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731138316643

thread_type:
official_review

<thread_object>
creator: reviewer_53mq

<content>
summary:
The paper proposes a method SGU-SQL, which represents user queries and databases into unified and structured graphs and employs a tailored structure-learning model to establish a connection between the user queries and the databases. The linked structure is then decomposed into sub-syntax trees, guiding the LLMs to generate the SQL query incrementally

strengths:
This paper provides interesting ideas for text-to-sql, such as graph representation of query & data schema, query schema linkage, syntax tree based decomposition with LLMs, etc.  Although these concepts are not brand new, it is interesting to see how authors leverage on them with graph + LLMs

weaknesses:
**1 Method writing should be clearer**

It is not very clear how does schema-linkage is used in SQL generation by decomposition (sec 3.1)

How is score eq21 used in your algorithm? 

minor: 

PRELIMINARIES has detailed equations, etc, however this section seems not help introduce the method, may confuse readers (training GNN? prompting?)


**2. Performance is not competitive on more complex dataset like BIRD** 
To measure the effectiveness of the method, we need to show competitive results on complex realistic dataset. 

Table 2 is BIRD performance (exe acc 61), which is lower than SOTA 74 https://bird-bench.github.io/
SGU-SQL focus on leveraging structure-aware links between queries and database schema, so to prove effectiveness of the method, the method need to be tested on more challenging database, like BIRD. 

Table 1 Spider.  Many of the baseline methods are not Text-to-SQL specific method, therefore performance is not competitive. 
additionally only show method is good on Spider is not enough.  

ideally test performance on test split by submitting to leaderboard for generation ability



**3. need convincing ablation study to show each component designs is helpful**

Need ablation study on effectiveness of graph representation of query (sec 3.1) i.e.say compared with standard natural language representation without graph representation

Need ablation study on effectiveness of graph representation of database, compared with code representation “CREATE TABLE (...)” or serialized representation: table1: c1,c2 .. | table 2: c1, c2 … 

Need convincing ablation qualitative and qualitative results on the effectiveness of  “structure-aware linking”: effectiveness of DUAL GRAPH ENCODING(i.e. linking the query to the appropriate tables and columns in the database ). Since link query and database structure is one of key contribution, there should be clear ablation study

Need convincing ablation study to show “syntax-based decomposition” is effective. what if we do not do the “DECOMPOSING QUERY WITH SYNTAX TREE” (3.2.1), only directly generation SQL out. 
Table 3 compares with other basic decomposition approach. relatively simple prompt decomposition method can get on par text-to-sql performance ​​https://arxiv.org/pdf/2312.11242 (86.75 vs 87.5)

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1733033543908

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Area Chairs and All Reviewers (1/2)

comment:
Dear Area Chairs and Reviewers,

We sincerely thank you for your time, effort and invaluable feedback during the author-reviewer discussion phase. Your insights have been crucial in helping us refine and improve our work. To facilitate the next stage of review and discussion, we would like to summarize the key contributions of our paper, the main points raised during the discussion and our response. 

### **Motivation**

LLM-based text-to-SQL models often struggle to comprehend complex database structures and accurately interpret user intentions. Decomposition-based methods have been proposed to enhance the performance of LLMs on complex tasks, but **decomposing SQL generation into subtasks is non-trivial** due to the declarative structure of SQL syntax and the intricate connections between query concepts and database elements. In this paper, we propose a novel **structure-guided text-to-SQL framework** (SGU-SQL) that incorporates **syntax-based prompting** to enhance the SQL generation capabilities of LLMs. 


### **Contribution**

SGU-SQL offers significant improvements over existing approaches in two key aspects.

- **Syntax-based Prompting.** While traditional methods attempt to generate entire SQL queries in one step or rely on simple decomposition strategies, SGU-SQL breaks down the complex generation task in a syntax-aware manner. This ensures that the generated queries maintain both semantic accuracy (correctly capturing user intentions) and syntactic correctness (following proper SQL structure).
- **Combine with Structure-aware Linking.** Our framework's unique combination of structure-aware linking and syntax-guided decomposition creates a robust bridge between user intentions and database structures. The structure-aware linking ensures that each natural language concept is correctly mapped to the corresponding database elements, while the syntax-guided decomposition ensures these mappings are properly utilized in constructing the SQL query.

### **Main discussion points and our responses.**

- **Impact of Structure Linking**. The structure-aware linking creates a foundation for accurate decomposition by establishing precise mappings between natural language elements and database schema components. Following the suggestions from Reviewer 53mq, b8Gr and vRvu, we added more details about the **design principles**, **training process** and **efficiency analysis** of the structure linking in Section 3.1.3, and conducted a comprehensive ablation study on the backbone GNN models for structure learning (Appendix C.1 and D.2).
- **Performance on BIRD and SPIDER**. Following the suggestions from Reviewer 53mq and EsJa,  we included the **latest baselines** and conducted additional experiments with **different LLMs as backbones** to achieve a thorough evaluation of SGU-SQL in (Appendix C.3).
- **Comparison with CoT and Few-shot-based Methods**. Following the suggestions from Reviewer 53mq and b8Gr, we conducted additional experiments to compare our SGU-SQL with other advanced prompting strategies to verify the effectiveness of our syntax-based decomposition strategy (Appendix C.2).
- **Additional Experiment on SPIDER-2.** SPIDER-2 is currently the most challenging text-to-SQL dataset available. Following the suggestion from Reviewer vRvu, we extended our evaluation on **Spider 2.0-Snow** and **Spider 2.0-Lite** (Appendix D.1).
- **Future Work.** Discussing potential extensions is crucial for the research community. Following the suggestion from Reviewer vRvu, we have identified several promising future research directions from **three perspectives** (Appendix E).

### **Detailed analysis of concerns mentioned by Reviewer EsJa**

After the first round of responses, Reviewers 53mq and EsJa raised insightful questions regarding our model's performance on the BIRD and Spider benchmarks. In our second-round response, we provided thorough clarifications and additional analyses to address their concerns. **Reviewer 53mq** has acknowledged our explanations and positively updated their evaluation, while we have not yet received feedback from **Reviewer EsJa**. To prevent any potential confusion in subsequent reviews, we now present a detailed examination of these concerns.

**Performance on BIRD**

For a thorough evaluation of SGU-SQL's performance, we included top-performing models from the BIRD leaderboard as baselines.

**a. Baseline selection**

As shown in Table 1, the current BIRD leaderboard features a mix of academic research and industrial solutions. For a comprehensive evaluation, we selected baselines from two categories:

- **Open-source models** with published papers and available code (MAC-SQL, Super-SQL, E-SQL, **CHESS**).
- **High-performing methods** with only available papers (**PURPLE**, **Distillery**, **CHASE-SQL**).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents SGU-SQL, a model that leverages syntactic and structural information of user queries and database schemas to generate more accurate SQL queries. There are three components in the system. Given a user query and a database schema, a graph encoder matches phrases in the user query to corresponding database elements (entity/schema linking), such linking information is then used to decompose the original task into sub-tasks. Those sub-tasks are subsequently solved by LLMs by predicting their corresponding SQL queries. The authors provided extensive experiment results, claiming that incorporating structural information can effectively improve text-to-SQL accuracy. However, due to issues with writing it is unclear how the method actually works (more details below).

**Strengths:**

* While the idea of leveraging syntactic and structural information for text-to-SQL is not new (53mq), the idea of composing a graph-encoding approach together with LLMs is interesting and effective (53mq, vRvu, EsJa), and could help “reduce the input of LLM(s) to some extent” (b8Gr)

* Evaluation is fairly comprehensive (EsJa), demonstrating “strong results on the most popular SQL datasets SPIDER and BIRD” (vRvu), “achieving substantial improvements over naive few-shot prompting and chain-of-thought (CoT) methods” (EsJa).

**Weaknesses:**

The following issues are addressed during the rebuttal phase:

* Performance is not competitive on more complex datasets like BIRD (53mq) — authors provided more through evaluation results that shows strong results on BIRD with more recent LLMs.

* Ablation needed to show impact of different components (53mq) — addressed with additional ablation results

* Evaluation on more complex text-to-sql tasks like SPIDER-2 (vRvu) — partially addressed with preliminary results on SPIDER-2.
Comparison with more recent SoTA methods like  CHESS, Distillery, and CHASE-SQL — addressed by including results from these models, where the proposed SGU-SQL still outperforms these models.

The major issue yet to be addressed is clarity and technical writing:

* 53mq: “Method writing should be clearer”

* vRvu: “Specific discussion on the model used for structure linking seem insufficient”

* EsJa: “crucial tables … have been relegated to the appendix. These tables contain valuable data and should be included in the main body of the paper”

After carefully reading the submission, I agree with the reviewers comments on writing issues. For example, it is not clear how the model was trained, and how it works end-to-end at inference time. Note that the proposed approach has two components: a trainable GNN that learns structural information of natural language queries and DB schemas, and a prompting method using LLMs to generate SQL queries. It is not clear how the trainable component works with the LLM. While the authors provided further clarifications in the updated version of the draft, without a running example to illustrate the entire approach, it is still difficult for readers to understand how the approach work end-to-end. 

Other missing technical information includes the models / methods used to generate the syntactic and semantic parse of a natural language query. Indeed, to my knowledge, the relation types in Table 9 do not map to any existing well-known syntactic / semantic taxonomy.

Therefore, given the remaining issues with technical presentation. I believe that the paper could indeed benefit from another round of major revision. Therefore, the recommendation is Rejection.