READ-SQL: Reasoning Path Decomposer for Text-to-SQL

Authors: Yaxun dai, Haiqin Yang, Mou Hao, Pingfu Chao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Text-to-SQL, Tabular Reasoning, SQL Decomposition, Abstract Syntax Trees, Self-Correction

Abstract:
Text-to-SQL is a longstanding task aimed at automatically converting natural language questions into SQL queries for database retrieval. Despite impressive advancements, particularly with Large Language Models (LLMs), existing methods still struggle with issues such as misinterpreted, omitted, or unwanted constraints.  To address these challenges, we propose READ-SQL, a novel framework employing a \underline{re}asoning p\underline{a}th \underline{d}compos\underline{er}, \textbf{READ}ER, for text-to-SQL tasks.  READER decomposes SQLs into clauses, sub-SQLs, and reasoning paths, supporting data preparation and confidence level determination in post-processing.  READ-SQL comprises two main models: a Generator and a Corrector, both trained via LoRA for parameter efficiency.  Based on READER's decomposition, READ-SQL generates two types of augmented data using an LLM: question/SQL pairs and question/reason pairs.  The Generator is trained on both original and augmented data to identify constraint changes and enhance reasoning.  The Corrector is trained on data from READER’s post-processing, improving self-correction by refining high-confidence SQLs and addressing low-confidence elements.  Extensive experiments show that READ-SQL significantly outperforms leading baselines, with READ-SQL-3B achieving 57.37\% execution accuracy on BIRD’s dev set, surpassing several 7B-parameter models and setting a new state-of-the-art with fewer parameters.  Additionally, READER and the Corrector show broad applicability when integrated with LLMs or other base models.

Decision: reject
Reviewer scores: 5, 6, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730079702085

thread_type:
official_review

<thread_object>
creator: reviewer_gsf2

<content>
summary:
This paper studied the Text-to-SQL task, and proposed a READ-SQL framework to improve the model's understanding of questions and SQL clauses. READ-SQL parsed the SQL into AST and decomposed into clauses to form the sub-SQLs and reasoning paths. Based on the sub-SQL and reasoning, READ-SQL generated augmented data with the LLM to train a Generator sensitive to constraint changes and reasoning. READ-SQL also trained a Corrector to recognize low-confident clauses for self-correction. Finally, the authors conducted extensive experiments to demonstrate the effectiveness of the proposed method.

strengths:
1. The idea of aligning the natural language and structured syntax to correctly interpret the constraints is interesting.
2. The paper proposed to generate the augmented training data via clause decomposition of SQL, which is novel.
3. The proposed method outperformed leading baselines, and surpassed larger models on BIRD dataset.

weaknesses:
1. The proposed seemed to be an incremental improvement of the existing Text-to-SQL methods, and the technical contributions seemed to be limited.
2. The inference pipeline of the proposed framework is unclear. The paper detailed introduced the data generation process, but it did not show how the trained generator and corrector were used to generate the SQL, i.e., the paper lacked an overall description from the text input to the SQL output.
3. The motivation behind the proposed method should be clarified more clearly. For example, what is the purpose of constraint identification and decomposition, and why can it solve the problems mentioned in the introduction.
4. Some technical introductions are unclear, which makes it difficult to understand the paper.
- What does the constraint or sub-SQL mean, and what can be considered as a constraint?
- What are the goals of the Generator and Corrector, how are they used in SQL generation, and how is the Corrector trained and realized?
- The proposed method regenerated the question based on the sub-SQL, so what is the relation between the generated question and the original ones, and have the authors considered their consistency? Besides, how did the authors ensure that the sub-SQL could solve the generated question?

questions:
See the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730279881821

thread_type:
official_review

<thread_object>
creator: reviewer_hcl3

<content>
summary:
READ-SQL is a framework designed to improve the conversion of natural language queries into SQL commands, an area known as text-to-SQL. This framework incorporates two main models, Generator and Corrector, which work with a core module, READER. READER decomposes SQL queries into components, helping to build reasoning paths. The Generator initially formulates SQL based on input questions and generated data, while the Corrector refines these SQLs by focusing on high- and low-confidence components. Together, they achieve significant accuracy and efficiency improvements over comparable models, even those with larger parameter sizes, as shown in the extensive experiments on benchmarks like BIRD and Spider​.

strengths:
1. READ-SQL outperforms existing models on the BIRD benchmark, achieving high execution accuracy and efficiency.
2. Extensive testing on multiple datasets and with various experimental setups (e.g., ablation studies, different model sizes) demonstrates the framework's consistency and robustness.

weaknesses:
1. The paper highlights three key issues in text-to-SQL: misinterpreted constraints, omitted constraints, and unwanted constraints. However, these issues are neither addressed nor mentioned in the experimental section, nor are they resolved in the proposed approach, which I find confusing.
2. The authors state, “However, existing methods primarily focus on optimizing SQL generation without improving the model’s understanding of the question or establishing strong relationships between questions and SQL clauses.” From my perspective, schema linking already serves to establish relationships between queries and SQL structure, which weakens this point as a unique limitation. Additionally, READ-SQL does not appear to be significantly better than other methods in terms of understanding the question. Furthermore, they mention, “Additionally, since these methods do not employ an end-to-end architecture, there is potential for information loss during the process.” Based on my understanding, T5-based and GPT-4-based approaches share a similar end-to-end structure with READ-SQL, as they transform inputs directly into SQL queries. Therefore, this limitation also seems unconvincing. Overall, the motivation for READ-SQL remains unclear to me.
3. Although augmented data enhances performance, generating question/reason pairs involves complex processing that may pose implementation challenges. More importantly, even with additional training data, the performance gains on the Spider dataset are less significant than on the BIRD dataset. This raises the question of whether the observed improvements might be due to overfitting, as the paper indicates that most of the augmented data originates from the BIRD dataset.

questions:
1. What specific advancements does READER bring compared to earlier SQL decomposers?
2. How scalable is READ-SQL beyond 3 billion parameters, and are there plans for optimizing even larger-scale models?
3. Through augmented data generation, where question/SQL pairs and question/reason pairs are created, READ-SQL enhances comprehension of complex queries. Open-sourcing this related data would further support the broader community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558654809

thread_type:
official_review

<thread_object>
creator: reviewer_ezyc

<content>
summary:
READ-SQL introduces an innovative approach that leverages Abstract Syntax Trees (ASTs) to efficiently extract chains of reasoning by incorporating each SQL constraint, as well as to identify all sub-SQL components. This strategy supports their development of a two-step process involving generation and correction. The generator is trained on both SQL generation and reasoning tasks, enabling the model to learn step-by-step reasoning necessary for producing accurate SQL queries. Using the constraints generated from the READER, corrector will find the low-confidence constraints and try to fix them. Experimental results indicate that this method achieves state-of-the-art (SOTA) performance with open-source LLMs under 3 billion parameters.

strengths:
1) The primary strength of this paper lies in the READER method. By decomposing SQL queries into sub-queries, the proposed approach effectively facilitates multi-step reasoning, guiding the model to generate the final answer.

2) The proposed approach achieves state-of-the-art performance compared to all models with fewer than 3 billion parameters on two well-known benchmarks: BIRD and Spider.

weaknesses:
1) The main weakness of this paper lies in the experimental section. The baselines used are primarily methodologies that do not represent the most recent advancements in the field, such as Distillery, CHASE-SQL, and CHESS. Although the paper states that the main baseline is the SFT CodeS model itself, it would be beneficial to compare their approach with more advanced fine-tuning methods, such as using zero-shot chain-of-thought prompting to generate reasoning steps and training the CodeS model with those reasoning paths.

2) According to the results shown in Table 2, there are two instances where the performance of READ-SQL is lower than the SFT version of the CodeS model. This may indicate overfitting on the Spider dataset, which could explain why READ-SQL outperforms the SFT CodeS model on Spider specifically.

3) To better assess the effectiveness of their proposed correction methodology, it would be valuable to compare it with basic self-correction methods, such as those outlined in the DIN-SQL paper. This comparison would help determine whether the improvement comes solely from an additional LLM call or from the specific advantages of the proposed method.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721148117

thread_type:
official_review

<thread_object>
creator: reviewer_8ary

<content>
summary:
This paper introduces READ-SQL, a novel framework for enhancing the text-to-SQL task by using a reasoning path decomposer called READER. The framework aims to address challenges such as misinterpreted or omitted constraints in SQL generation. READ-SQL comprises a Generator and a Corrector, both of which leverage READER to break down SQL queries into sub-SQLs and reasoning paths for better data preparation and correction. Extensive experiments show that READ-SQL outperforms competing baselines, achieving strong results with fewer parameters and demonstrating applicability across different models and scenarios.

strengths:
1. The proposed framework achieves promising results using small LLMs (1B and 3B models), which is advantageous for local deployment and reduces computational resource requirements in real-world applications. Given that the text-to-SQL community primarily focuses on larger proprietary LLMs, this work advances research on smaller-scale LLMs, providing practical benefits for scenarios such as on-device AI.

2. This paper introduces common errors in text-to-SQL and presents a detailed error analysis to validate the framework. Experiments conducted on five well-recognized datasets, including BIRD and Spider variants, provide strong evidence of the framework's effectiveness, showcasing comprehensive experimental support.

3. Overall, the framework is technically sound and novel. The paper is well-structured and easy to follow, effectively presenting the methodology, experiments, and results, which allows readers to easily understand the contributions and significance of the proposed approach.

weaknesses:
1. The baseline included in the paper is insufficient. As a submission in October 2024, and in reference to recent studies [1][2], the paper should compare against more advanced baselines, such as [3][4][5][6]. Although this comparison of performance may not be entirely fair due to the varying scale of model parameters, the authors could highlight their advantages in other aspects, such as model size, GPU resources, and runtime. Comparing to up-to-date baselines would strengthen the paper's position for submission to a top conference.

2. The paper lacks a detailed runtime analysis. Given the focus on using smaller-scale LLMs, adding an analysis of training and inference times would underscore the practical advantages of the lightweight design. Such an analysis would further emphasize the framework's efficiency in real-world settings, particularly for on-device and resource-constrained applications, thereby enhancing the paper's impact.

3. The performance of the proposed framework seems limited in terms of generalization across different datasets. While it shows significant improvement on the more challenging BIRD dataset (with more complex database environments), its performance on simpler Spider-based databases is less pronounced, and in some cases, even demonstrates negative optimization. This discrepancy seems counterintuitive, especially since the framework does not explicitly claim to be suited only for complex scenarios.

[1] Zijin Hong, et al. "Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL" arXiv preprint, 2024.

[2] Xinyu Liu, et al. "A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?" arXiv preprint, 2024.

[3] Tonghui Ren, et al. "PURPLE: Making a Large Language Model a Better SQL Writer" In Proceedings of ICDE, 2024.

[4] Mohammadreza Pourreza, et al. "DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models" arXiv preprint, 2024

[5] Boyan Li, et al. "The Dawn of Natural Language to SQL: Are We Fully Ready?" In Proceedings of VLDB, 2024.

[6] Ge Qu, et al. "Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation" In Findings of ACL, 2024.

questions:
1. Can READ-SQL work on larger-scale models (e.g., 13B, 34B LLMs)? Since the proposed framework already achieves comparable performance with small-scale LLMs, it would be interesting to explore whether the framework could outperform GPT-4-based methods when incorporating 70B models. Alternatively, if READ-SQL cannot work on larger-scale models, it would be helpful to explain why.

2. Since the most direct comparison for the model at the same scale is with SFT CodeS-1B and SFT CodeS-3B, could the authors provide a detailed analysis of the specific types of questions where READ-SQL shows improvement? A statistical analysis of correct samples based on difficulty level, along with a straightforward case study comparing READ-SQL with CodeS, would help readers better understand the contributions and improvements.

3. Does READ-SQL struggle when handling ambiguous questions? The improvement brought by READ-SQL in robustness datasets such as Spider-Syn and Spider-Realistic (+1.3 and +0.4) is not as significant as in standard datasets like BIRD and Spider (+2.35 and +0.8). Moreover, why does READ-SQL-3B even show negative optimization in Spider-DK (-0.2)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730933252000

thread_type:
official_review

<thread_object>
creator: reviewer_e8xr

<content>
summary:
This paper presents READ-SQL, a framework that uses SQL decomposition for two main purposes: (1) data augmentation for training an SQL generator and (2) self-correction by training a corrector that uses common and differing clauses among multiple generated SQLs to refine the SQLs. The model demonstrates improved performance over baselines on the BIRD, Spider, and various Spider benchmark variants.

Some comments on presentation and typos:

1. Line 301: "the above three kinds of data" — only two types are explicitly listed.

2. Figure 4: The caption refers to a yellow color not present in the figure.

3. Table 4: The first two rows label “coder-1.3B-1.3B” with “1.3B” duplicated unnecessarily.

strengths:
1. The proposed SQL decomposer, utilizing AST parsing, is both novel and efficient, providing a scalable approach.

2. The self-correction method based on common and differing clauses is innovative.

3. The framework enhances performance for smaller models.

weaknesses:
1. The writing could be improved to be more direct and accessible. Section 3.4 is incomplete — it only describes the extraction of common and different constraints without explaining the subsequent self-correction process. Some insights can be found in Figure 4, but the absence in the main text hinders comprehension.

2. The comparisons with other models are somewhat unfair; READ-SQL-3B includes two 3B models (i.e., a 3B generator and a 3B corrector), so a direct comparison with CodeS-3B may not be appropriate.

3. The performance improvements are modest, especially considering the combined use of two 3B models.

questions:
1. Line 44 mentions three typical categories of text-to-SQL errors. Are there any statistics on the frequency of these error types?

2. How would the performance be affected if question-SQL pairs and question-reason pairs were combined into question-reason-SQL chain-of-thought data for fine-tuning CodeS?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732513707678

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Comment (part 1)

comment:
> **Q1: Contribution of our Work**

**A1:** We argue that the contribution of READ-SQL is not merely an incremental improvement over existing Text-to-SQL methods; rather, it has the potential to inspire mainstream adoption for real-world Text-to-SQL applications.

Additionally, the developed SQL decomposer, READER, has proven effective in enhancing performance, with a 3B LoRA model surpassing a 7B fully fine-tuned model on the BIRD development set.  For a more detailed analysis, please refer to A2 below. Furthermore, the innovative design of READER can inspire new ideas for code analysis in other languages, such as Python and C/C++.

> **Q2: Detailed analysis of READ-SQL generalizability**

**A2:** We present our experimental findings, demonstrating that READ-SQL consistently outperforms SFT CodeS across various domains, with no signs of overfitting, except on the Spider Test Set at the 3B scale.
To evaluate the generalization capabilities of READ-SQL, we reproduced the results of SFT CodeS on three Spider variants (Spider-Syn, Spider-Realistic, and Spider-DK), allowing for a detailed comparison across different difficulty levels. These results are included in Appendix A.5.9 of the revised paper. For fairness, we report only the Generators and their corresponding EX scores for each dataset.



| Dataset          | Model        | Easy       | Medium     | Hard       | Extra       | All        |
|:-----------------|:-------------|:-----------|:-----------|:-----------|:------------|:-----------|
| Spider-Syn       | SFT CodeS-1B | 80.2       | 68.0       | 58.2       | 40.2        | 64.7       |
| Spider-Syn       | Generator-1B | 77.4(-2.8) | 69.5(+1.5) | 61.0(+2.8) | 39.6(-0.6)  | 65.1(+0.4) |
| Spider-Syn       | SFT CodeS-3B | 83.4       | 74.8       | 67.8       | 58.0        | 73.1       |
| Spider-Syn       | Generator-3B | 84.3(+0.9) | 76.1(+1.3) | 70.6(+2.8) | 56.2(-1.8)  | 73.9(+0.8) |
| Spider-Realistic | SFT CodeS-1B | 88.1       | 77.3       | 63.6       | 41.2        | 70.1       |
| Spider-Realistic | Generator-1B | 87.2(-0.9) | 79.8(+2.5) | 69.7(+6.1) | 42.3(+1.1)  | 72.2(+2.1) |
| Spider-Realistic | SFT CodeS-3B | 95.4       | 84.2       | 70.7       | 57.7        | 78.9       |
| Spider-Realistic | Generator-3B | 93.6(-1.8) | 86.2(+2.0) | 74.7(+4.0) | 62.9(+5.2)  | 81.1(+2.2) |
| Spider-DK        | SFT CodeS-1B | 79.1       | 69.5       | 51.4       | 40.0        | 63.2       |
| Spider-DK        | Generator-1B | 79.1(+0.0) | 70.7(+1.2) | 51.4(+0.0) | 50.5(+10.5) | 65.8(+2.6) |
| Spider-DK        | SFT CodeS-3B | 82.7       | 76.8       | 55.4       | 54.3        | 70.7       |
| Spider-DK        | Generator-3B | 82.7(+0.0) | 76.4(-0.4) | 62.2(+6.8) | 46.7(-7.6)  | 69.9(-0.8) |

Our findings are as follows:


- The Generator generally outperforms SFT CodeS on medium and hard difficulty levels.
- On easy and extra-hard levels, its performance is occasionally weaker than SFT CodeS.
- A significant performance gap between READ-SQL and SFT CodeS is observed in Spider-DK.
- These results indicate that adding extra training data significantly benefits the model on medium-difficulty tasks but may slightly hinder performance on simpler or extremely complex tasks.  More analysis can be explored in the future work.
- This further suggests that READ-SQL leverages additional training data more effectively on challenging datasets, such as BIRD, compared to simpler ones like Spider.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732513806329

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Comment (part 2)

comment:
> **Q2: Detailed analysis of READ-SQL generalizability (part 2)**

In the reproduction process, we adopt open-source check-points and the code but observe that the performance of SFT CodeS does not fully align with the results reported in the paper. We share the inference logs in the supplementary material. Below is the updated results (only EX):

| model        | BIRD Dev | Spider Dev | Spider Test | Spider-Syn   | Spider-Realistic | Spider-DK    |
| :----------- | :----------- | :------------- | :-------------- | :----------- | :--------------- | :----------- |
| SFT CodeS-1B | 49.54        | 77.80          | 77.50           | 64.70        | 70.10            | 62.80        |
| Generator-1B | 51.76(+2.22) | 80.20(+2.40)   | 77.00(-0.50)    | 65.10(+0.40) | 72.20(+2.10)     | 65.80(+3.00) |
| READ-SQL-1B  | 52.87(+3.33) | 80.70(+2.90)   | 78.80(+1.30)    | 65.00(+0.30) | 71.50(+1.40)     | 65.60(+2.80) |
| SFT CodeS-3B | 55.02        | 83.30          | 81.90           | 73.10        | 78.90            | 70.30        |
| Generator-3B | 56.98(+1.96) | 83.80(+0.50)   | 80.90(-1.00)    | 73.90(+0.80) | 81.10(+2.20)     | 69.90(-0.40) |
| READ-SQL-3B  | 57.37(+2.35) | 84.20(+0.90)   | 81.20(-0.70)    | 74.00(+0.90) | 80.30(+1.40)     | 71.60(+1.30) |
| SFT CodeS-7B | 57.00        | 85.40          | 83.50           | 74.80        | 82.30            | 72.90        |

Our findings suggest the following:

- A standalone Generator typically outperforms the SFT CodeS.
- At the 1B scale, on three Spider variants, incorporating a Corrector does not surpass the performance of the Generator alone but still achieves higher results than the SFT CodeS.
- In out-of-domain scenarios, READ-SQL exhibits minimal or slightly negative performance gains, while its improvements are more pronounced on in-domain datasets.
- READ-SQL does not outperform SFT CodeS on the Spider Test.
- On the BIRD Dev, READ-SQL surpasses SFT CodeS-7B. Notably, even with just the Generator, READ-SQL-3B demonstrates competitive performance, trailing the SFT CodeS-7B by only 0.02%.

The following problems exist in the above phenomenon:
- Why is Corrector less effective than Generator on the Spider-related datasets ?

We compare the average input length of Corrector in the BIRD Dev, Spider Dev, Spider Test and related variants. The results are as follows:


|         | BIRD Dev | Spider Dev  | Spider Test | Spider-Syn   | Spider-Realistic | Spider-DK    |
| :----------- | :----------- | :------------- | :-------------- | :----------- | :--------------- | :----------- |
| Input length | 1468        | 697          | 712           | 738        | 663            | 792 

The results indicate that, due to the lack of external information in the Spider-related datasets, such as column descriptions and column value details, Corrector has limited information to utilize, leading to occasional misjudgments. Corrector performs less effectively on the Spider-related datasets compared to the BIRD dataset.


> **Q3: Re-explaining the reasoning process of READ-SQL**

**A3:** The main text lacks a detailed explanation of READ-SQL's reasoning process. We have added a detailed reasoning process in lines 304-323 of the revised version. Here, we provide supplementary information:

READ-SQL's reasoning unfolds in two key stages: SQL generation and self-correction.
SQL generation : 
- Step 1: We adopt the method proposed by CodeS[1] for schema linking and to structure the input for the Generator model.
- Step 2: A special prefix token, [SQL], is appended to the input, prompting the Generator model to produce SQLs. 

Self-Correction process is shown in Figure 4:
- Step 1: Parsable SQL in the results are parsed one-by-one by READER into a clause set. 
- Step 2: By comparing the clause set of each SQL, we identify high-confidence and low-confidence clauses. 
- Step 3: The high-confidence clauses are combined to form the basic SQL.
- Step 4: Table names, column names, and values are extracted from low confidence clauses by Elasticsearch to retrieve relevant database information, including table schema, foreign keys, and value matching details. 
- Step 5: The final input is constructed following the method shown in Figure 4 and fed to the Corrector.
- Step 6: The first executable SQL generated by the Corrector is selected as the final result.



reference:

[1] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proc. ACM Manag. Data, 2(3):127, 2024b. doi: 10.1145/3654930. URL https://doi.org/10.1145/3654930
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533546303

thread_type:
official_comment

<thread_object>
creator: area_chair_aaf5

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:
- November 26: Last day for reviewers to ask questions to authors.
- November 27: Last day for authors to respond to reviewers.
- November 28 - December 10: Reviewer and area chair discussion phase.

Thank you for your timely attention to this matter.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes READ-SQL, a framework that improves text-to-SQL performance through SQL decomposition and self-correction mechanisms. The framework utilizes AST parsing to decompose SQL queries into components and reasoning paths, employing a Generator for initial SQL formulation and a Corrector for refinement based on confidence levels. 

Some reviewers acknowledge the framework's strengths in achieving competitive performance with smaller models and its novel approach to data augmentation through SQL decomposition. The results show improvements over baselines on benchmarks like BIRD and Spider, particularly for models under 3B parameters. However, reviewers raise several significant concerns. First, the technical novelty appears incremental, with unclear advantages over existing approaches like schema linking and end-to-end methods. Second, the experimental evaluation lacks comparisons with recent advanced baselines, and the performance gains may be attributed to overfitting, particularly on the BIRD dataset. Third, the paper's presentation has notable gaps, especially in explaining the self-correction process and inference pipeline. While the authors provided responses to these concerns, multiple reviewers indicated that their core reservations about technical novelty and contribution remain unaddressed. 

Overall, given these fundamental limitations, the work would benefit from substantial improvements before meeting the publication threshold.