Title: SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Text-to-SQL

Error Detection

Error Correction
Abstract: Text-to-SQL systems translate natural language (NL) questions into SQL queries, enabling non-technical users to interact with structured data. While large language models (LLMs) have shown promising results on the text-to-SQL task, they often produce semantically incorrect yet syntactically valid queries, with limited insight into their reliability. We propose SQLens, an end-to-end framework for fine-grained detection and correction of semantic errors in LLM-generated SQL. SQLens integrates error signals from both the underlying database and the LLM to identify potential semantic errors within SQL clauses. It further leverages these signals to guide query correction. Empirical results on two public benchmarks show that SQLens outperforms the best LLM-based self-evaluation method by 25.78% in F1 for error detection, and improves execution accuracy of out-of-the-box text-to-SQL systems by up to 20%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SQLens, an error detection and correction framework for text-to-SQL translation. The authors introduce a weak supervision approach to aggregate database and LLMs error signals and iteratively fix the most critical errors. Evaluation on BIRD and Spider shows performance improvements, presenting its effectiveness in error detection and correction, and the authors also include ablation studies to validate the effectiveness of each module and signal.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. Semantic errors remain a bottleneck in current text-to-SQL translation, and studies in this paper is timely.\n2. The use of weak supervision to aggregate noisy error signals is a new and great attempt, as it also circumvents the need for labeled data.\n3. The experiments and ablation studies are thorough and systematic, and the reported results are convincing.\n\n------\n\n### **Weaknesses**\n\n1. While the iterative correction framework provides marginal improvements over the fix-all setting, it introduces concerns about increased token cost and latency, which may hinder practical deployment.\n2. Although I believe weak supervision for aggregating noisy error signals is a good attempt, I still have major concerns:\n   - The quality and accuracy of individual LFs are not well validated and seem largely heuristic.\n   - The weak supervision model is treated as a black box, with little analysis of its sensitivity to LF quality, number, or correlation. It’s also unclear how much value comes from the aggregation model versus a few strong signals used directly. If possible, an ablation study on individual signals could help clarify this.\n   - In table 4, the recall of error detection for individual error signals are quite low, indicating the difficulty of effectively detecting these signals.\n   - While results on Spider are reported, it is unclear whether the classifier used in that setting was trained on Spider training set or transferred from another dataset.\n   - The paper omits important details about the training setup for the weakly supervised classifier, such as the model architecture and training procedure.\n3. Some key technical details lack sufficient clarity and elaboration:\n   - Line 209: \"Error Selector uses an LLM to prioritize which error to fix first...\" – How is this prioritization determined by the LLM in practice? By prompt or some other strategies?\n   - Lines 180-183: \"We use a generative model to estimate the joint distribution... and train a classifier...\" – How exactly is this classifier trained? What is the backbone model and its training strategies?\n   - Line 289: \"We conduct a microbenchmark on BIRD...\" – How is the microbenchmark constructed? Is it based on random or stratified sampling? How many examples are included, and how are the signal labels obtained?\n\n------\n\n### **Minor**\n- I personally think the definitions and examples in the main text are a bit verbose; condensing them could free up space for more technical details.\n\nQUESTIONS:\n**Please refer to the weaknesses.**\n\n**Additional Questions:**\n1. The EX performance of DIN-SQL is inferior to vanilla in Table 1, which is somewhat counterintuitive.\n2. The error correction methods in the experiment show an improvement of a few percentage points on other text-to-SQL approaches, but *alone on DIN-SQL*, all methods have an extremely high increase (15-20%), which is a bit questionable.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose SQLENS, a framework for detecting and correcting semantic errors in LLM-generated SQL queries. While the problem is important and the approach is systematic, the contribution feels incremental and the evaluation has some limitations.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\nS1. Authors propose an important problem: semantic error detection in text-to-SQL.\n\nS2. The signal design is comprehensive. Authors show 14 error signals which covers diverse erry types and are well-motivated by real SQL problems.\n\nS3. Experiments on standard benchmarks with multiple baseline text-to-SQL methods are promising.\n\n# Weakness:\nW1. The problem is similar to self-correction, but authors didn't compare those popular self-correction methods in the experiment, such as Magic [1].\n\nW2. The improvement is marginal. \n\nW3. The method has been only verified on one LLM backbone: Claude 3.5 Sonnet.\n\nW4. There are no cost analysis for this methods. Only LLM calls are not professional and questionable. For example, whether this latency is relevant to network connect? Is there a linear relationship between number of LLM calls and cost? Given authors design specific prompt techniques leading to more token consumption in each call, so number of LLM calls cannot be equal to the cost. Also such reduction can be observed in Other llm backbone?\n\n\n\n# Reference:\n[1] MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL.\n\nQUESTIONS:\nq1. Whether this latency is relevant to network connect? Is there a linear relationship between number of LLM calls and cost? Given authors design specific prompt techniques leading to more token consumption in each call, so number of LLM calls cannot be equal to the cost. Also such reduction can be observed in Other llm backbone?\n\nq2. Is this work also effective to other LLM backbone?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new framework for Text2SQL. To goals are threefold: (1) Detect errors in the input, (2) Fix the error, (3) Provide the fixed version as final output. Tha proposed approach is based on employs a learning-based approach that aggregates complementary signals identifying potential errors.\n\nSTRENGTHS AND WEAKNESSES:\nI agree with the authors that the problem of Text2SQL is far from being completely solved. A huge body of work has been devoted to specializing LMs to effectively address the task of adopt in-context learning to address the limitations of existing LLMs. In this work, instead, the authors mainly address the issues of semantic error detection and solving, which are particularly relevant to this field.\n\nI have concerns on the strategy used to produce reliable error signals. The learning-based approach based on weak supervision is quite risky, as a lot of false signals are likely generated on complex datasets. The current results, reported on Bird and Spider v1, fail to demonstrate the actual robustness of the approach to more complex cases (see, e.g., Spiderv2, BEAVER).\n\nThe formulation of the concept of semantic ambiguity is not clear enough. The related literature appears to be also incomplete. See, for example, \n\nFuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and Amr El Abbadi. 2025. Sphinteract: Resolving Ambiguities in NL2SQL through User Interaction. Proc. VLDB Endow. 18, 4 (December 2024), 1145–1158. https://doi.org/10.14778/3717755.3717772\n\nWithout a clearer formalization of semantic ambiguity, a detailed comparison with all SOTA, and an empirical evaluation on complex datasets the novelty and effectiveness of the proposed approach remains questionable.\n\nQUESTIONS:\n- Is the definition of semantic ambiguity consistent with the SOTA?\n- Are you able to empirically compare with the SOTA on specific ambiguity types?\n- How did the authors handle databases with a large number of tables and more complex queries (e.g., in business domains)?\n\nLIMITATIONS:\n- The authors explored a relatively limited number of LLMs and prior works. The scope of the empirical evaluation and the experimental comparisons with SOTA can be expanded.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SQLENS, an end-to-end framework for fine-grained detection and correction of semantic errors in LLM-generated SQL queries. To address the issue of syntactically correct but semantically incorrect outputs, SQLENS integrates error signals from both the underlying database and the LLM to identify potential errors at the clause level. It uses weak supervision to aggregate noisy signals and\ntrain a classifier for error detection, and introduces a step-by-step correction mechanism guided by detailed error reports. Experimental results show that SQLEN outperforms existing LLM-based self-evaluation methods by 25.78% in F1-score for error detection and improves execution accuracy by up to 20% on benchmark datasets such as BIRD and Spider.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n- The paper proposes fine-grained text-to-SQL error categories that cover the majority of possible errors.\n- This paper conducts a comprehensive comparison across SQL generation and various baselines, demonstrating that SQLENS effectively improves the final execution performance.\n\n\n## Weaknesses:\n- Although the paper selects comprehensive SQL generation methods as baselines, it does not make a comprehensive selection of fix baselines, such as SQLFixAgent (https://arxiv.org/abs/2406.13408 ) and Self-debug (https://arxiv.org/abs/2304.05128 ).\n- The paper only uses Claude 3.5 Sonnet as the correction model, which limits the generalization ability of SQLENS. Is the method effective on other closed-source models (e.g., GPT-4o) or open-source models (e.g., Qwen)?\n\nQUESTIONS:\n- The workflow of the paper is not clearly explained. How many error reports are generated for one SQL query? Can the authors show an end-to-end example of a SQL query from generation to correction?\n- In the Error Selector, which errors are most frequently selected first for correction?\n- How many tokens does SQLENS require to fix a single SQL query?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}