SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
Authors: Yue Gong
Chuan Lei
Xiao Qin
Kapil Vaidya
Balakrishnan Murali Narayanaswamy
Tim Kraska
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Text-to-SQL
Error Detection
Error Correction
Primary area: applications

Abstract: Text-to-SQL systems translate natural language (NL) questions into SQL queries, enabling non-technical users to interact with structured data. While large language models (LLMs) have shown promising results on the text-to-SQL task, they often produce semantically incorrect yet syntactically valid queries, with limited insight into their reliability. We propose SQLens, an end-to-end framework for fine-grained detection and correction of semantic errors in LLM-generated SQL. SQLens integrates error signals from both the underlying database and the LLM to identify potential semantic errors within SQL clauses. It further leverages these signals to guide query correction. Empirical results on two public benchmarks show that SQLens outperforms the best LLM-based self-evaluation method by 25.78% in F1 for error detection, and improves execution accuracy of out-of-the-box text-to-SQL systems by up to 20%.

SQLENS: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
Yue Gong
Chuan Lei
Xiao Qin'
Kapil Vaidya
Balakrishnan Narayanaswamy"
Tim Kraska
Amazon Web Services o Oracle Corporation
*Snowflake Inc.
Parallel Massachuseett Institute of Technology
(yuegongy, muralibn, timkraska @amazon. com, chuan 1ei Lei@oracle. com, xiao. qin@snowf lake. com, kapilvaidya228gmail. com
Abstract
Text-to-SQL systems translate natural language (NL) questions into SQL queries, enabling non-technical users to interact with structured data. While large language models (LLMs) have shown promising results the text-to-SQL. task, they of- ten produce semantically incorrect yet syntactically valid queries, with limited insight into their reliability. We propose SQLENS, an end-to-end framework for fine-grained detection and correction of semantic errors in LLM LLM--ennrated SQL. SQLENS integrates error signals from both the underlying database and the LLM to identify potential semantic errors within SQL clauses. It further leverages these signals guide query correction. Empirical results on two public benchmarks show that SQLENS outperforms the best --sed self-evaluation method by 25. 78% in F1 for error detection, and improves execution accuracy of out-of-the- box text-to-SQL. systems by up to 20%.
Introduction
Text-to-SQL systems translate natural language (NL) questions into SQL query, enabling non-
technical users to query relational databases and extract insights [32] Large Language Models (LLMs)
have significantly advanced this task, with recent methods [44] 34 29 [6] achieving promising results
on public benchmarks such as BIRD (25) and Spider [55]. Such LLM-based text-to-SQL. solutions
have been adopted by data platforms like AWS 4. Databricks
Snowflake
41 etc.
Despite these advances, 4LL--genrrted queries remain error-prone [4]. The best pest-performin method on the BIRD leaderboard [5] only achieves execution accuracy of around 75% on the dev set, still producing more than 400 incorrect SQL queries out of 1534 NL questions. While considerable effort has been devoted improving the retrieval and generation stages in LLM-bbssed text-to-SQL systems [44 34 29 such as schema linking, Chain-of-Thought (CoT) prompting, task decomposition, and the inclusion of cell values, these approaches are still insufficient to eliminate errors entirely, as LLMs do not always adhere to the provided instructions.
A major gap in text-to-SQL systems is the lack of fine-grained explainable error detection-especialyy for semantic errors, where queries execute but return incorrect results. Detecting such errors requires understanding both query logic and database structure, which most existing methods fail to capture [33
48 23). They generate SQL queries without estimating quality or confidence. making it difficult for users to debug [22 In this paper, address three challenges in error detection and correction:
Work done at AWS by Chuan Lei, Xiao Qin, and Kapil Vaidya.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Challenge 1: identify semantic errors at the clause level. Semantic enrors often arise from various causes, from inconsistencies with external knowledge to the LLM's misunderstanding of the database schema. Identifying them is challenging due to NI ambiguity and the complexity of SQL queries, and requires joint reasoning over all these sources.
data, and schema 44 18
Challenge 2: predict the correctness of a SQL query with noisy error signals. Error signals are noisy proxies for true semantic errors. Each signal evaluates the semantic correctness of the generated SQL from a specific perspective, potentially missing the full context. The challenge is to aggregate the decisions from multiple noisy signals to determine whether SQL query semantically incorrect.
Challenge 3: fix SQL queries without correctness oracle. Correcting SQL queries is difficult because it must be done cautiously to avoid breaking queries that may already be correct, no correctness oracle is assumed. Additionally, the order in which errors are fixed is crucial, as errors can be interdependent, and addressing the root cause can resolve many related issues.
Stafe-of-the-art approaches. Existing text-to-SQL methods often rely on LLM self-reflection for error detection and correction [3 23
but suffers from self- seff-preference bias 31 52 resulting in low recall. RL-based approaches are emerging, using rewards based on execution accuracy, syntactic correctness, or LLM evaluations 36 27.. but still lack the granularity needed for precise error identification. These methods typically offer only a single confidence score for the full SQL query, without pinpointing specific erroneous clauses explaining the cause- cause-limiting debuggability and reducing trust in text-to-SQL assistants [6]
Error Reports
Most Critical Error
Error Fixer
Error Detector
Error Selector
Stop when errors detected or the maximum iterations are reached
Original SQL Query & NL question
Update
Revised SQL Query
Error Detector
Error Fixer
Corrected SQL Query
SQL Auditor
Final Revised SQL Query
Guardrail Signal
Figure 1: An overview of SQLENS.
SQLENS overview. We introduce SQLENS, an end-to-end framework that leverages diverse error signals from both the database and LLM to detect and correct semantic SQL errors. As shown in Figure D SQLENS parses the SQL query into an Abstract Syntax Tree (AST) and uses its Error Detector to collect noisy signals for identifying potential errors (Challenge 1). To handle noisy signals, SQLENS uses weak supervision to combine them into probabilistic ground truth-ani trains a classifier to predict semantic correctness and generate detailed error reports (Challenge 2). SQLENS's Error Selector then decomposes the SQL correction task and guides the LLM through iterative fixes (Error Fixer) based on prioritized errors (Challenge 3), and the SQL Auditor selects the better query between the original and revised versions.
needing
Contributions. Our contributions be summarized follows: (1) we present SQLENS that holistically exploits fine-grained error signals for predicting the semantic correctness of SQL queries and correcting erroneous queries using these signals; (2) SQLENS leverages both LLM and database signals to diagnose SQL queries and employs weak supervision framework to aggregates noisy error signals for estimating semantic correctness; (3) we design sequential error correction strategy that guides the LLM to fix SQL queries step-by-ster prioritizing the most critical errors to minimize cascading mistakes; and (4) experimental results on BIRD [55 and Spider (53) show that SQLENS improves semantic error detection by 25. 78% in F1 score over the best LLM self-evaluation method and boosts execution execution accuracy by an average of 3% over existing text-to-SQL solutions.
2 Related Work
Text-to-SQI. Large language models (LLMss) have demonstrated significant capabilities in natural language understanding the model scale increases. LLM-based text-to-SQL systems, such as

---

In this paper, table T consists of of columns C (C1, C2...., Cn) A join relationship J between two tables T; and Tý is based on common attributes (i.c., joinable columns T.COm and Ty.Cn, respectively). database instance D {(Ti, T2,,,,,.. .2,..,17y ,,), J comprises a of tables and Question (2) How many clients opened their accounts in Jesenik branch were women?
External Knowledge (RC): has region names; female refers gender
SELECT
dient
distric
account
COUNT(DISTINCT client.client
distictid
disikui
Join Graph
66 Sub-optimal Join free
SELECT
cLient
COUNT(DISTINCT client.client.
INNER 10IN account account acrount
FROM
ELLien..clinnt
client
INNER JOIN district ON By: Incoriet
INNER JOIN district ON
accomt.distrric 1d districc.
client.districs id h district.
WHERE
WHERF
<liinntgenndrr
eg
Violation
'F' AND district.A2 "Jesenkk';
client-gender
AND district.A2 "jesenik') ex: Empty Predicote
Generated SQL Query [Output: 0)
Ground Truth SQL Query (Output: 26)
Figure A running example from BIRD, where each client is guaranteed have account.
48 decompose the task into sub-tasks using multi-agent frameworks for effective execution. Other approaches like MCS-SQL [23], CHESS [44] and Chase-SQL [34] follow similar workflows-retriee
DIN-SQL 33 and MAC-SQL
relevant context, selecting schema, and synthesizes SQL queries. Recently RL-based reasoning models for text-to-SQL have been explored, where rewards are typically based execution accuracy, syntactic correctness, or M-baaedd self-evaluation [36 27. Our proposed error detection can be naturally integrated into this framework as reward functions, providing more semantically meaningful supervision guide and enhance model reasoning.
Error defection and correction in text-to-SQL. Early approaches g 53
56 focus primarily on
binary correctness classification, offering limited insight into the nature of the errors. LLM-based text- often rely on either SQL execution feedback or LLM self-reflection to judge correctness. In contrast, SQLENS produces interpretable, clause-level error signals that can guide both user debugging and downstream learning frameworks. L.I. self-evaluation techniques (20 17 assess response quality, while Tian al. [45) and Self-RAG []] improve confidence estimation and factual accuracy through fine-tuning and retrieval-basee self-critique. However, these are built for general tasks and do not tackle text-to-SQIE specific challenges.
to-SQL systems
33
35 23
3 Methodology
3.1 Problem Formulation
a set of join relationships J between these tables.
Definition (Text-to-SQL Algorithm) A text-to-SQL algorithm f takes as input a natural language question Q, database instance D, and optionally external knowledge K. and generates SQL query = f(Q,D,
Figure 2 presents an example using a question from the BIRD benchmark, asking for the number of female clients who opened accounts at the Jesenik branch. The benchmark also provides external knowledge, such as annotations on a column name and a cell value. The predicted SQL query is generated by text-to-SQL algorithm to answer the question.
Definition (Semantic Error) A semantic error results in the SQL query q failing correctly answer the natural language query Q. Formally,
do(e) = Olq, +
The operation do(e) denotes intervention in the generation of q due e, causing mismatch between the observed output Ogq. D) and the expected correct output ( D), thereby identifying as the semantic error.
For example, the generated SQL query Figure 2 is semantically incorrect with an output of whereas the correct output is 26 based on the ground truth SQL query. First, the query incorrectly

---

Question Ambiguity
Data Ambiguity
insufficient Evidence
Value Ambiguity
Column Ambiguity
Table Similarity
Legend
Signal
Incorrect Question Classe Linking
Eridence Violation
Abnormel Result
Incorrect Join Predicate
LLM Signal
Incorrect Subquery
Unnocessary Subquery
Empty Predicate
Suboptimal
Semantic Misalignment
Figure 3: The causal graph of semantic errors and signals
uses jesenik in the predicate, leading to the empty result. Secondly, the query violates the evidence specified in the external knowledge, using gender 'Female Femmlee' instead of gender= 'F'. Even with the correct predicates, the SQL query would still produce an incorrect result of 23 due an incorrect join predicate and suboptimal join path. Specifically, there is no valid join path between the client and account tables, indicating the join predicate client. client__ieeee __id account. account, id in the query is hallucinated. Moreover, the account table involved the join path is redundant as the client and district tables can be directly joined based the database.
Problem 3.1 (Semantic Error Detection) Given natural language question Q. database in- stance D, optionally external knowledge K,, and the output SQL query q = f(Q, D, P.) generated by lext-to-SQL algorithm the task is determine q semantically correct and, not, identify set of potential semantic errors & in
For a semantically incorrect SQL query, our goal is to fix it to correctly answer the given NL question.
Problem 3.2 (Semantic Error Correction) Given a semantically incorrect SQL query q, the task is correct such that the corrected query d', accurately answers the natural language question Q. such that ( D) = Og',, D).
3.2 SQLENS Error Detection
Accurately identifying semantic errors can be inherently difficult due to the complexity of SQL queries, as well as the ambiguities present natural language (NL.) queries, data, and database schemas. Our insight is that many semantic errors in L.LM-generated queries exhibit common patterns that can be detected through carefully crafted signals. Building on observations and insights from recent text-to-SQL studies 48 23 35), categorize common semantic errors follows.
1. Question ambiguity. The user's 's questions may inherently contain ambiguities and be interpreted in multiple ways. For example, if user asks "What were the total sales last quarter?" in database where the sales table includes both gross_sales and net net_sail sales columns, either column could be chosen
the question.
2. Data ambiguity. In real-world databases, multiple tables columns with similar or identical names can arise due to data integration, versioning, table transformations, and other factors, leading to ambiguitics. For example, if asks "What are the average salaries by department?", but the database contains both a dept table and a department table, selecting the wrong table would result in incorrect query results.
3. Semantic misalignment. Even when NI. questions and databases are unambiguous, a semantic gap can still cause mismatches between the generated SQL and the NI. question. For example, as shown in Figure 2 an incorrect join predicate (client. client id account. www.uutttcccottiioo occurs due to the LLM misinterpreting join relationships the financial database.
An error signal acts proxy for detecting semantic errors in SQL queries. SQLENS's Error Detector combines signals from both the database and LLM to detect these errors. Figure illustrates how each signal in SQLENS corresponds to common types of semantic errors.
Database-based error signals. Database-based signals are designed to identify semantic mis- alignment and inherent ambiguity within the data. These signals draw inspiration from diverse SQL workloads including real-life scenarios and benchmarks such as TPC-DS 30, Redset [46].
BIRD 225), etc. These signals can be reliably and efficiently obtained without LLMs by: (1) extracting

---

Database Signals
Sub optiooatllll Join Tree
Labeling Function 1
Correctness:
Error Signal: Detects evidence
Wrong
Enror Report
Incornect Predicate
Labeling Function 2
violation that yields an empty
Empty Predicate
Labeling Function 3
Corractness Prediction
Correction Instruction:
Problematic Clauses: (district.AZ 'eeenikk clent.gender = 'Female']
LLM Signals
Evidence Violation
Labeling Function
Nolsy Labelers
Generative Model
Figure 4:
Signal aggregation using weak supervision.
information from the query plan (e.g., Empty Predicate). (2) checking (meta- >daa information from the underlying database (e.g., Suboptimal Join Tree, Value Ambiguity), or (3) leveraging general heuristics derived from the above query workloads (e.g., Unnecessary Subquery). Appendix we present examples of incorrect and correct SQL query pairs based on NL questions from weal-world applications, highlighting the specific SQL clauses each signal targets.
LLM-based error signals. For semantic errors due to insufficient evidence or LLM hallucination, we need consider both the NL question and the SQL query simultaneously and understand the LLM's reasoning process. Therefore, SQLENS introduces LLM- error signals that dig into question ambiguity and the LLM's reasoning process [50]. All LLM-based signals are obtained from single LLM call, significantly reducing the cost and latency compared to multiple individual calls. Detailed descriptions of the error signals, along with illustrative examples, are provided in
Appendix C
The corresponding prompts used to collect these signals included in Appendix D
Effective error detection in text-to-SQL requires integrating both database and LLM-based signals: the former excel at identifying structural or value errors, while the latter capture logical inconsistencies requiring nuanced interpretation of the NL question and query. Since these signals are inherently noisy, SQLENS avoids relying on any single source. Instead, it employs a learning- -baed approach (Section 3.3 to aggregate complementary signals, combining database analysis with LLM reasoning. This makes SQLENS general and extensible framework for robust error detection and correction.
3.3 Aggregating All Signals Using Weak Supervision
The signals that collect are inherently noisy-each may incorrectly flag correct SQL clauses or overlook actual errors. To mitigate this, we adopt a weak supervision framework (shown in Figure 4 that aggregates these diverse signals, treating each as a labeling function (LF) that provides partial and noisy supervision. Inspired by prior work [38 [5], this approach not only combines the signals but also learns their accuracies and correlations to better approximate ground truth. In SQLENS, each error signal 8 acts as a L.,, identifying potential issues in SQL query q. Formally,
if [()) > 0 I= Ax) {-1 if [((()) =
where I = 1 indicates that the error signal s has detected least one problematic SQL clause (i.e., likely incorrect), and I = -1 indicates that no issues are found. However, relying solely negative signals limits coverage. To improve recall, we introduce three positive labelers that assign I (i.e., likely correct) when no issues are detected by: (1) Xat labels a SQL query as correct if no error signals are detected; (2) >as labels a SQL query as correct if no database-based signals are detected; and (3) Aum labels SQL query correct if no LLM-based signals detected.
Each query q is thus represented by decision vector Ag { (), ... )s, (q). Awr(g), Aao Aum ). We use a generative model to estimate the join distribution p(A, Y), where Y is the unobserved true label, by optimizing:
Oppt arg min log Epo(A,Y)
The resulting probabilistic labels p(A, (A, Y) reflect the estimated correctness of each SQL query, which use to train classifier for downstream semantic error detection.
5

---

"signal description": "The SQL query uses more tables than necessary in the join 4 clauses, which may lead to potential errors.", "correction instruction": "Review and revise the SQL query to include only the essential tables in the join clauses.", "problematic clauses";
"tables used in the JOIN clauses": ["client", "account", "district"), "optimal set of tables to join"; ["client", "district"]
},
"confidence"; "high"
}
Figure 5: Example error report for Suboptimal Join Tree
3.4 SQLENS Error Correction
9 often rely on self-reflection without grounding their corrections. This leads to two issues: (1) hallucinations due generation bias B1 [2 and limited factual grounding [39]. and (2) degradation of originally correct queries when relying on single, unreliable correctness estimate. Other methods similarly assume that all SQL queries are incorrect 11 2), ignoring the risk of over-correction.
As discussed in Section [ LLM-based text-to-SQL methods
33
SQLENS error report. In SQLENS, an error signal is associated with SQL clauses and provides detailed error report. An error report consists (1) signal description which explains the error signal and the conditions that trigger it; (2) example(s) (optional) to clarify the meaning of the signal; (3) correction instruction for correcting the SQL query; (4) problematic clauses identified as potential sources of error. Such error report offers rich context for the identified errors in SQL queries; and (5) confidence bucketized into high, medium, and low based on the weight assigned to each labeler in error detection. Figure 5 presents example of an error report generated by the Suboptimal Join Tree signal for the SQL query shown in Figure 2
SQLENS correction strategy. A straightforward approach of using error signals for SQL query correction is to concatenate all the detected error information and ask the L.MM to fix all errors at once. However, providing too much information can overwhelm the LLM, causing it to lose focus [26]. To address this, SQLENS decomposes the SQL correction task, guiding the LLM to fix the query step-by-step. SQLENS prioritizes fixing the most critical error in each iteration, allowing the LLM to focus on resolving one issue at a time, reducing the risk of distraction [21 |9 [1] By addressing the most critical error first, the total number of errors can also be reduced more quickly, dependent errors can be automatically resolved if the root cause is fixed. This iterative process continues until no error signals remain or the maximum number of iterations is reached. Algorithm [ Appendix E presents the pseudocode for the error correction. The correction prompt is provided in Appendix D.8
Error Selector. Given SQL query, schema, and ranked list of error reports, the Error Selector uses an LLM to prioritize which error to fix first, refining the ranking based on relevance to the original query 43
28
37 The highest-priority error is then passed to the Error Fixer.
Error Fixer. The Error Fixer takes the query and a selected error report, applies a targeted correction using contextual signals (Algorithm) D and validates the revised query using SQL parser. syntax issues arise, the fixer uses the parser feedback iteratively resolve them. Corrected queries are re-evaluated by the Error Detector; the process repeats until no errors remain or a maximum number of iterations reached. Fixed signals are removed to prevent redundant corrections.
SQL Auditor. To avoid overlooking persistent errors, a high-precision signal can he designated as guardrail to trigger final adjustments. To prevent degradation of originally correct queries, the final revised query and the original are both evaluated by an LLM--based SQL Auditor, which selects the version that best aligns with the input question.
4 Experimental Evaluation
4.1 Experimental Setup
Datasets. We evaluate on the dev sets of two standard text-to-SQI. benchmarks: BIRD 25 and Spider [55]. SQL queries for evaluation were generated using four state-of-the-art text-to-SQL

---

Table 1: End-to-end aceuracy improvement on BIRD.
Method
A mcc.(Nnaa)
A Acc.(Nix))
Nad
Nix
Nurok
Self-Reflection
59.07 (+0.00%)
60.6(+1153%)
22
21
Vanilla
SQLENS Fix-ALL
61.15 (+2.08%)
62.34 (+3.22%%
30
47
17
SQLENS
62.54 (+3.47% %)
63.66(+4.599% %)
50
66
16
Self-Reflcction
54.63 (+15.14%)
56.37(+16.888) 68%%)
209
233
24
DIN-SQI.
SQLENS Fix-ALL
58.11 (+18.62%)
59.71 22%)
257
279
22
SQLENS
59.99 (+20.50%)
61.45 (+21.96%
283
303
20
Self-Reflection SQLENS Fix-ALL
60.12 (+0.80%)
62.63(+2.59%%
12
39
27
MAC-SQL
62.43 (+2.39%)
03.84(+3.8055)
36
57
21
SQLENS
64.03 (+3.99 %)
65.23 (+5.19%)
18
Self-Reflection
67.99 (+0.08%)
69.42 (+1.51%)
12
23
11
CHESS
SQLENS Fix-ALL
68.96 (+1. .55%)
70.21(+2.30%)
19
SQLENS
69.74 (+1.83%)
70.53(+2.6226)
12
systems with Claude 3.5 Sonnet as the backbone LLM: (1) basic schema-aware prompt (Vanilla)
listed in Appendix D.I (2) DIN-SQL [33], (3) MAC-SQL [88, and (4) CHESS [44]. Semantic
correctness was determined by comparing execution results of generated queries against gold queries.
Evaluation metrics. We evaluate SQLENS on two tasks: (1) end-to-end correction, measuring execu- tion accuracy gains benchmarks; and (2) semantic error detection, framed binary classification. For correction, in addition to the execution accuracy gain A Acc., we also report Nix (incorrect - correct), Noreak (correct + incorrect), and the net improvement Nu Nus Neeak For error detection, we treat semantically incorrect queries as the positive class and report accuracy, AUC, and precision/recall/11, over 5-fold cross- cross-validaie validatio...
33 where the model is given the schema, sample values, and original SQL query, and asked debug without any explicit error information. (2) SQLENS + Fix-ALL: Provides the LLM with all error signals detected by SQLENS simultaneously, prompting it correct the query based the full set of error reports.
End-to-end correction baselines. (1) Self Self-Reflection: A standard LLM prompt
Semantic error detection baselines. (1) LLM Self-Eval (Bool):
Following Kadavath et al. 20 the LLM is asked whether the SQL query correctly answers the question (yes/no). (2) LLM Self-Eval (Prob): Based on Tian et al. [45), the LLM outputs confidence score (0-1) for query correctness; scores above 0.5 are treated correct. (3) Supervised SQLENS: A classifier trained error signal outputs as features and gold correctness labels derived by comparing the generated SQL against the ground truth. We use AutoGluon B to automatically select the best-performing model. All methods, including SQLENS, use Claude 3.5 Sonnet as the backbone LLM.
SQLENS Setup. SQLENS is implemented in Python 3.9, with the weak supervision component built using Snorkel's LabelModel 40 We configure SQLENS to perform up 3 correction iterations.
4.2
Main Results
As shown in Table I SQLENS consistently improves end-to-end accuracy across all out-of-the- box text-to-SQL systems and outperforms both this realistic sehting-where inputs include both correct and incorrect queries-SQLENt achieves the highest net improvement (Nne), fixing more queries and introducing fewer regressions. For example, with queries generated by the vanilla prompt, SQLENS corrects 49 and 20 more queries than Self-Relf Ceff-Rfleccioon and Fix-All ALL, respectively, yielding 3.47% and 2.08% higher accuracy. It improves DIN-SQL's performance by 20.5% and boosts CHESS from 67.91 to 69. 99.74%. When assuming all inputs are incorrect-as done by other methods-SQUI SQLENS achieves an average gain of 433%. Notably, L.LM self-reflection breaks the most correct queries, highlighting the need for grounded, signal-driven correction. The results Spider are in Appendix F.I
and Fix-ALL on BIRD. In
7

---

Comparison with state-of-the-art correction frameworks. To contextualize our improvements, we further compare SQLENS against recent etror-correction systems: MAGIC [3]. Self- Self-Debug [TO] and SQL.FiXAgent 7 on the BIRD dev set. As shown in Table SQLENS achieves the highest execution accuracy post-correction (65.23%) surpassing the strongest baseline (SQLFixAgent, 60.17%) by 5.06 points. Despite starting from stronger base SQL. generation method (MAC-SQL, 60.04% EX accuracy vs. 57.17% prior work), SQLENS still achieves to the largest relative improvement (+5.19%), demonstrating its robustness and scalability even when built on high-quality initial queries.
Table 2:
Comparison with recent SQL correction frameworks on the BIRD dev set. Each framework evaluated under the SQL generation method that yields its best reported results.
Correction Framework
SQL Generation Method
EX Acc. (Initial - Corrected)
MAGIC 3
DIN-SQL.
33
56.52 55.13 (+2.61)
Self-Dehug 10
DIN-SQI.
33
56.52 57.22 (+0.72)
SQI.FixAgent
CodeS-7b
57.17 + 60.17 (+3.00)
SQLENS (Ours)
MAC-SQL
60.04 + 65.23 (+5.19)
4.3
Ablation Studies
Error detection performance analysis. We evaluate SQLENS's effectivenes. in predicting semantic correctness, Results on BIRD are shown in Table with additional results on Spider provided in Appendix F22
On BIRD, SQLENS outperforms all baselines in Accuracy, Recall, and F1, demonstrating superior ability to identify erroneous SQL queries. For DIN-SQL, SQLENS achieves an F1 of 78.88, 21.45 points higher than the best LLM self-evaluation baseline (57.43). Although LLM Self-Eval. (Prob) attains the highest precision on MAC-SQL, it suffers from the lowest recall.
When aggregating signals, the supervised version of SQLENS achieves higher accuracy, AUC, and precision, benefiting from access to gold labels during training to better assess signal reliability. In contrast, the weak supervision variant yields higher recall and F1 by leveraging agreement among signals without relying on ground truth. While this leads to slightly lower precision and accuracy, it enables broader error coverage by capturing diverse aspects of SQL errors.
We further analyze the impact of SQL Auditor and the guardrail signal on SQLENS eтror correction through an ablation study on BIRD, as shown in Table
Table 4: Effectiveness of SQL Auditor and guardrail signal on BIRD.
Method
Nes
Ni.
Naerak
Impact of SQL Auditor. SQL Auditor helps reduce the number of SQL queries bro- ken during the correction process. However, this comes at the expense of fixing fewer queries overall. The decision to enable SQL Auditor depends on the desired objective: whether to prioritize conservative corrections with minimal regressions more aggressive corrections aimed at maximizing the total number of fixed queries.
SQLENS
50
66
16
Vanilla
w/o SQLAuditor
53
72
19
Guardrail+AAdiie
SQLENS
283
303
DIN-SQL
SQLAuditor
288
308
w/o Guardrail-Auditor
281
304
23
SQLENS
60
78
MAC-SQL
w/o SQLAuditor
57
78
21
w/o Guardrail-AAudioo
46
SQLENS
28
40
12
CHESS
SQLAuditor
Guardrail-Audttr
Impact of guardrail signal. To isolate the effect of the guardrail signal, we compare the results with and without it before running SQL Auditor. As shown in Table 4 the use of the guardrail signal increases the net improvements by raising the total number of fixes while reducing the total number of regressions.
4.4 Effectiveness of Individual Signals
We conduct microbenchmark on BIRD to evaluate the effectiveness of each individual signal in both emorr detection and correction.
8

---

Table 3: Effectiveness of SQLENS error detection on BIRD.
Method
Acc.
AUC
Precision
Recall
F1
L.M Self-Eval, (Bool)
60.53 (=230)
X
57.70 (1118
10.02 (#454)
16.97 (#733)
I.I.M Self-Fval. (Proh)
59.76 (=1.10)
64.23 (#:295)
64.22 (#:22.97)
2.72 (+1.89)
5.17 (43,4%)
Vanilla
Supervised SQLENS
66.58 (#2.5)
65.12" (#338)
71.83'
(499.5)
31.77 (4738)
43.32 (#6.7)
SQLENS
64.63 (2))
61.90 (22.18)
58.11 (4:7900)
48.74' (2:4.31)
52.94' (23.00)
LLM Self-Eval. (Bool)
61.52 (m1
X
86.83 (#2.115
42.99 (#2.72)
57.43 (#2.0))
DIN-SQL
LLM Self-Eval. (Prob)
49.57 (=:156)
73.01 (1003)
92.27' (1:505
17.84 (#2.09)
29.83 (4:30))
Supervised SQLENS
76.96' (22.00)
83.55' (1:238)
85.90 (22.94)
74.13 (E3.27)
79.53" (22244)
SQLENS
75.29 (=233))
81.49 (#1..4)
81.64 (#2211)
76.41" (++3.50)
78.88 (#22119)
LLM Self-Eval. (Bool)
61.50 (27)
X
65.51 (2:14
7.83 (1:2.1))
13.94 (2:30))
MAC-SQL
L.LM Self-Eval. (Proh)
61.37 (m0.8)
64.60
(#22.19)
72.697 (=:12.19)
5,16 (+:1.55)
9.61 (+2276)
Supervised SQLENS
67.09 (-3.6)
65.07
(+4.0)
66.19
(+:0883)
38.11 (+22.0)
48.16 (#4229)
SQLENS
67.43' (#4435)
64.27 (+-442)
63.27 (+5.64)
45.10' (4:300))
52.63' (+5.22)
LLM Self-Eval. (Bool)
67.98 (-1.5)
X
50.89
(+ (0.10.7)
15.71 (41100)
23.81 (#522)
CHESS
LLM Self-Eval. (Prob)
68.50 (-0.22)
64.23'
(::1665
61.87 (+:13.00)
4.90 (+:)
9.03 (+2.7)
Supervised SQLENS
72.10' (2:22))
62.95
(1322))
72.43" (1:899
23.06 (19.2))
33.96 (2800)
SQLENS
69.35
(=0000
63.34 (#2.90)
52.54 (#:2.91)
44.69' (46608)
48.17' (4:435)
X indicates that the classification is not threshold-bused. f Top result: top two are bolded.
Table 5: Error detection using individual error signal.
Table 6: Error correction via each error signal on MAC-SQL.
Signal Name
BIN-SQI.
MAC-SQI.
CHESS
DB-based Ervor Signals
Precision
Recall
.oo
Precision
Recall
Precision
Recall
N_
Signal Name
No.
Nuwa
N...
Eupty Preditate
Abnormal Result
99:6
96.07
37.01
46.33
309
75.81
100
5.66
81:25
AA
13.27
66
Dii-based Signals
Insorictt Subquery
queries detected
queiies detected
Insorest GROUP BY
68:75
5.27
Farpty
Incorrett Predicate
Insoemz Filter Sulquary
Subeptinal Similarity
15.57
130
62.34
8:76
GROUU
73.06
4.55
67.31
Prodicatc
Ulncces Salquary
92.31
1.44
Sulsuptinall Jin
Amhigulty
66.22
5.87
Table Amiiiuutty
Unnecessury Subquery
LLM--ussed Evour Siynals
Precision
Recall
No
Precision
NN
Precision
Recall
N,
Ambiguity
Ambigatty
L.MM-hood Signalss
Evidence Evidencs
82.4
12.34
103
Culumn Ambigaity
LLM Self Check
86.71
43.0
359
65.28
15.71
Evidence Violation
Question Classe Linking
87
12.34
102
LLM Self Check
Individual signal in error detection. Table 5 presents the performance of individual error signals on SQL queries generated by MAC-SQL, DIN-SQL, and CHESS on BIRD. For MAC-SQL, 13 of 14 signals exceed 60% precision. Notably, Abnormal Result achieves 100% precision in identifying 40 errors, and Suboptimal Join Tree detects the most errors, though at 62% precision. Signals like Empty Predicate and Incorrect Join Predicate perform consistently well across all systems.
In contrast, signals such as Unnecessary Subquery and Insufficient Evidence show lower precision and limited impact due to low coverage on the queries generated by MAC-SQL. and CHESS. Among L.LM-based signals, LLM Self-Check is the most reliable, consistent with results in Table 3
The lower precision of Suboptimal Join Tree with MAC-SQL and CHESS stems from redundant joins that do not affect query results, though the suggested optimal trees often align with the gold query. While such joins may not cause semantic errors, they can harm execution efficiency. Incorrect Join Predicate consistently achieves over 90% precision. Overall, database- signals provide better coverage and precision than LLM-bbsed ones.
Individual signal in error correction. We evaluate the correction effectiveness of individual error signals using SQL queries generated by MAC-SQL. For each signal, we pass all flagged queries and their corresponding error reports to the relevant Signal Fixer. Results are shown in Table Among database- and net gain of 9 corrected queries. Emply Predicule also performs well, fixing 8 and breaking only For L.L. signals, Evidence Violation yields the highest net gain, correcting queries. All signals produce a positive net correction, highlighting the usefulness of SQLENS's error signals. We
signals, Suboptimal Join Tree achieves the strongest impact with No. = 16

---

also analyzed the fix sequences for SQLs generated by DIN-SQL. Among 415 queries with multiple detected errors, Error Selector most often chose Empty Predicate (248 cases) as the first fix, followed by Evidence Violation (98 cases) and Column Ambiguity (36 cases). Empty Predicate errors are typically addressed first because they are unambiguous and directly verifiable through execution results, making them strong anchors for initiating correction.
4.5 Token Cost Analysis
We report the detailed token consumption of SQLENS on the BIRD dataset. The four main stages- Error Detector, Error Selector, Error Fixer, and SQL Auditor-consume an average of 3,141, 2.345, 2,262, and 2,215 tokens per query, respectively. A substantial portion of these tokens (approximately 1,915 per component) comes from the database description, which contains the schema and sample cell values. SQLENS performs up to three correction iterations per query, resulting in a theoretical worst-case cost of 25,459 tokens. In practice, however, most queries are corrected within a single iteration, with an average token usage around 9,963 per query.
The overhead is minimal relative to state-of-the-art text-to-SQL systems. For example, CHASE- SQL [35] consumes approximately 160,000 tokens to generate single SQL query. Manual SQL annotation by experts is even more costly, making SQLENS a practical and efficient solution for real-world use.
4.6 Analysis of Unfixed SQL Queries
Some SQL queries remain incorrect even after processing by SQLENS. To understand the causes, we conducted detailed analysis and identified three main factors:
Metric limitations. In some cases, the corrected SQL query produces the correct result but does not exactly match the provided ground-truth. For example, the fix may be semantically equivalent but different columns alternative formats, highlighting the limitation of the exact-match metric.
LLM execution errors. The semantic errors are correctly detected, and SQLENS provides appro- priate correction instructions. However, the L.M occasionally fails to follow these instructions accurately, preventing successful correction-aa inherent limitation of the model itself.
Incomplete error detection. In certain cases, the detected semantic errors do not fully capture the root cause. Consequently, even after fixing the identified issues, some errors remain in the query.
5 Conclusion
We present SQLENS, a novel framework to leverage both database- and LLM-based error signals for clause-level semantic error detection and correction in text-to-SQL. SQLENS weak supervision to aggregate noisy signals, predicts query correctness, and applies LLM-guided iterative correction. outperforms LLM self-evaluation error detection and improves execution accuracy of out-of-the- box text-to-SQL systems by up to 20% without relying on a correctness oracle.
6 Limitations
SQLENS leverages L.MS SQL error detection and correction. Hence inherits certain limitations from the underlying LLM. For example, context window constraints may restrict how much error information can passed at once, and some corrections may exceed the LLM's current reasoning capabilities. In our analysis of unfixed SQL queries (Section 4.6). we observe cases where SQLENS correctly identifies semantic errors and provides appropriate guidance, but the LLM fails to apply the correction. SQLENS also incurs computational overhead from aggregating multiple signals and performing iterative corrections, which may introduce high latency in real-time settings. However, offline, general-purpose SQL debugging tool, designed to refine and
SQLENS is intended validate generated queries asynchronously. rather than being deployed in latency-sensitive pipelines.
10

---

References
[1] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil. and Hannanch Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024. URL https:://openrevieeeee/e htt.://www.......ooooooooa/aaaaa
[2] Arian Askari, Christian Poelitz, and Xinye Tang. Magic: Generating self-correction guideline for in-context text-to-sql. 2024. URL httpst/:::// http://aaxx.ooogbb/////aaoo
12692
[3] Arian Askari, Christian Pölitz, and Xinye Tang. MAGIC: generating self-correttion guideline for in-context text-to-sql. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 March 2025, Philadelphia, PA, USA, pages
AAAI Press, 2025. doi: 10.1609/AAAI V39122.
34511. URL. htt.hhttss//o...rgg///t/ http://o........./////aa./.221..
[4] AWS. Amazon q generative sql. https://finnyull.co//yj//// 2024. Accessed: 2024- 10-17.
[5] BIRD-bench. Bird leaderboard. https://bird-Dench.gee 1 2023. Accessed: May 2025.
[6] Nik Bear Brown. Enhancing trust in llms: Algorithms for comparing and interpreting llms. arXiv preprint arXiv:2406.01944 2024.
[7] Jipeng Cen, Jiaxin Liu, Zhixu Li, and Jingjing Wang. Sqlfixagent: Towards comantic-accurall text-to-sql parsing via consistency-enhant
multi-agent collaboration. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 March 4, 2025, Philadelphia, PA, USA, pages 49-57. AAAI Press, 2025, doi: 10.1609/AAALV3311.3300 URL htttss://dii org/ /01 1609/ aaai. aaai.v3911. www.1. 31979
[8] Shijic Chen, Ziru Chen, Huan Sun, and Yu Su. Error detection for text-to-sql semantic parsing. arXiv preprint arXiv:2305. 13683, 2023.
[9] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023. httpttt////xiiii org/ ab2//304.... 05128
[10] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large lan- guage models self-debug. In The Twelfth International Conference Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrevvie:
[11] Ziru Chen, Shijie Chen, Michael White, Raymond Mooney, Ali Payani, Jayanth Srinivasa, Yu Su, and Huan Sun. Text-to-sql error correction with language models of code, 2023. URL https://arxiv org/ abs/ab/s. 13073
[12] Databricks. Databricks assistant.
wwweaayuul...oo//eaee comm/cdvabbxx 2023. Accessed: 2024-
10-17.
[13] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon- Tabular: Robust and Accurate AutoML for Structured Data, March 2020. http://///arxi www. www.//00...om 06505 arXiv:2003.06505 [es, stat].
[14] Avrilia Floratou, Fotis Psallidas, Fuheng Zhao, Shaleen Deep, Gunther Hagleither, Wangda Tan, Joyce Cahoon, Rana Alotaibi, Jordan Henkel, Abhik Singla, Alex Van Grootel, Brandon Chow, Kai Deng, Katherine Lin, Marcos Campos, K. Venkatesh Emani, Vivek Pandit, Victor Shnayder, Wenjing Wang, and Carlo Curino. NL2SQL is a solved problem... not! In 14th Conference Innovative Data Systems Research, CIDR 2024, Chaminade, HI, USA, January 14-17, 2024, www www.cidrdh.org. 2024. URL httpt:t///// //www. cidrdb. org/ cidr2024/ papers/ 074-floratou. pdf

---

[15] Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Re. Fast and three-rious: Speeding up weak supervision with triplet methods. Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning. volume 1 19 of Proceedings of Machine Learning Research, pages 3280-3291. PMIR, 13-18 Jul 2020. httttttss:///oeceeeee
mlr aatesss/119/ff22/2 hhmmmmm0..mmmmm.mmmmmmmmmmmmmm
[16] Dawei Gao, Haibin Wang. Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation, 2023. URL
https://arxiv.. org/ abs/ 15363
[17] Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pages 6577-6595, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.1 20055///122224.aaalloooggg https://aclanthoole
URL
org/ wrg//024..222 www.-aallllla -onac 366
(18) Zezhou Huang, Pavan Kalyan Damalapati, and Eugene Wu. Data ambiguity strikes back: How documentation improves gpt's text-to-sql, 2023. URL httpp::/aaxxv..rra...p.. wrtp./////abbb//11
18742
[19] Pallavi Jayawant and Kerry Glavin. Minimum spanning trees. Involve, Journal of Mathematics, 2(4):439-45)) 2009.
[20] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-oodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language Models (Mostly) Know What They Know, November 2022. URL http://arxiv. www.///g///////oeo//ooo/eeeeeeet 05222 DXXiv2207.005221 [cs].
[21] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint 22w.:2220000026666 2022.
[22] Joseph Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical society, 7(1):48-50) 1956.
[23] Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. MCS-SQL: Leveraging Multiple Prompts and Multiple- Choice Selection For Text-to-SQL Generation, May 2024. URL https://arxiv/ org/ abs /wwp////........... 07467v1
[24] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proc. ACM Manag. Data, 2(3):127, 2024. doi: 10.1145/3654930. URL
https://doi: org/ 1rrg/10.r 1145/3654930
[25] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Rongyu Cao. Ruiying Geng, Nan Huo. Xuanhe Zhou, Chenhao Ma, Guoliang Li. Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM Already Serve as A Database Interface? Blg Bench for Large- Scale Database Grounded Text-to-SLLs November 2023. httl:tttp///aaat
org/ abs/ 2305. 03111 mtXiv:230500311 [cs].
[26] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost the middle: How language models use long contexts, 2023. URL
http://arxiv.or www./oorg/aaao. abs/ 00172
[27] Peixian Ma, Xialie Zhuang, Chengjin Xu, Xuhui Jiang, Ran Chen, and Jian Guo. Sql-rl: Training natural language to sqi reasoning model by reinforcement learning, 2025. URL https: //ta:i//aviiv.aa org/ abs//504....///////// 08600
12

---

[28] Xueguang Ma, Xinyu Zhang, Ronak Pradeep. and Jimmy Lin. Zero-shot listwise document reranking with large language model, 2023. URL https::/arxiv http:////.a.i....raaaa////////// htt//ba//3300 02156
[29] Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-sql in the age of well-reasoned language models, 2024. URL
https://arxiv wwwgg/bb//440..000 07702
[30] Raghunath Othayoth Nambiar and Meikel Poess. The making tpe-ds. In VLDB, volume pages 1049-1058, 2006.
[31] Arjun Panickssery, Samuel Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations, 2024. URL httttt://a/xiiii www. htt///bs//200 2404. 13076
[32] Liana Patel, Siddharth Jha, Carlos Guestrin, and Matei Zaharia. Lotus: Enabling se- mantic queries with llms over tables of unstructured and structured data. arXiv preprint arXiv:2407.11418, 2024.
[33] Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: Decomposed In-Centext Learning of Text-to-SQL with Self-Correction, April 2023. URL https://arxiv: http://axxvv.... org/ 2b34
11015v3
[34] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Ycounoh Chung, Shayan Talaei, Gau- rav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O. Arik. Chase-sql: Multi-path reasoning and preference optimized candidate selection text-to-sql, 2024. URL
https://arxiv.. org/ abss//1100 01943
[35] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gau- rav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik. Chase-sql: Multi-path reasoning and preference optimized candidate selection text-to-sql. arXiv preprint arXiv:2410.01943, 2024.
[36] Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, and Sercan "O. Arik. Reasoning-squ Reinforcement learning with sqi tailored partial rewards for reasoning-enhancee text-to-sql, 2025. URL https://arxiv
Brg/abs/2503 23157
[37] Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-source large language models, 2023. URL https://arxiv
org/abs/22 2209 15088
[38] Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11 (13):269-282 November 2017. ISSN 2150-8097. doi: 10. 10.14778/3157794
3157797. URL httph:tpp... :eattps:/dii acm. http:///..aa.oorr/////o.........
[39] Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmenta- tion reduces hallucination in conversation. urXiv preprint arXiv:2100400556,,20 2021.
[40] Snorkel Team. Labelmodel 0.9.3 documentttnnnnnnnnnootoo io/en en ee/ee//0..0 .0.9..3 httpl//ee....ciiiieee//////////a
html 2021. Accessed: 2025-10-22.
[41] Snowflake. Snowflake copilot. htttps:/ttinuurl.comm/ 10-17.
2024. Accessed: 2024-
[42] Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. Calibration and correctness of language models for code, 2024. URI wwwpaawwwww..gggaa//.....00..
[43] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. chatgpt good at search? investigating large language models as re-ranking agents, 2023. URL https: /arxiv.iv.. org/ abs/ abs/2304. 09542
13

---

[44] Shayan Talaci, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. CHESS: Contextual Harnessing for Efficient SQL Synthesis, June 2024. URL h
//arxiv. ott//ogg////aaao/oooeee 16755 arXiv:2405.16755 [cs].
[45] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback, October 2023.
URL hwtp:httpp//iii..oot org/ abs /200. 14975 arXiv:2305. 14975 [es].
[46] Alexander van Renen, Dominik Horn, Pascal Pfeil, Kapil Eknath Vaidya, Wen- jian Dong, Murali Narayanaswamy, Zhengchun Liu, Gaurav Saxena, Andreas Kipf. and Tim Kraska. Why tpe is not enough: An analysis of the amazon redshift fleet. In VLDB 2024, 2024. URL hittps:/wwwx amazon pcwencc/ppblicaiiioonn. hoy-tpp-is--nt--eoou-- an annpough----aallysss-----
[47] Bing Wang, Yan Gao, Zhoujun Li, and Jian- Guang Lou. Know what I don't 't know: Handling ambiguous and unknown questions for text-to-SQL. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 5701-5714, 14, Toronto, Canada, July 2023, Association for Computational Linguis- aadinngs-acl.352 URL atttps:/acclanthol.gg .og.olloooooooo.oooooo.oo 2023
tics. doi:
aandings-acl.333
[48] Bing Wang, Changyu Ren, Jian Yang. Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. MAC-SQL: A Multi-Agent Collabora- tive Framework for Text-to-SQL,, June 2024. URL httttt::p////ii org/ www.obag////eeeeoeeeo....taaaeee 2312. 11242 arXiv:2312.11242 [cs].
[49] Lei Wang. Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
[50] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Cfain-of-thoouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248331 2022.
[51] Zhuofeng Wu, He Bai, Aonan Zhang, Jiatao Gu, VG Vinod Vydiswaran, Navdeep Jaitly, and Yizhe Zhang. Divide-or-conquer! which part should you distill your Ilm?, 2024. URL. http:///aaxxvv..g/aaa/s////
15000
[52] Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. Pride and prejudice: LLM amplifies self-bias in self-refinement, In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 15474-15492 Bangkok. Thailand, 1/200/1.111212.2ooggggg
August 2024. Association for Computational Linguistics. doi: URL htttps::/aclaatoolllgg. org/ 2024. caal-llongg 826
[53] Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. Model-based interactive semantic parsing: A unified formulation and a text-to-sql case study. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP'19) 2019.
[54] Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. An imitation game for learning semantic parsers from user interaction. arXiv preprint xxXi:2205.00689, 2020.
[55] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL. Task. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun uinic Tsujii, editors, Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 3911 1911-399 1-3911, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 18.18653/v 10.18653/1//D18-4442 1/D18-1455.
ort/:/o-g/1/111 hwwechhooogy.oro///e B0///D18-111 1425
[56] Jichuan Zeng, Xi Victoria Lin, Caiming Xiong, Richard Socher, Michael R Lyu, Irwin King, and Steven CH Hoi. Photon: A robust cross-domain text-to-sql system. arXiv preprint 10X2v 2020.
14

---

NeurIPS Paper Checklist
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we claim that we contribute SQLENS, an end-to-end framework for error detection and correction in Text-to-SOL. We then explain each component in SQLENS, and show that SQLENS outperforms the best LLM- self-evaluation method by 25.78% in Fl for enror detection, and improves end-to-end execution accuracy by up to 20%. Therefore, the main claims accurately reflect the paper's is contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to seeech-tt-texxx system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofss can either appear in the main paper the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all necessary details to ensure full reproducibility of our experiments. Specifically, (1) Section details the methodology, including the architecture and the design of each component within SQLENS, and (2) we provide the full description of all error signals in Appendix and C
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases. in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-soures models, it may be that access to the model is limited in some way (c.g., to registered users). but it should be possible for other researchers have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
16

---

Answer:
[No]
Justification:
The code and data in this research are subject to company policy and cannot be released publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines pattps://niips.cc/ubbe
for more details.
While encourage the release of code and data, understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurIPS code and data submission guidelines (https: //ip//iip.s./.p ccciips.ccpuublli cc/public/
reproduce
guides/CodeSuueee wwwabmmmssioPPooicyy onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide seripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all necessary details understand our results. Specifically, in Section 4.1, we detailed our experimental setup, including datasets and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct five-fold cross-validation for error detection and report the standard deviation for each method in Table 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
17

---

It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If eπrоr bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: We specify the exact LLM used in our experiments, and its pricing characteristics are publicly documented. In Appendix F33 we report the average number of L.LM calls and the latency for fixing SQL query.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppw//eeere......c//////aa
Answer: [Yes]
Justification:
The authors have read and conformed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive societal impacts in the introduction, highlighting how SQLENS can assist text-to-SQL users in debugging A.L.-generatee SQL queries and help them build trust in text-to-SQL solutions.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.,
disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put place for responsible release of data or models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All third-party assets employed in this work are appropriately cited, and have verified compliance with their respective licenses and usage terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwperswithcode.eeoopo..ppppppppe thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
19

---

Answer: [NA]
The code and data in this research are subject to company policy and cannot be released publicly.
Justification:
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve L.MSs as any important, original, non-standard components.
20

---

for what Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:///eaarrisscc//bbbeeeeee//s should or should not be described.
21

---

A SQLENS Use Case
Figure 6 shows a text-to- Loo---Q solution that generates SQL query for a given NL question. Although the generated query has no syntax errors, the user executes the query and finds that it produces empty results, indicating the query incorrect. Under the status quo in text-to-SQL assistants M [T], users must invest significant time manually debugging semantic errors in queries. Our goal is to provide users with a Debug SQL functionality, which automatically performs deep semantic analysis of the query, identifying eπrors such as incorrect join predicates in the FROM clause or incorrect usage of the provided external knowledge in the WHERE clause. This is followed by an automated correction process that generates an updated SQL query for the user to validate. Using Debug SQL, users' manual efforts in query troubleshooting are significantly reduced, consequently building trust in the text-to-SQI. assistant.
B DB-based Error Signals
The following signals detect semantic misalign- ment, which occurs when the L.M misinterprets the question or misuses the data.
many opened accounts women? (Extemal Inowledge- Fumale refers to gender F?
Jesonik branch to N2=Jesemkk
Suboptimal join tree signal targets the FROM and JOIN clauses, checking if the SQL query uses the optimal join tree to connect the re- quired tables for answering the NL question. While query optimizers can eliminate redundant tables that do not affect the query results, LLMs may generate unnecessarily complex join trees, potentially leading to incorrect outcomes. Let Dryy S DD denote the tables needed to answer the question. The optimal join tree the min- imum Steiner tree 7" [19 spanning Deq. If the query includes extra tables beyond Dreq. the Suboptimal Join Tree signal is flagged. For ex- ample. Figure 7 (right) shows an optimal join tree connecting tables A and C. SQLENS iden- tifies columms not involved in JOIN clauses as Dreg and uses Kruskal's algorithm 22 to find the minimum Steiner Tree the pre-built join graph. the query involves more tables than necessary, the signal is raised. False positives may occur when multiple valid join trees exist.
ESuScc COUNT (DLEVIECC elisnt. client, id)
2lien:
optimal Join
1i00e 20TH account ON
Ellent, el Lent id womonnt account,
district
Incortect Join Predicate
account diniircct_5 diotvict dootviet.diourfe. diaurtot__
Kross
Evidence Viclation
A00 Bistrist.
"jaanik'
Delua SQL
natebook)
Dabug query
Figure 6: Text-to-SQL. debugging use case.
A
B
Suboptimal Join Tree
Optimal Join Tree (Minimum Steiner Tree)
Figure 7: Steiner Trees spanning and C
To implement this signal, SQLENS first identifies the columns in SQL query that are not involved in the JOIN clauses. Based on these columns, SQLENS searches for the minimum Steiner Tree on the join graph built offline to connect the relevant tables. If the SQL query involves more tables in the join than necessary, compared to the minimum Steiner Tree, SQLENS raises Suboptimal Join Tree flag. This signal can produce false positives when a suboptimal join strategy does not impact the correctness of the SQL query. For instance, if a query asks for the total sales from a specific store, directly select the sales column from the store_stor sales table. However, if the query unnecessarily joins the store_salee table with a store _eccaiin table, even though the store location table is not needed get the correct sales data, the result would still be correct but the join is suboptimal.
Incorrect join predicate signal checks whether SQL query uses an invalid join predicate. For example, in Figure? 2 the predicted SQL incorrectly joins client id with account id, which does not
22

---

exist in the corresponding schema graph. To implement this signal, SQLENS first extracts all the join predicates from the SQL query. It then identifies two types of correct join predicates: (1) using a primary key-foreign key (PK/FK) join explicitly defined in the database schema, and (2) derived from PK/FK joins. Specifically, two columns C, and C3 both reference the same primary key C& (i.e., they refer to the same entity), C₁ and C₁ can be used in join predicate. This signal may generate false positives when the PK/FK relationships are not fully documented in the database.
Empty predicate signal detects if there is a predicate within a SQL query that yields an empty result. This signal is useful to detect semantic misalignment errors, including wrong column selection, wrong value usage or wrong comparison operator. SQLENS extracts all comparisons between a column and literal from a given SQL, executes each of them individually and records the output size. If there is a predicate that yields empty rows, this signal is flagged. This signal may lead to false positives when an empty predicate is intentional.
Abnormal result signal detects whether a SQL outputs an abnormal result that does not provide much information. SQLENS executes the SQL and records its output. The output is considered abnormal if (1) it is empty, (2) it contains a column full of zeros, or (3) it contains a column full of NULLS. This signal extends beyond the empty predicate to evaluate the entire SQL output. In addition to detecting empty predicates, it can identify empty intermediate execution results, making it effective for catching semantic misalignment in the intermediate steps of SQL query. This signal may incorrectly flag SQL query when the abnormal result is intentional, though this scenario is rare in practice.
Incorrect filter in subquery signal detects the problematic filtering in a subquery. Filters in a subquery often follow the pattern columm (SELECT.... When the subquery returns multiple rows, the filter condition becomes ambiguous (e.g., IN or '='), potentially leading to errors. SQLENS uses regular expressions to match this pattern and executes the extracted subquery separately. it returns more than one row, the signal is flagged. Additionally, SQLite is lenient with SQL semantics, allowing column (SELECT...) to match only the first value returned by the subquery. While the query might still be correct if the first value happens to be the desired one, relying on this behavior is generally considered poor practice in SQL writing.
Incorrect GROUP BY signal detects any standalone GROUP BY clause without accompanying aggregate functions such as MAX, COUNT. A misused GROUP By clause can change the SQL semantics and lead to an incorrect result. In SQLite, a standalone GROUP BY behaves the same as the DISTINCT operator. While the query may still be correct when using GROUP BY substitute for DISTINCT, this approach is generally considered poor practice.
Unnecessary subquery signal indicates if there is an excessive use of subqueries in SQL query, which leads to inefficiencies, increased complexity, and a higher likelihood of errors. This signal counts the number of subqueries in a SQL query and flags it problematic if the count exceeds specified threshold. In our evaluation, this threshold is set to 3. False positives may arise when the subqueries are necessary for performance optimization or specific logic, even though they exceed the threshold.
The following signals are designed to capture the inherent ambiguity within the data.
Value ambiguity signal detects incorrect column selections when a value used in an NL question appears in multiple columns. For example, "New York" can appear in both "state" and "city" columns. To identify ambiguous value, this signal extracts all values used in the SQL and finds columns that contain a used value via an inverted index built offline. If there is an alternative column that is closer to the question semantically, the signal flags the corresponding value as ambiguous. It is possible that the originally selected column in the SQL is correct, as the semantic distance may not always accurately determine which column containing the value is the best candidate to answer the question.
Table similarity signal detects potential errors in table selection by identifying alternative tables with similar structures. It extracts all columns used in the SQL, groups columns by their tables, and searches for other tables that contain the same groups of columns. If such a table is found, it suggests that the alternative table could have been used, indicating a potential mistake in the original chosen table. False positives can occur when the chosen table is actually correct, but another table with a similar structure exists, leading the system to incorrectly flag a possible error.
23

---

Table 7: DB-based Error Signals.
Signal Name
Incoerect SQL Query
Correct SQU Query
SQL Clause
SELECT cid FRDM cards
SELECT id FROM cadss
WHERE woanddiiigoooooooiiiiii
WHERE candKindookssld
Abnormal
Resilt
AND c.onnKingkkki
AND c.lasFoil 1 AND c.s.ullAA -0
NOT NULL
contKingsoold NULL:
WHERE
AND cisoversiiedd = AND c cissromo O; Ougee Stoe:
Outpor Stze: 25067
Predicate
Emrpty
JOIN hond hond 18 bbond ki
WHERE Ihhond 20.5000,2_55
SELECT Lanom, catoom, FROM counected
Tatoes, id connected WHERE T.bond id 2T5000_2_5;
WHERE
Filter
SELECT Name FROM
ISEEECT FROM users
WHERE Usald
SELECT FROM WHERE Usestd
(SELECT FROM
SUBQUERY
Salquery
WHERE DisplayName- 'Pierne')
WHERE Display Name 'Piene ):
SELECT s.City Ennollmenn (K:
SELECT Ciy FROM fpmm
GROUP
Inconect
FRDM JOIN schools
GROUP BY City. F'mmllmment (K-12)
CCDSCude CCDSCCode
m..USSCl 12CCDSCode
INNER JOIN
GROUP BY T2.City
AS
ORDER BY Faallllnlllmnn (K-12)) ASC LIMIT 5.
ORDER BY SUMTTI 'Fareliment (K-12)) ASC LIMIT
SELECT (SELECT COUNT(DISTINCT cliest.elion_
FRDM eliest INNER JOIN
SELECT COUNTCTI diient
Incorrect Juin
Predictte
INNER JOIN district
WHERE distric a2 'Jesenik'
ON sccoums district id districtdisrioo..
client socount account
FROM clieni AS TI INNER JOIN district
CN distiict id d2.diurict id
WHERE T1.gender "F" AND 12.12
'Jeennkk:
JOIN
AND cliens
'P') num_/tmaak_cliess
Subopiimal
JOIN disp di ON c.llient idd diclieet id
SELECT daa FROM cliest
SELECT TI a3 FROM district TI
INNER JOIN cliNe T2
FROM JOIN
Join Tree
JOIN rocount ON di accoum naccont id
JOIN distnet a.distiet d.disteeJd
ON disrie id 12.distric_jl
WHERE 12 id
c.clsent.66 2541 LIMIT
SELECT AVG(r.point)) AS avorecore score
SELECT AVG:T2 FROM drivers AS TI
FROM meiits JOIN drivens dl riliverld dddrierrd
INNER JOIN dhiverSandings AS T2
Table
JOIN ON craceld rammcced
T1 dnvverdd
SELECT
Similarity
foresam 'Lewis'
duuuraame 'Hlamitton'
INNERJOIN races ON
AND musseld SELECT razeld FROM moes
WHERE T1.orrsame- AND 13mmme TTurksh Grand Prx
AND T'.samame- "Hamilon"
WHERE name LIKE *6 16Tukkissh Grand Prick" k.
Unnecessry
Subquery
(SELECT cartie FROM cards WHERE c.uuid
(SELECT FROM milingsi) card name.
ISELECT suid FROM rulingsi) artist.
(SELECT c.sspromo FRCM carls WHHEE camid
SELECT (SELECT conme FROM cards WHERE cuuld
(SELECT uuid FROM culings)) AS & supromo;
FROM
SELECT 11
GROUP BY artist ORDER BY
ON auid undd
COUNTI T uuid) DESC LIMIT
Vl TTLisPromo
Juling zulings iuPoom
SUBQUERY
SELECT "artie"
SELECT
Vilue
JOIN Horeiaa_data" (ON uuid "/:///
INNER JOIN Iorcign_
SELECT WHERE
Amhiguity
WHERE "wstemurk" - "'hyycaiinn"
ON TI auid - T2.mid
AND c''ttaa IS NOT NULL GROUP BY c/uutar"";
WHERE 12languago- 'Phyretian"
In Table 7 present examples of incorrect and correct SQL query pairs based NL questions from woold applications, highlighting the specific SQL clauses each signal targets.
C LLM-based Error Signals
Evidence violation signal identifies cases where the generated SQL query contradicts the evidence provided in the question or external knowledge. For example, if the question specifies retrieving rows only about active employees, but the SQL query does not include a condition to filter out inactive employees, this would trigger an evidence violation. The prompt used by SQLENS can be found in Appendix D.2
Insufficient evidence signal assesses whether the available evidence is adequate confirm that the SQL query correctly answers the user's question. For example, if the question lacks a clear explanation of domain-specific concept, the LLM is prone hallucination. This signal essentially verifies that the L.M has enough information and context to provide a correct response. The prompt shown in Appendix D.3
Incorrect question clause linking signal evaluates the LLM's confidence in the generated SQL clauses. This signal first prompts the LLM to map the concepts, entities, and expressions in the user question to the corresponding clauses in the SQL query. For each identified link, the LLM is then asked to indicate its confidence in the generated clause by responding with simple yes signal is flagged when there is at least one clause with low confidence. The prompt can be found in Appendix D.4
This
Column ambiguity signal identifies whether there are columns in the database that are very similar to those used in the SQL query and could also be used to answer the user's question. This signal
24

---

Table 8: LLM-based Error Signals.
SQL Query
Query
Clause
Column Amhiguity
ORDER BY Mug DESC LIMIT OFFSET
SELECT School, StreetAhr FROM
JOIN schooks ON we edk CDSCalle
FROM sassores AB T1
INNER schools AS T2 ON CDSCude
ORDER BY MlAhsSocMMahh DESC LIMIT 5,
SELECT MailSneet, 12.School
SELECT
Evidenee: set cards with "Angel Merey" in it refers to name 'nngel of Mency'
SELECT COUNT(DISTINCT translaioon
Evatenee Violaton
FROM set crassaaionn WHERE setCode IN
(SELECT saCrdd FRDM cards
Any classe identified by LLM
SELECT COUNTC) FROM transsatioon WHERE snCode
WHERE
insndatioo NOT NULL:;
Q: How mony stoms with indine and silfur type
SELECT COUNTIDIS TINGT GASE WHEN eloment
elements there is single hoad moleculas?
THEN TI id SE NULL END) AS indine
Insufficient Evidence
Evidence: with iodine element refor element sultur element
element 's": type
THEN ELSE NULL END)
COUNT(DISTINCT CASE WHEN element =
idenitiee
classe
single type
JOIN counecsed
12.atom.ii hond WHERE honl type oype:
It nor clear what "single bond moleculer" refers
ON 72.bond 38
180 top players' whose
above
SELECT playeeer_naaeee FROM Player AS
accurary.
INNER JOIN
AS
Inceerect Question
Chass Linking
Mayer Atmibutes po pplyyer api pa paplayer api
SELECT p.paayr__ nume Player
pl api api_id (2,p01yycc..ppp.1_
WHERE il hright > GROUP
identified by LLM
WHERE pheight > 180 ORDER BY palheading Locurcy
socurccy
ORDER BY CAST/SUMIC heading accuracy) AS REAL)
DESC LIMIT 10;
COUNTII player sia. apt.pppppp (dd) DESC LIMIT 10;
SELECT p.Paragrahh
Pangraphs
paragagsh
Paragraphs
Self Check
Documents p.Docusent D.Documet__D WHERE D.ocumeer_Namm 'Wecooe to NY:
documeett id documem WHERE 12 documenn_nunee "Customser reviews":
idaniitied by LLM
suggests that a SQL is prone to wrong column selection. The prompt for this signal is shared in Appendix D.5
The above signals provide specific error causes for the LLM to detect. Given that studies have shown LLMs possess the ability validate their answers 20 (5), also incorporate signal that prompts the LLM to provide an overall assessment of its own output.
LLM self-check asks an LLM to determine whether the proposed SQL correctly answers the user question considering database and any available external knowledge. The prompt can found in Appendix D.6 and D.7
The signal examples are listed in Table 8 False positives can occur for the LLM-based signals when the LLM misinterprets the question, has limited understanding of the specified knowledge, experiences hallucination, or exhibits bias when evaluating its own output.
D Prompts
D.1 Prompt for Vanilla Text-to-SQL
Role: You are an expert SQL database administrator responsible for crafting precise SQL queries to address user questions.
Context: are provided with the following information:
1. A SQLite database schema
2. A user question
3. Relevant evidence pertaining to the user's question
Database Schema:
Consists of table descriptions
Each table contains multiple column descriptions
Frequent values for each column provided
Your Task:
1. Carefully amalyze the user question, evidence and the database schema.
2. Write SQL query that correctly answers the user question
Format your SQL query using following markdown:
sssll
YOUR SQL QUERY HERE
25

---

[Question] {question}
[Evidence] (evidence)
[Database Info]
{db_desc_ate)
[Answer]
D.2 Prompt for Evidence Violation
You are provided with user question and proposed SQL query that solves the question. Your task is to determine whether the proposed SQL query reflects all evidence specified in the question.
Here is typical example:
Example
Question] How many users are awarded with more than badges? more than 5 badges refers to Count (Name) 5; user refers to UserId
[SQL query] SELECT UserId FROM SELECT UserId, COUNT (DISTINCCT Name) AS FROM + badges GROUP BY UserId ) WHERE T.num 6;
[Answer]
"jsson
{{
vvioolate_evidennee true
"explanation: "This SQL query violates the evidence in the question because it counts the number users with than badges, than badges. Additionally, uses (DUSTIDI Name) instead COUNT(Name) as specified in the question.
}}
Question Solved, Make sure you generate valid json response.
Here is new example, please start answering:
[Question] fquestion). {evidence}
[SQL query] {sql_query)
[Answer]
D.3 Prompt for Insufficient Evidence
You provided with sqlite database schema, question, and proposed SQL query that solves the user question. Your to determine whether you have sufficient evidence to determine whether the SQL answers the user question correctly.
+
Output JSON object in the folloving format. Make sure you generate valid json response.
+
[Answer] json
{{
26

---

"mmsufficientt_vvdeeeee true/false
"explanation:: "why the evidence is sufficient"
}}
Question Solved.
Here example, please start anawering:
[Question] {question). (evidence)
{db_ (db_desc_str)
[SQL query]
quql_)uery>
[Answer]
D.4 Prompt for Question-Clause Linking
You are given SQLite database schema, user question, and a proposed SQL query intended to address the user's question. Please follow these steps:
1. Link the concepts, entities,
expressions in the user question to the + corresponding clauses in the SQL query.
2. For each link you have identified, indicate whether you confident in the generated clause by answering "yes" "no."
Output a JSON object in the following format. Make sure you generate a valid jaon 4 response.
[Answer]
json
{(
"<(entity in the question, the corresponding SQL clause)>"; "Cyes/no>*
}}
Please start answering the following question: [Question] {question). {evidence).
{db_desc_str)
[SQL query] (sql_query)
[Answer]
D.5 Prompt for Column Ambiguity
As an experienced and professional database administrator, you are provided with SQLite database schema, user question, and proposed SQL query intended to solve the user question. The database schema consists of table descriptions, + each containing multiple column descriptions.
Your task is to determine whether there are columns in the database that are very
. similar to the ones used in the SQL query and could also be used to answer the user's s question.
Here typical example:
Example
27

---

[Question] Which state special schools have the highest number of enrollees from grades 1 through 12? State Special Schools refers to DOC 31; Grades -> through means K-12
[DB_ID] califoomiaa_scool.. [Database Schema]
Table: frpm
(CDSCode, CDSCode.), (Enrollment (K-12), Enrollment (K-12) .),
]
Table: satscores
(cds, cds. Column Description: California Department Schools), (sname, school name. Value examples: [None, 'Kiddle College High', JJhnn F. Kennedy High', 'Independence High', 'Foothill High', 'Washington High', 'Redwood High'l.), (enrol112, enrollment (1st-12nd grade)
]
Table: schools
(CDSCode, CDSCode.),
(DOC, District Ownership Code. Value examples: ('54', '52', '00', '56', '98', '02'].),
]
[SQL query] <SQL> SELECT T2. FROM schools AS T1 INNER JOIN satscores AS T2 ON T1. CDSCode T2. cds WHERE T1.DOC '31' AND T2. enrol111 IS NOT NULL ORDER BY
+
enrol1111 DESC LIMIT 1; </SQL>
[Answer]
jjoon
{{
galternative_columi true
"explanation"; 'Arpm.Enrollmew" (K-12) can also be used determine the number 4 of enrollees from grades through 12. This column is very similar to 4 satacores. enroll12 used in the proposed SQL."
}}
Question Solved, Make sure you generate a valid json response.
Please start answering the following question.
[Question] {question). (evidence)
(db_dess (db_desc_str)
[SQL query] {aql_query)
[Answer]
D.6 Prompt for LLM Self-Check (True/False)
You provided with SQLite database schema,
question, and proposed determine + whether the proposed SQL correctly answers the user question. Your answer
SQL query intended to solve user question. Your task
+ should be in the form of JSON object with two keys: "correct" and "explanation". Provide an explanation only if the SQL is incorrect.
You need to generate the ansver the following format:
[Answer] jsson
28

---

"
"correct"; false,
"explanation": "your explanation why the SQL is incorrect" }}
Make you generate valid json response.
Please start answering the following question: [Question]
(question). (evidence)
[Database Info]
{db_ [ib_desc_strr
[SQL query] {sql_query)
[Answer]
D.7 Prompt for LLM Self-Check (Probability)
You are provided with user question, SQLite database schema and proposed SQL query intended solve the question.
Your task is to evaluate the proposed SQL query and provide the probability that it correctly answers the user question.
Provide this probability as decimal number between
You need to generate the answer in the following format:
[Answer]:
jjson
{
"probability"; <the probability between 0.0 and 1.0 that the SQL correctly answers the question>
}}
Make sure you generate a valid jaon response.
Please start answering the following question: [Question]
(question). (evidence)
(Database Info]
{db_ {db_desc_str)
[SQL query] (sql_query)
[Answer]
D.8 Prompt for SQL Query Correction
Role: You are an experienced and professional database administrator taaked with
analyzing and correcting SQL queries that are potentially wrong.
Context: are provided with the following information:
1. A SQLite database schema
2. A user question
3. A proposed SQL query intended
the question
29

---

4. An report for the proposed SQL query. The error report suggests potential in the SQL.
Database Schema:
Consists of table descriptions
Each table contains multiple column descriptions
Frequent values for each column provided
Your Task:
1. Analyze the error report
2. Determine if the SQL query needs to be fixed. You choose not modify the
4 SQL if it correct.
3. the proposed SQL is incorrect, generate a correct SQL query to answer the
4 user question
Instructions:
1. Review the provided information carefully
2. Use SQL format in code blocks for any SQL queries
3. Explain your reasoning and any changes made to the query
4. Avoid using overly complex queries. For example, ... EXISTS (SELECT FROM
+ table WHERE condition) can substituted with JOIN.
[Question]
{question}
[Evidence]
(evidence)
[Database Info] {db_desc)
[old SQL] ssll {old_sql)
[Error Report] ferror_ree repoort
Now, please analyze the error report, decide whether SQL needs to fixed and - generate a correct SQL to answer the user question if you think the proposed SQL is indeed wrong.
[Correct SQL]
E SQLENS Error Correction Pseudo Code
Algorithm presents the pseudocode for the error correction. In essence, SQLENS's enror correction a greedy, sequential algorithm. The original SQL query is first processed by SQLENS'S Error Detector, which evaluates given set of signals on the input query and generates a list of error reports for the detected signals (Line 1). practice, users can select subset of signals to fix queries based their precision in detecting errors development dataset. These reports then sent to the Error Selector, which identifies the most critical error address for the current iteration (Line 5).
The Error Selector component takes as inputs the original SQL query, database schema, and list of error reports ranked by their confidence, and then determines the most critical error to address first using an LLM. Essentially, the LLM adjusts the ranking of these error reports based on its understanding of their relevance to the original SQL 43 identified, the next step is to correct the query using the error information provided by that signal, handled by the Error Fixer (Line 6).
28 37 Once the most critical error is
30

---

Algorithm 1: SQLENS Error Correction
Input: ctx: Correction context, including the database, external knowledge, and the natural language question; q: Original SQL query; S: Set of emor signals; mar iler; Maximum number of
iterations; Eg: Guardrail signal
Output: Corrected SQL query. q
ErrorDeetectrr
i
q'
+-
while
[&] Oori < maz maa_iter do
select the current most critical error
Srlectelector
q + Error FerorFiierre Fiter(etx, e,4')
remove error signal that has been fixed before
S ()te)
E + Error GrrorDeeeccor(dts..
iiit1
10 end
/* fix guardrail signal if detected
Error Detector (cta, (cg])] then
12
q' + ErrroFiser (cta, gg.9))
13 end
SQLAuditor (cta, ,'
15 return q
Table 9: Statistics of SQL queries generated by sat-of-the-box text-to-SQU systems.
Approach
Dataset
Acc. 1'
Acc. 2
SQLs
Incorrect
Semantic Err.
Vanilla
BIRD
55.41
59.07
1534
684
589
DINSQL
BIRD
35.53
39.49
1534
989
MACSQL
BIRD
58.87
60.04
1534
631
601
CHESS
BIRD
67.60
67.91
1534
497
490
Vanilla
Spider
79.11
79.65
1034
216
209
DINSQL
Spider
76.31
77.66
1034
MACSQL
Spider
78.92
79.69
1034
218
208
Accuracy over all queries 2 *Accuracy over queries without syntax errors
The Error Fixer component takes the original SQL query, and specific error report along with the correction context (specified in Algorithm I as input. It analyzes the error, determines how to fix the query, and generates new query. avoid syntax errors introduced the revision process, we include Syntax Checker in the Error Fixer, which uses SQL parser to validate the syntax. If there is any syntax error, the Error Fixer uses the parser's errоr message to iteratively revise the SQL query until it is valid. Note that SQLENS avoids fixing the same error repeatedly by removing the error signal from the signal collection after it has been used for correction (Line 7). Once the current error is fixed, we Error Detector the revised SQL query. If it contains no further errors, corrected SQL query returned. Otherwise, correction process iterates until errors resolved or the maximum number of iterations is reached (Lines 4-10).
The above process generates revised SQL query, but may still contain some errors that need to be fixed. To ensure the most critical error is corrected, we introduce final check to identify if any further adjustments are needed. The signal with the highest precision detecting incorrect SQL queries can be configured as a "guardrail signal" to catch and fix remaining errors (Lines 11-13). As mentioned earlier, the revision process may break an originally correct SQL query. To address this issue, the original SQL and the final revised SQL are passed to LLM-based SQL Auditor for the final evaluation (Line 14).
F Additional Experimental Evaluation
Table 9 presents the text-to-SQL performance of the base SQL generators BIRD and Spider.
31

---

F.1 End-to-End Evaluation on Spider
Table 10 shows the end-to-ena evaluation results on Spider benchmark. In terms net improvement, SQLENS outperforms
by 10, 20, and 9 SQL queries on MAC-SQL, DIN-SQL, and Vanilla prompt, respectively. SQLENS consistently fixes more SQL queries compared the baselines (Nix). Its performance is similar to its variant, Fix-ALL, with SQLENS fixing 2 more SQL queries on MACSQL and nearly the same number on the other two methods. However, SQLENS produces more regressions than than on BIRD, causing SQLENS to trigger more corrections, which results in a higher number of regressions.
This is because error signals have lower accuracy on Spider
Table 10: End-to-end accuracy improvement on Spider.
Method
A Anee(Nne)
A Acc.(Naa
Nnet
Nix
Nureak
Vanilla (Initial Acc.=79.65%); # input SQLs=
Self-Reflection
80.14 (+0.49%%
80.72 (+1.07%)
11
6
SQLENS w. Fix-ALL
81.11 (+1.46% %))
81.4 (+1.75%)
15
18
3
SQLENS
81.11 (+1.46%) %)
81.5 %))
15
19
4
DIN-SQL (Initial (ecc=77.66%)) input SQLs=1011
Self-Reflection
80.51 (+2.85%)
81.1 (+3.44%)
29
35
6
SQLENS Fix-ALL
82.58 (+4.92%)
82.88 (+5. -5.22%%
50
53
3
SQLENS
82.48 (+4.82%%
83.27 (+5.61 %)
49
57
8
MAC-SQL (Initial Acc.=79.6%%. # input SQLs=
Self-Reflection
81.06 (+1.37%)
81.45 (+1.76%)
14
18
4
SQLENS Fix-ALL
81.74 (+2.05%)
82.03 (+2.34%)
21
24
3
SQLENS
81.93 (+2.24%)
82.23 (+2.54%)
23
26
3
F.2 Error Detection on Spider
On Spider benchmark, we observe similar results, with SQLENS outperforming LLM self-evaluation methods in terms of both recall and F1 score (Table [1]. A notable observation is that LLM self-evaluation tends to be overly confident in the generated SQL queries, leading to low recall in identifying incorrect queries. For instance, LLM Self-Eval. (Prob) only identifies 5.16% of the incorrect SQL queries on those generated by MAC-SQL. The overall F1 score on Spider is lower than on BIRD because the text-to-SQL algorithms already achieve around 80% accuracy Spider, which is significantly higher than on BIRD. This creates more challenging scenario, as most remaining errors fall within the long-tail distribution.
F.3 Discussion
As confirmed by our experimental results, SQLENS is highly effective in error detection and correc- tion, particularly benefiting non-technical users with limited SQL knowledge. However, like many other LLM--ased text-to-SQL. solutions, our method incurs some costs and latency. On average, SQLENS makes approximately LLM calls with end-to-end latency of less than 30 seconds. To further reduce costs and latency, users can configure cheaper error signals, provide ground truth labels for SQL correctness, or limit the number of iterations SQLENS's eгтоr correction process. As noted in the motivating example in Section | SQLENS is not designed for real-time, interactive SQL query correction. Instead, it functions as a debugging tool, detecting and correcting errors in SQL queries asynchronously.
E4 Effectiveness of Individual Error Signals on BIRD and Spider
32

---

Table 11: Effectiveness of SQLENS detection Spider.
Method
Acc.
AUC
Precision
Recall
F1
Vanilla
LLM Self-Eval. (Bool)
74.39 (23.12)
21.59 (#1300)
9.13 (#521)
12.77 (#742)
L.M Self-Eval. (Prob)
77.41 (21143))
57.42 1.300
13.33 (217778
2.90 (#:89)
4.77 (26.38)
Supervised SQLENS
80.72' (#1006)
61.02
(#:3.28)
75.79
(#17)24)
10.56
17.79 (#6622)
SQLENS
74.49 (1:486)
59.49 (1:5.9)
34.91
(214.18)
29.22'
(2:1161)
31.76 (2:1275)
DIN-SQL.
LLM Self-Eval. (Bool)
75.69
(1:246)
40.38 (21133)
16.78 (24424)
23.64 (1:00))
LLM Self-Eval. (Prob)
76.58 (2:123))
58.60 (t 1.78)
41.07 (#1797)
5.73 (#2.23)
9.90 (2:372)
Supervised SQLENS
82,48'
(2:1.27)
67.89'
(6:3666
73.68' (2745)
33.57 (29.14)
45,51 (1:991)
SQLENS
76.38 (#2275)
67.14 (4-4.20)
47.18
(4:5.76)
48.09' (1-6.91)
47.59' (#:6.17)
MAC-SQL
LLM Self-Eval. (Bool)
76.85 (1:1.60)
31,20 (2:1122)
11.52 (24111)
16.81 (1:598)
LLM Self-Eval. (Prob)
78.51 (1:121))
57.81
(1:442)
42.52 (1:30.00)
5.31 (#244)
9.07 (1:370)
Supervised SQLENS
79.98' (::1,39
61.59*
(#3.10)
54.83' (2:21.06
9.65 (18I))
15.10 (2:11.29)
SQLENS
74.41 (d:2244)
58.99
(#3.99)
35.88 (2:522))
32.23' (d:433)
33.88' (d:4.99)
indicates that classification is not threshold-based Top result. Top two results are highlighted in bold.
Table 12: Individual error signal performance (VanillaBIIRD).
Table 13: Individual enror signal performance (Vanilla+Spider).
Signal Name
Precision
Recall
N
Signal Name
Precision
Recall
N
DB-based Signals
DB-based Signals
Abnormal Result
98.96
16.13
95
Abnormal Result
60.0
14.35
30
Empty Predicate
85,34
11.88
70
Empty Predicate
50.0
2.39
5
Incorrect Filter in Subquery
No Queries
Detected
Incorrett Filter in Subquery
100.0
0.48
Incorrect GROUP BY
40.0
2.38
Incorrect GROUP BY
64.29
4.31
Incorrect Join Predicate
100
1.87
11
Incorrect Join Predicate
100.0
5.26
11
Subuptimal Join Tree
45.9
14.26
84
Suboptimal Join Tree
42.86
5,74
12
Table Similarity
67.35
Table Similarity
Queries Detected
Unnecessary Subquery
33.33
0.34
2
Unnecessary Subquery
100.0
0.48
Value Ambiguity
53.85
7.13
42
Value Ambiguity
33.33
1.44
L.M--based Signals
I.MM-based Signals
Column Ambiguity
68.75
1.87
11
Column Ambiguity
33.33
0.48
1
Evidence Violation
61.11
1.87
11
Evidence Violation
100
1.44
Insufficient Evidence
29.03
1.53
Insufficient Evidence
7.14
LLM Self Check
60.82
10.02
59
LLM Self Check
20.65
9,09
19
Question Clause Linking
53.49
3.9
23
Question Clause Linking
70.0
3.35
7
33

---

Table 14: Individual error signal performance (DIN-SOL+Spider).
Table 15: Individual error signal performance (MAC-SQL+Spider))
Signal Name
Precision
Recall
N
Signal Name
Precision
Recall
N
DB-ba Signals
DB-based Signals
Abnormal Result
73.61
23.35
Abnormal Result
14.42
Empty Predicate
82.98
17.18
39
Empty Predicate
52.94
4.33
Incorrect Filter in Subquery
No Queries Detected
Incorrect Filter in Subquery
100
1.44
Incorrect GROUP BY
43.33
5.73
Incorrect GROUP
50
3.37
Incorrect Join Predicate
11.45
Incorrect Join Predicate
100
3.85
Suboptimal Join Tree
33.33
5.73
13
Suboptimal Join Tree
38.71
5.77
12
Table Similarity
No Queries Detected
Table Similarity
No Queries Detected
Unnecessary Subquery
Unnccessary Subquery
33.33
0.48
Value Ambiguity
25.0
1.32
Value Ambiguity
25.0
1.44
LLM-based Signals
LLM--bssed Signals
Column Ambiguity
1.76
Column Ambiguity
Evidence Violation
80
1.76
4
Fvidence Violation
58.33
3.37
Insufficient Evidence
40
7.93
18
Insufficient Evidence
18.92
3.37
Question Clause Linking
53.33
3.52
8
L.M Check
31.17
11.54
LLM Self Check
39.58
16.74
38
Question Clause Linking
40
2.88
34


Reviews:
[{"review_id": "qbYrA4DBMU", "reviewer": "Reviewer_PeED", "summary": "This paper proposes SQLENS, an end-to-end framework for fine-grained detection and correction of semantic errors in LLM-generated SQL queries. To address the issue of syntactically correct but semantically incorrect outputs, SQLENS integrates error signals from both the underlying database and the LLM to identify potential errors at the clause level. It uses weak supervision to aggregate noisy signals and\ntrain a classifier for error detection, and introduces a step-by-step correction mechanism guided by detailed error reports. Experimental results show that SQLEN outperforms existing LLM-based self-evaluation methods by 25.78% in F1-score for error detection and improves execution accuracy by up to 20% on benchmark datasets such as BIRD and Spider.", "questions": "- The workflow of the paper is not clearly explained. How many error reports are generated for one SQL query? Can the authors show an end-to-end example of a SQL query from generation to correction?\n- In the Error Selector, which errors are most frequently selected first for correction?\n- How many tokens does SQLENS require to fix a single SQL query?", "limitations": "YES", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n- The paper proposes fine-grained text-to-SQL error categories that cover the majority of possible errors.\n- This paper conducts a comprehensive comparison across SQL generation and various baselines, demonstrating that SQLENS effectively improves the final execution performance.\n\n\n## Weaknesses:\n- Although the paper selects comprehensive SQL generation methods as baselines, it does not make a comprehensive selection of fix baselines, such as SQLFixAgent (https://arxiv.org/abs/2406.13408 ) and Self-debug (https://arxiv.org/abs/2304.05128 ).\n- The paper only uses Claude 3.5 Sonnet as the correction model, which limits the generalization ability of SQLENS. Is the method effective on other closed-source models (e.g., GPT-4o) or open-source models (e.g., Qwen)?", "was_revised": true, "final_justification": "The concerns have been addressed through structured explanations and comprehensive experiments. The rating is thus raised.", "rebuttal": "We thank the reviewer for their thoughtful feedback, which has greatly contributed to improving our submission. We appreciate your insights and hope that our detailed responses and new results below address your concerns. We kindly ask you to consider revising your score in light of these updates.\n\n> W2: The paper only uses Claude 3.5 Sonnet as the correction model, which limits the generalization ability of SQLENS. Is the method effective on other closed-source models (e.g., GPT-4o) or open-source models (e.g., Qwen)?\n\nWe thank the reviewer for raising this concern. To evaluate the generalization ability of SQLens, we conducted a preliminary experiment by switching the correction model from Claude 3.5 Sonnet to GPT‑4o. Results are summarized below:\n\n| SQL Generation Method | Initial Accuracy | With SQLens (GPT‑4o) | With SQLens (Claude 3.5 Sonnet) |\n|-|-|-|-|\n| Vanilla| 59.07| **62.75% (+3.68%)**  | 62.54 (+3.47%)|\n| DIN-SQL| 39.49| **61.08% (+21.59%)** | 59.99 (+20.50%)|\n| MAC-SQL| 59.32| 62.18% (+2.86%) | **64.03 (+3.99%)** |\n| CHESS| 67.91| **69.84% (+1.93%)**  | 69.74 (+1.83%)|\n\nAcross all SQL generation methods, SQLens consistently improves execution accuracy regardless of the correction model employed. In 3 out of 4 cases, GPT‑4o outperforms Claude 3.5 Sonnet. These results demonstrate that SQLens is not dependent on a specific large language model and generalizes effectively across different closed-source models. A more comprehensive generalization analysis, including additional models such as Qwen, will be presented in the final version.\n\n> W1: Although the paper selects comprehensive SQL generation methods as baselines, it does not make a comprehensive selection of fix baselines, such as SQLFixAgent and Self-debug.\n\nTo address this concern, we first clarify our experimental setup and then provide direct comparisons with the suggested self-correction baselines. We appreciate the reviewer highlighting these works and will include a detailed discussion of them in the full paper.\n\n**On self-correction baselines and experimental comparison:**\n\nFirst, our baseline comparisons focus on evaluating SQLens in conjunction with strong baseline SQL generation models that already incorporate self-correction mechanisms similar to those used in methods like SQLFixAgent, Self-Debug, and MAGIC [1]. SQLens demonstrates its added value beyond existing self-correction strategies, effectively measuring improvements in a setting where such techniques are already in place. \n\n[1] MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL\n\nMore specifically, the state-of-the-art SQL generation methods we evaluate include the following self-correction paradigms:\n\n- **CHESS** employs a detailed, multi-step reasoning process in its prompts, which is similar to the guideline proposed in MAGIC (Askari et al., 2024).\n\n- **DIN-SQL** includes a self-correction phase where the model identifies potential issues and suggests clause-level fixes using chain-of-thought prompting, similar to Self-Debug.\n\nTherefore, the effectiveness of SQLens is demonstrated on top of these strong baselines, reflecting its added value beyond existing self-correction methods.\n\n**Quantitative performance comparison:**\n\nSecond, we also compare SQLens to the best results reported in MAGIC, Self-Debug, and SQLFixAgent on the BIRD dev set:\n\n| Method|SQL Generation Method Initial EX Accuracy|EX Accuracy after Correction|\n|-|-|-|\n| MAGIC (Askari et, al, 2024) | 56.52 (using DIN-SQL)| 59.13 (+2.61)|\n| Self-Debug (Chen et al., 2023)| 56.52 (using DIN-SQL)| 57.24 (+0.72)|\n| SQLFixAgent (Cen et al., 2025)| 57.17 (using CodeS-7b) | 60.17 (+3.00)|\n| **SQLens**| **60.04 (using MAC-SQL)**| **65.23 (+5.19)**|\n\nNotably, SQLens achieves larger gains **despite using a much stronger starting point for SQL generation** (60.04% EX accuracy vs. 56.52%), indicating its ability to further improve robust baselines. In the full paper, we will provide a more thorough comparison by configuring all methods with the same SQL generation approach and backbone model.\n\n**Conceptual distinction from self-correction methods:**\n\nWhile methods like MAGIC and Self-Debug focus on LLM-driven self-correction, SQLens fundamentally differs in both the source and structure of its error detection and correction:\n\n- **Evidence source:** Unlike self-correction methods that rely solely on LLM introspection, SQLens draws on *external, verifiable signals* for error detection. This includes both (a) database-based signals (such as empty predicate, sub-optimal join tree, and incorrect join predicate) obtained through static analysis and query plan analysis without any LLM involvement and thus more efficient, and (b) LLM-based signals, which are treated as one component within a larger ensemble of evidence, not as the sole decision maker.\n- **Actionable, clause-level error reports:** SQLens provides structured error reports that precisely map detected errors to specific SQL clauses, supply targeted correction instructions, and indicate confidence. This enables users and systems to pinpoint and address faults more effectively than free-form LLM critiques.\n- **General-purpose SQL debugger:** SQLens is designed to serve as an explainable, modular debugging framework for post-hoc auditing across text-to-SQL systems, rather than as a one-off LLM prompt or single-pass correction step.\n\n# Questions\n\n> Q1: The workflow of the paper is not clearly explained. How many error reports are generated for one SQL query? Can the authors show an end-to-end example of a SQL query from generation to correction?\n\nThanks for pointing this out. For each SQL query, SQLens generates a single error report, which consolidates the information from all individual error signals detected for that query. Each error signal provides details such as a description, suggested correction, the problematic clauses, and a confidence score. The aggregated error report brings together all these findings, giving a comprehensive summary for the user or the subsequent correction agent.\n\n**End-to-end example:**\n\nUsing the SQL query in Figure 2 of our original submission as an example:\n\n1. First, the SQLens error detector analyzes the generated SQL and identifies four error signals: Suboptimal Join Tree, Incorrect Join Predicate, Evidence Violation, and Empty Predicate.\n2. The final error report aggregates details from all these signals, including the type and location of each error, suggested corrections, and confidence levels.\n3. This comprehensive error report is then provided to the Error Selector, which prioritizes and guides the iterative correction process as outlined in Figure 1.\n\nAn example entry in an error report is shown in Figure 5 in the original submission. \n\n> Q2: In the Error Selector, which errors are most frequently selected first for correction?\n\nThank you for the insightful question. We analyzed the fix sequences for SQL queries generated by DIN-SQL. Out of the 415 queries where multiple error signals were flagged, the top three most frequently selected errors to be fixed first by the Error Selector are as follows:\n\n| Signal Name| Explanation| Times Selected First |\n| - | -|-|\n| Empty Predicate    | Detects if there is a predicate within a SQL query that yields an empty result. | 248|\n| Evidence Violation | Identifies cases where the generated SQL query contradicts the evidence provided in the question or external knowledge. | 98 |\n| Column Ambiguity   | Detects if there are columns in the database that are very similar to the ones used in the SQL query and could also be used to answer the user's question. | 36  |\n\nEmpty Predicate tends to be chosen first because it is an unambiguous error, directly verified by the database execution result. It is also straightforward to identify and fix, such as correcting predicate formatting. This highlights the effectiveness of error signals grounded in database-based evidence.\n\n> How many tokens does SQLE require to fix a single SQL query?\n\nWe measured the token consumption of SQLens on the BIRD dataset, breaking it down by component:\n\n| Component  | Total Tokens|\n| - | - |\n| Error Detector | 3141 |\n| Error Selector | 2345 |\n| Error Fixer| 2262|\n| SQL Auditor| 2215|\n\nA large portion of these tokens (about 1,915 tokens per component) comes from the database description string, which includes schema information and example values.\n\nSQLens performs at most 3 correction iterations, leading to a worst-case cost of 25,459 tokens per SQL query. However, in practice, most queries are corrected in a single iteration, consuming only ~9,963 tokens on average.\n\nThis token cost is much smaller than the cost of generating SQL queries with SOTA text-to-SQL models. For example, CHASE-SQL requires roughly 160,000 tokens to generate a single SQL query. Therefore, the token overhead introduced by SQLens is negligible compared to the total cost of query generation.\n\nFurthermore, the cost of generating queries via human annotation is still significantly higher than SQLens, making our approach more efficient in practice."}, {"review_id": "BwPA9IPJbF", "reviewer": "Reviewer_Bdfs", "summary": "The authors propose a new framework for Text2SQL. To goals are threefold: (1) Detect errors in the input, (2) Fix the error, (3) Provide the fixed version as final output. Tha proposed approach is based on employs a learning-based approach that aggregates complementary signals identifying potential errors.", "questions": "- Is the definition of semantic ambiguity consistent with the SOTA?\n- Are you able to empirically compare with the SOTA on specific ambiguity types?\n- How did the authors handle databases with a large number of tables and more complex queries (e.g., in business domains)?", "limitations": "- The authors explored a relatively limited number of LLMs and prior works. The scope of the empirical evaluation and the experimental comparisons with SOTA can be expanded.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "I agree with the authors that the problem of Text2SQL is far from being completely solved. A huge body of work has been devoted to specializing LMs to effectively address the task of adopt in-context learning to address the limitations of existing LLMs. In this work, instead, the authors mainly address the issues of semantic error detection and solving, which are particularly relevant to this field.\n\nI have concerns on the strategy used to produce reliable error signals. The learning-based approach based on weak supervision is quite risky, as a lot of false signals are likely generated on complex datasets. The current results, reported on Bird and Spider v1, fail to demonstrate the actual robustness of the approach to more complex cases (see, e.g., Spiderv2, BEAVER).\n\nThe formulation of the concept of semantic ambiguity is not clear enough. The related literature appears to be also incomplete. See, for example, \n\nFuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and Amr El Abbadi. 2025. Sphinteract: Resolving Ambiguities in NL2SQL through User Interaction. Proc. VLDB Endow. 18, 4 (December 2024), 1145–1158. https://doi.org/10.14778/3717755.3717772\n\nWithout a clearer formalization of semantic ambiguity, a detailed comparison with all SOTA, and an empirical evaluation on complex datasets the novelty and effectiveness of the proposed approach remains questionable.", "was_revised": true, "final_justification": "I appreciate the authors' feedback and effort in addressing the limitations of the the empirical validation. I still have some concerns on the risks of weak supervision, but overall the paper quality is fairly above average so I assigned rating 4.", "rebuttal": "Thank you for the insightful questions. Below, we provide clarifications and additional results that we hope address your concerns. We would appreciate your consideration of a score adjustment based on these updates.\n\n## On the relatively limited number of LLMs and prior works\n\n> The authors explored a relatively limited number of LLMs and prior works. The scope of the empirical evaluation and the experimental comparisons with SOTA can be expanded.\n\nTo address this concern, we conducted additional experiments using GPT‑4o as the correction model, in addition to Claude 3.5 Sonnet, and included comparisons with state-of-the-art self-correction methods such as MAGIC (Askari et al., 2024), Self-Debug (Chen et al., 2023), and SQLFixAgent (Cen et al., 2025).\n\n**Results with GPT-4o**:\n\n| SQL Generation Method | Initial Accuracy | With SQLens (GPT‑4o) | With SQLens (Claude 3.5 Sonnet) |\n|-|-|-|-|\n| Vanilla| 59.07| **62.75% (+3.68%)**  | 62.54 (+3.47%)|\n| DIN-SQL| 39.49| **61.08% (+21.59%)** | 59.99 (+20.50%)|\n| MAC-SQL| 59.32| 62.18% (+2.86%) | **64.03 (+3.99%)** |\n| CHESS| 67.91| **69.84% (+1.93%)**  | 69.74 (+1.83%)|\n\nAcross all SQL generation methods, SQLens consistently improves execution accuracy regardless of the correction model employed. In 3 out of 4 cases, GPT‑4o outperforms Claude 3.5 Sonnet. These results demonstrate that SQLens is not dependent on a specific large language model and generalizes effectively across different closed-source models. A more comprehensive generalization analysis, including additional models such as Qwen, will be presented in the final version.\n\n**Comparison with SOTA self-correction methods:**\nWe also compared SQLens to the best reported results for MAGIC, Self-Debug, and SQLFixAgent on the BIRD dev set:\n\n| Method|SQL Generation Method Initial EX Accuracy|EX Accuracy after Correction|\n|-|-|-|\n| MAGIC (Askari et, al, 2024) | 56.52 (using DIN-SQL)| 59.13 (+2.61)|\n| Self-Debug (Chen et al., 2023)| 56.52 (using DIN-SQL)| 57.24 (+0.72)|\n| SQLFixAgent (Cen et al., 2025)| 57.17 (using CodeS-7b) | 60.17 (+3.00)|\n| **SQLens**| **60.04 (using MAC-SQL)**| **65.23 (+5.19)**|\n\nNotably, SQLens achieves larger gains **despite using a much stronger starting point for SQL generation** (60.04% EX accuracy vs. 56.52%), indicating its ability to improve advanced SQL generation baselines further. In the full paper, we will provide a more thorough comparison by configuring all methods with the same SQL generation approach and backbone model.\n\n## On the definition of semantic ambiguity\n\n> The formulation of the concept of semantic ambiguity is not clear enough. The related literature appears to be also incomplete. See, for example, Fuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and Amr El Abbadi. 2025. Sphinteract: Resolving Ambiguities in NL2SQL through User Interaction. \n\n> Is the definition of semantic ambiguity consistent with the SOTA? Are you able to empirically compare with the SOTA on specific ambiguity types?\n\nThank you for raising this important point. We will expand our discussion of semantic ambiguity and related literature, including Sphinteract, in the revised paper.\n\nOur definition of semantic ambiguity is consistent with the state-of-the-art, as formalized in Sphinteract. Specifically, our error signals map directly to the ambiguity types they define: for example, our Column Ambiguity signal corresponds to the AmbColumn type, and our Value Ambiguity signal aligns with AmbValue. We will clarify this mapping and explicitly reference these works in the revision.\n\nHowever, semantic ambiguity is only one aspect addressed by SQLens. The aim of SQLens is to provide a comprehensive clause-level debugging framework for a broad spectrum of SQL errors, leveraging both database-derived and LLM-derived signals. This enables SQLens to support actionable debugging across a variety of failure types, not just ambiguity.\n\nA direct empirical comparison with Sphinteract is beyond the scope of this paper, as Sphinteract focuses on ambiguity resolution via user interaction, while SQLens is designed to automatically detect and explain a range of errors through detailed error reports. In this regard, our evaluation is more closely aligned with recent self-correction approaches like MAGIC, Self-Debug, and SQLFixAgent, and we have included those comparisons in our results above.\n\n## On concerns about the weak supervision method\n>I have concerns on the strategy used to produce reliable error signals. The learning-based approach based on weak supervision is quite risky, as a lot of false signals are likely generated on complex datasets. \n\nThank you for highlighting this important concern. We agree that managing noise and potential false signals is a core challenge in any weak supervision framework, particularly for complex datasets. We have designed SQLens with several safeguards and validation steps to address this:\n\n**1. High-Precision Individual Signals:** We rigorously evaluate each error signal’s precision and recall (Table 4, Page 8). Many database-based signals, such as Empty Predicate (96.07% precision on DIN-SQL) and Incorrect Join Predicate (near-perfect precision), demonstrate strong reliability on their own. This helps ensure that the foundation of our weak supervision framework is built on trustworthy signals.\n\n**2. LLM Discretion and Safeguards Against Regression:** We recognize that some correct queries could be mistakenly flagged by individual signals. To prevent unnecessary or harmful edits, SQLens does not force the SQLFixer to apply corrections simply because an error signal is present. The LLM-based SQLFixer always has the explicit option to make no change if it determines the flagged error is not valid. Additionally, SQLens incorporates a SQLAuditor and a Guardrail signal (see Figure 1) to further safeguard against regressions—ensuring that proposed corrections do not degrade the original query’s correctness. These mechanisms together help maintain high precision and prevent overcorrection.\n\n**3. Net Effect on Correct Queries:** To ensure that SQLens does not degrade performance by incorrectly modifying already correct queries, we report $N_{fix}$ (number corrected), $N_{break}$ (number broken), and $N_{net}$ (net improvement) for all experiments (Table 1, Page 7). SQLens consistently achieves a positive $N_{net}$, even when the input set includes both correct and incorrect queries. For example, on MACSQL, SQLens fixes 78 queries while breaking only 18 ($N_{net}$ = 60), confirming that the benefits of correction substantially outweigh false positives.\n\nIn summary, we agree that weak supervision has tradeoffs. However, acquiring ground-truth labels at clause-level granularity is extremely expensive, especially for complex datasets like Spider v2 or BEAVER. Our weak supervision approach, implemented using the Snorkel [1] library, has shown high F1 performance for semantic error detection, even without labeled data (Table 2). We also demonstrate that weakly-supervised SQLens outperforms LLM self-evaluation baselines by large margins (e.g., +21.45 F1 points on DIN-SQL).\n\n[1] Snorkel: rapid training data creation with weak supervision.\n\n## On generalization beyond BIRD/Spider v1\n> The current results, reported on Bird and Spider v1, fail to demonstrate the actual robustness of the approach to more complex cases (see, e.g., Spiderv2, BEAVER).\n> How did the authors handle databases with a large number of tables and more complex queries (e.g., in business domains)?\n\nWe appreciate the reviewer’s point about evaluating on more complex datasets such as Spider v2 and BEAVER. Our current experiments focused on BIRD and Spider v1 to enable direct comparison with prior works. However, SQLens is designed to be model- and dataset-agnostic, and in principle can be applied to newer benchmarks. As future work, we plan to include evaluations on Spider v2 and BEAVER, and will clarify this intent in the final version."}, {"review_id": "wzSyYktQIr", "reviewer": "Reviewer_nn1f", "summary": "Authors propose SQLENS, a framework for detecting and correcting semantic errors in LLM-generated SQL queries. While the problem is important and the approach is systematic, the contribution feels incremental and the evaluation has some limitations.", "questions": "q1. Whether this latency is relevant to network connect? Is there a linear relationship between number of LLM calls and cost? Given authors design specific prompt techniques leading to more token consumption in each call, so number of LLM calls cannot be equal to the cost. Also such reduction can be observed in Other llm backbone?\n\nq2. Is this work also effective to other LLM backbone?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n\nS1. Authors propose an important problem: semantic error detection in text-to-SQL.\n\nS2. The signal design is comprehensive. Authors show 14 error signals which covers diverse erry types and are well-motivated by real SQL problems.\n\nS3. Experiments on standard benchmarks with multiple baseline text-to-SQL methods are promising.\n\n# Weakness:\nW1. The problem is similar to self-correction, but authors didn't compare those popular self-correction methods in the experiment, such as Magic [1].\n\nW2. The improvement is marginal. \n\nW3. The method has been only verified on one LLM backbone: Claude 3.5 Sonnet.\n\nW4. There are no cost analysis for this methods. Only LLM calls are not professional and questionable. For example, whether this latency is relevant to network connect? Is there a linear relationship between number of LLM calls and cost? Given authors design specific prompt techniques leading to more token consumption in each call, so number of LLM calls cannot be equal to the cost. Also such reduction can be observed in Other llm backbone?\n\n\n\n# Reference:\n[1] MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL.", "was_revised": true, "final_justification": "In summary, this work partially resolved my concerns (3/4), and I always appreciate value of this work, just need more details and careful analysis and statistics, therefore, I change my rating to the positive.", "rebuttal": "Thank you so much for your detailed and constructive comments, and we truly appreciate them. We hope the clarification and additional results provided below address your concerns. We kindly ask you to consider the possibility of a score adjustment.\n\n## On the effectiveness of SQLens on other LLM backbones\n\n> W3. The method has been only verified on one LLM backbone: Claude 3.5 Sonnet.\n\n> q2. Is this work also effective to other LLM backbone?\n\nWe thank the reviewer for raising this concern. To evaluate the generalization ability of SQLens, we conducted a preliminary experiment by switching the correction model from Claude 3.5 Sonnet to GPT‑4o. Results are summarized below:\n\n| SQL Generation Method | Initial Accuracy | With SQLens (GPT‑4o) | With SQLens (Claude 3.5 Sonnet) |\n| --------------------- | ---------------- | -------------------- | ------------------------------- |\n| Vanilla               | 59.07            | **62.75% (+3.68%)**  | 62.54 (+3.47%)                  |\n| DIN-SQL               | 39.49            | **61.08% (+21.59%)** | 59.99 (+20.50%)                 |\n| MAC-SQL               | 59.32            | 62.18% (+2.86%)      | **64.03 (+3.99%)**              |\n| CHESS                 | 67.91            | **69.84% (+1.93%)**  | 69.74 (+1.83%)                  |\n\nAcross all SQL generation methods, SQLens consistently improves execution accuracy regardless of the correction model employed. In 3 out of 4 cases, GPT‑4o outperforms Claude 3.5 Sonnet. These results demonstrate that SQLens is not dependent on a specific large language model and generalizes effectively across different closed-source models. A more comprehensive generalization analysis, including additional models such as Qwen, will be presented in the final version.\n\n## On comparison with other self-correction methods\n\n> W1. The problem is similar to self-correction, but authors didn't compare those popular self-correction methods in the experiment, such as Magic [1].\n\nThanks for highlighting the connection to self-correction approaches. We appreciate the opportunity to clarify the distinctions and our experimental choices:\n\n**On self-correction baselines and experimental comparison:**\n\nFirst, our baseline comparisons focus on evaluating SQLens in conjunction with strong baseline SQL generation models that already incorporate self-correction mechanisms similar to those used in methods like SQLFixAgent[1], Self-Debug[2], and MAGIC[3]. SQLens demonstrates its added value beyond existing self-correction strategies, effectively measuring improvements in a setting where such techniques are already in place.\n\n[1] SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration\n\n[2] Teaching Large Language Models to Self-Debug\n\n[3] MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL\n\nMore specifically, the state-of-the-art SQL generation methods we evaluate include the following self-correction paradigms:\n\n- **CHESS** employs a detailed, multi-step reasoning process in its prompts, which is similar to the guideline proposed in MAGIC (Askari et al., 2024).\n\n- **DIN-SQL** includes a self-correction phase where the model identifies potential issues and suggests clause-level fixes using chain-of-thought prompting, similar to Self-Debug.\n\nTherefore, the effectiveness of SQLens is demonstrated on top of these strong baselines, reflecting its added value beyond existing self-correction methods.\n\n**Quantitative performance comparison:**\n\nSecond, we also compare SQLens to the best results reported in MAGIC, Self-Debug, and SQLFixAgent on the BIRD dev set:\n\n| Method|SQL Generation Method Initial EX Accuracy|EX Accuracy after Correction|\n|-|-|-|\n| MAGIC (Askari et, al, 2024) | 56.52 (using DIN-SQL)| 59.13 (+2.61)|\n| Self-Debug (Chen et al., 2023)| 56.52 (using DIN-SQL)| 57.24 (+0.72)|\n| SQLFixAgent (Cen et al., 2025)| 57.17 (using CodeS-7b) | 60.17 (+3.00)|\n| **SQLens**| **60.04 (using MAC-SQL)**| **65.23 (+5.19)**|\n\nNotably, SQLens achieves larger gains **despite using a much stronger starting point for SQL generation** (60.04% EX accuracy vs. 56.52%), indicating its ability to improve robust baselines further. In the full paper, we will provide a more thorough comparison by configuring all methods with the same SQL generation approach and backbone model.\n\n**Conceptual distinction from self-correction methods:**\n\nWhile methods like MAGIC and Self-Debug focus on LLM-driven self-correction, SQLens fundamentally differs in both the source and structure of its error detection and correction:\n\n- **Evidence source:** Unlike self-correction methods that rely solely on LLM introspection, SQLens draws on *external and verifiable signals* for error detection covering a wide range of error causes. This includes both (a) database-based signals (such as empty predicate, sub-optimal join tree, and incorrect join predicate) obtained through static analysis and query plan analysis without any LLM involvement and thus more efficient, and (b) LLM-based signals, which are treated as one component within a larger ensemble of evidence, not as the sole decision maker.\n- **Actionable, clause-level error reports:** SQLens provides structured error reports that precisely map detected errors to specific SQL clauses, supply targeted correction instructions, and indicate confidence. This enables users and systems to pinpoint and address faults more effectively than free-form LLM critiques.\n- **General-purpose SQL debugger:** SQLens is designed to serve as an explainable, modular debugging framework for post-hoc auditing across text-to-SQL systems, rather than as a one-off LLM prompt or single-pass correction step.\n\n**On the “marginal improvement” concern:**\n\nWhile the absolute gains in end-to-end execution accuracy may appear modest, it is important to consider the following:\n\n1. **Challenging baseline**: SQLens is evaluated on much stronger SQL generation baselines than those used in prior self-correction papers. These models already incorporate advanced techniques like chain-of-thought prompting and self-correction, making further improvements inherently difficult. In this context, the gains achieved by SQLens are significant.\n2. **Value beyond accuracy**: SQLens adds substantial value beyond raw accuracy by providing clause-level error reports. These reports map errors to specific SQL clauses and offer actionable guidance, enabling users and engineers to better understand and address underlying issues.\n3. **Handling cases where automated correction falls short**: In practice, we observe that there are cases where SQLens accurately detects and localizes errors and provides clear instructions for correction, but the underlying LLM does not successfully apply the fix. Because SQLens generates a comprehensive clause-level error report, users and engineers can still easily understand, verify, and manually resolve issues in the query when automated correction falls short.\n\nIn summary, SQLens transforms error detection and correction into a more transparent and more controllable workflow. This new paradigm for clause-level debugging and correction is critical for real-world deployment and trust.\n\nWe hope this addresses the reviewer’s concerns and clarifies the unique value of SQLens beyond what existing self-correction methods provide.\n\n## On cost analysis:\n> W4. There are no cost analysis for this methods. ....\n\nThanks for pointing this out. We recognize that reporting only the number of LLM calls is not sufficient for a comprehensive cost analysis. To address this concern, we conducted a detailed token cost analysis of SQLens on BIRD, breaking it down by component, and report the results here. \n\n| Component  | Total Tokens|\n| - | - |\n| Error Detector | 3141 |\n| Error Selector | 2345 |\n| Error Fixer| 2262|\n| SQL Auditor| 2215|\n\nA large portion of these tokens (about 1,915 tokens per component) comes from the database description string, which includes schema information and example values.\n\nSQLens performs at most 3 correction iterations, leading to a worst-case cost of 25,459 tokens per SQL query. However, in practice, most queries are corrected in a single iteration, consuming only ~9,963 tokens on average.\n\nThis token cost is much smaller than the cost of generating SQL queries with SOTA text-to-SQL models. For example, CHASE-SQL requires roughly 160,000 tokens to generate a single SQL query. Therefore, the token overhead introduced by SQLens is negligible compared to the total cost of query generation.\n\nFurthermore, the cost of generating queries via human annotation is still significantly higher than SQLens, making our approach more efficient in practice.\n\n> Whether this latency is relevant to network connect?\n\nThe only component in our system that involves network communication is the LLM API call, which requires sending a request to a remote server and waiting for the response. Therefore, the reported latency does include the network round-trip time for LLM calls. Other components, such as database signal extraction and local processing, do not incur network latency."}, {"review_id": "FOxeWfAd4s", "reviewer": "Reviewer_ATVr", "summary": "The paper proposes SQLens, an error detection and correction framework for text-to-SQL translation. The authors introduce a weak supervision approach to aggregate database and LLMs error signals and iteratively fix the most critical errors. Evaluation on BIRD and Spider shows performance improvements, presenting its effectiveness in error detection and correction, and the authors also include ablation studies to validate the effectiveness of each module and signal.", "questions": "**Please refer to the weaknesses.**\n\n**Additional Questions:**\n1. The EX performance of DIN-SQL is inferior to vanilla in Table 1, which is somewhat counterintuitive.\n2. The error correction methods in the experiment show an improvement of a few percentage points on other text-to-SQL approaches, but *alone on DIN-SQL*, all methods have an extremely high increase (15-20%), which is a bit questionable.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. Semantic errors remain a bottleneck in current text-to-SQL translation, and studies in this paper is timely.\n2. The use of weak supervision to aggregate noisy error signals is a new and great attempt, as it also circumvents the need for labeled data.\n3. The experiments and ablation studies are thorough and systematic, and the reported results are convincing.\n\n------\n\n### **Weaknesses**\n\n1. While the iterative correction framework provides marginal improvements over the fix-all setting, it introduces concerns about increased token cost and latency, which may hinder practical deployment.\n2. Although I believe weak supervision for aggregating noisy error signals is a good attempt, I still have major concerns:\n   - The quality and accuracy of individual LFs are not well validated and seem largely heuristic.\n   - The weak supervision model is treated as a black box, with little analysis of its sensitivity to LF quality, number, or correlation. It’s also unclear how much value comes from the aggregation model versus a few strong signals used directly. If possible, an ablation study on individual signals could help clarify this.\n   - In table 4, the recall of error detection for individual error signals are quite low, indicating the difficulty of effectively detecting these signals.\n   - While results on Spider are reported, it is unclear whether the classifier used in that setting was trained on Spider training set or transferred from another dataset.\n   - The paper omits important details about the training setup for the weakly supervised classifier, such as the model architecture and training procedure.\n3. Some key technical details lack sufficient clarity and elaboration:\n   - Line 209: \"Error Selector uses an LLM to prioritize which error to fix first...\" – How is this prioritization determined by the LLM in practice? By prompt or some other strategies?\n   - Lines 180-183: \"We use a generative model to estimate the joint distribution... and train a classifier...\" – How exactly is this classifier trained? What is the backbone model and its training strategies?\n   - Line 289: \"We conduct a microbenchmark on BIRD...\" – How is the microbenchmark constructed? Is it based on random or stratified sampling? How many examples are included, and how are the signal labels obtained?\n\n------\n\n### **Minor**\n- I personally think the definitions and examples in the main text are a bit verbose; condensing them could free up space for more technical details.", "was_revised": true, "final_justification": "I appreciate the novelty of the proposed SQL fixing methods and the systematic experiments, and I increase my score to positive.\nHowever, during the rebuttal, my concerns about weak supervision were not fully addressed, so I prefer to remain a borderline recommendation.", "rebuttal": "Thank you for your thoughtful comments, which have helped us strengthen our work. We have included additional results and clarifications below, and we hope these updates address your concerns and may encourage you to reconsider your score.\n\n## On concerns about increased token cost and marginal improvement\n\n> W1: While the iterative correction framework provides marginal improvements over the fix-all setting, it introduces concerns about increased token cost and latency, which may hinder practical deployment.\n\n### On increased token cost\nTo address this concern, we conducted a detailed token cost analysis of SQLens on the BIRD dataset, breaking it down by component, and report the results here. \n\n| Component  | Total Tokens|\n| - | - |\n| Error Detector | 3141 |\n| Error Selector | 2345 |\n| Error Fixer| 2262|\n| SQL Auditor| 2215|\n\nA large portion of these tokens (about 1,915 tokens per component) comes from the database description string, which includes schema information and example values.\n\nSQLens performs at most 3 correction iterations, leading to a worst-case cost of 25,459 tokens per SQL query. However, in practice, most queries are corrected in a single iteration, consuming only ~9,963 tokens on average.\n\nThis token cost is much smaller than the cost of generating SQL queries with SOTA text-to-SQL models. For example, CHASE-SQL requires roughly 160,000 tokens to generate a single SQL query. Therefore, the token overhead introduced by SQLens is negligible compared to the total cost of query generation.\n\n### On latency and practical deployment\nRegarding latency, SQLens is designed as a debugging tool for offline or asynchronous use, not as a real-time component. As described in Section A (Figure 6), SQLens is invoked by users when they want to debug incorrect queries. The observed latency (~30 seconds) is well within acceptable bounds for such a debugging scenario, especially since manual debugging can take hours—particularly for complex semantic errors. As noted in Section 6, SQLens is not intended for latency-sensitive, production-in-the-loop deployment.\n\n**Configuration Options:** As noted in Section F.4, users can optimize SQLens for cost and latency by using a subset of error signals (such as database-only signals that require no LLM calls) or by limiting the number of correction iterations. These options make SQLens practical and flexible for real-world deployments with varying latency and cost requirements.\n\n**On the “marginal improvement” concern:**\n\nWhile end-to-end accuracy gains may appear modest, they are achieved on top of strong SQL generation baselines (e.g., CHESS) that already include advanced self-correction techniques—making further improvement challenging. Additionally, SQLens goes beyond raw accuracy by producing structured, clause-level error reports that help users understand and fix errors, even when the LLM fails to apply the correction. \n\n## On questions about weak supervision\n\n> The quality and accuracy of individual LFs are not well validated and seem largely heuristic.\n\n> The weak supervision model is treated as a black box, with little analysis of its sensitivity to LF quality, number, or correlation. If possible, an ablation study on individual signals could help clarify this.\n\nWe thank the reviewer for these thoughtful questions. Our original submission includes two tables that together provide a detailed analysis of individual signals, addressing both their quality and their standalone utility:\n\n- Table 4 validates the precision and recall of each signal for error detection, showing that many are highly reliable in isolation. For example, Empty Predicate achieves 96.07% precision on DIN-SQL, and Incorrect Join Predicate reaches near-perfect precision. These signals are grounded in reproducible, database-based evidence rather than purely heuristic rules.\n\n- Table 5 evaluates the effect of using each signal independently for correction, reflecting an ablation-style analysis. All signals yield a positive net gain in corrected queries, showing that all signals in SQLens are effective. For example, Suboptimal Join Tree corrects 9 SQL queries, and Evidence Violation (LLM-based) corrects 8.\n\nTogether, these results show that all error signals in SQLens contribute to reliable detection and effective correction.\n\n> In table 4, the recall of error detection for individual error signals are quite low, indicating the difficulty of effectively detecting these signals.\n\n> It’s also unclear how much value comes from the aggregation model versus a few strong signals used directly. \n\nWe would like to clarify the interpretation of recall in Table 4. The recall values do not reflect how well each signal detects its own specific error type. Rather, they indicate the proportion of all erroneous SQL queries that each individual error signal can identify. For example, if there are 100 incorrect SQL queries and the “Empty Predicate” signal is detected in 20 of them, its recall is 20%. Since each signal is designed to focus on a particular error source—targeting different SQL clauses—the recall for each one alone is expected to be relatively low. These signals are intentionally complementary, each covering a distinct class of errors. When combined, their collective recall increases significantly. As shown in Table 2, SQLens achieves over 44% recall in detecting erroneous SQL queries, outperforming LLM self-evaluation by an average of 34.6%. This substantial improvement highlights the value of aggregating diverse signals through weak supervision.\n\n>While results on Spider are reported, it is unclear whether the classifier used in that setting was trained on Spider training set or transferred from another dataset.\n\n>The paper omits important details about the training setup for the weakly supervised classifier, such as the model architecture and training procedure.\n\n> Lines 180-183: \"We use a generative model to estimate the joint distribution... and train a classifier...\" – How exactly is this classifier trained? What is the backbone model and its training strategies?\n\nWe use the LabelModel from the Snorkel library as our weakly supervised classifier, following the architecture described in [1]. The LabelModel does not require any gold labels for training. Instead, it takes as input the label vectors produced by our error signals and learns to classify each SQL query as correct or incorrect by modeling the agreement and disagreement among these signals.\n\nFor the Spider setting, the LabelModel is trained on the Spider training set. Regarding lines 180-183, the generative model/classifier referenced there is implemented using Snorkel LabelModel.\n\n[1] \"Snorkel: rapid training data creation with weak supervision.\"\n\n## On the clarity of key technical details\n\n> Line 209: \"Error Selector uses an LLM to prioritize which error to fix first...\" – How is this prioritization determined by the LLM in practice? By prompt or some other strategies?\n\nThe Error Selector is LLM-based. We prompt the LLM with the SQL query and its associated error report, and instruct it to choose which error should be corrected first. This prioritization is determined directly through the LLM’s response to the prompt.\n\n> Line 289: \"We conduct a microbenchmark on BIRD...\" – How is the microbenchmark constructed? Is it based on random or stratified sampling? How many examples are included, and how are the signal labels obtained?\n\nWe apologize for any confusion caused by our wording. By \"microbenchmark,\" we mean an in-depth evaluation of SQLens’s detailed performance, not a study based on subsampling. The results reported in Tables 4 and 5 are based on the entire BIRD dev set, without any random or stratified sampling.\n\n>How many examples are included, and how are the signal labels obtained?\n\nEach SQL query in the BIRD dev set is evaluated with our set of error signals, resulting in a vector of labels that indicate which signals are flagged for each query. For the experiments referenced in Line 289, we specifically assess the performance of each individual signal by applying only that signal to determine correctness and attempt correction—no aggregation of signals is used in these analyses.\n\nIf the reviewer’s question about \"examples\" refers to the content of LLM prompts: All LLM prompts in SQLens are zero-shot, with no in-context examples included. We find this zero-shot approach provides strong practical performance.\n\n## On the performance for DIN-SQL\n\n>The EX performance of DIN-SQL is inferior to vanilla in Table 1, which is somewhat counterintuitive.\n\nWe obtained these results by running the official DIN-SQL codebase, making only one modification: switching the backbone model to Claude 3.5-Sonnet. The observed performance drop may be due to the fact that DIN-SQL’s prompt was specifically tuned for GPT-4, while we used Claude 3.5-Sonnet instead. We will investigate this further in the full paper.\n\n>The error correction methods in the experiment show an improvement of a few percentage points on other text-to-SQL approaches, but alone on DIN-SQL, all methods have an extremely high increase (15-20%), which is a bit questionable.\n\nThis large boost is primarily because DIN-SQL starts from a much lower initial accuracy (39.49%). As mentioned above, the DIN-SQL prompting strategy—while effective on GPT-4—tends to reduce performance when used with Claude 3.5-Sonnet. This means the model itself is capable of generating correct SQL, but the prompt inhibits its accuracy. During the correction phase, since this limiting prompt is not present, the model is able to recover and achieve a much larger improvement, unlike other baselines, where the prompting strategy is already improving the model's default performance significantly."}]

Author rebuttal:

Meta-review: This paper presents SQLENS that is an end-to-end framework designed to detect and correct semantic errors in LLM-generated SQL queries. By combining signals from the database and LLM, it identifies clause-level errors, uses weak supervision to improve the detection, and selects error reports for LLM to revise the generated SQL statement. This stepwise correction approach, guided by error reports, achieves a 25.78% better F1-score and up to 20% higher execution accuracy on benchmarks like BIRD and Spider.

All reviewers consider this work is a practical improvement for LLM-based text-to-SQL tasks. Reviewers ask for more details and careful analysis of the results including case study and statistics on each error category listed. Especially, it would be great to have a failure case study on the cases that cannot be solved by SQLENS. Some reviewers have concerns on the risks of weak supervision method used in the paper as it is very easy to be biased to the data used in experiment, which may hurt the generalization of the technique.

As all reviewers give a positive score by recognizing its potential value in practice, I would suggest to accept this paper.
Decision: Accept (poster)