SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
Authors: Jinyang Li
Xiaolong Li
Ge Qu
Per Jacobsson
Bowen Qin
Binyuan Hui
Shuzheng Si
Nan Huo
Xiaohan Xu
Yue Zhang
Ziwei Tang
Yuanshuai Li
Florensia Widjaja
Xintong Zhu
Feige Zhou
Yongfeng Huang
Yannis Papakonstantinou
Fatma Ozcan
Chenhao Ma
Reynold Cheng
Venue: NeurIPS 2025 poster ()
Keywords: text-to-SQL
LLM
SQL issues
Primary area: applications

Abstract: Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging on SQL issues. In order to address this gap, we introduce **BIRD-CRITIC**, a new SQL issue debugging benchmark comprising 530 carefully curated PostgreSQL tasks (**BIRD-CRITIC-PG**) and 570 multi-dialect tasks (**BIRD-CRITIC-Multi**), which are distilled from authentic user issues and replayed within new environments to facilitate rigorous and contamination-free evaluation. Baseline evaluations on BIRD-CRITIC underscore the task's complexity, with the leading reasoning model **O3-Mini** achieving only 38.87% success rate on **BIRD-CRITIC-PG** and 33.33% on **BIRD-CRITIC-Multi**. Meanwhile, realizing open-source models for database tasks is crucial which can empower local development while safeguarding data privacy. Therefore, we present **Six-Gym** (**S**ql-f**IX**-Gym), a training environment for elevating the capabilities of open-source models specifically for SQL issue debugging. This environment leverages **SQL-Rewind** strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose *f*-Plan Boosting, which extracts high-level debugging plans automatically from SQL solutions, enabling the teacher LLMs to harvest and produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, **BIRD-Fixer**. Based on Qwen-2.5-Coder-14B, **BIRD-Fixer** raises its success rate to 38.11% on **BIRD-CRITIC-PG** and 29.65% on **BIRD-CRITIC-Multi**, surpassing many leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities for both research and industry.

SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
BBowen Qin Binyuan Hui Shuzheng Si a. Nan Huo 9:5 Xiaohan Xu Yue Zhang 0x eiiiieei Tang Yuanshuai Li
Jinyang Li 0. Xiaolong Li Ge&Ge Qu' @ Per Jacobsson
Florensia Widjaja Xiintong Zhu FFeige Zhou 5,5 Yongfnng Huang
Yannis Papakonstantinou Fatma Ozean 76 Chenhao Ma
a.s Reynold Cheng"
"HKU STAR Lab Google Cloud "CUHKSZ .CUHK
"THU <The BIRD Team
fj10725, xia0long, quge) @connect. hku. .kk
https://blrd-erriicc..ih
Abstract
Resolution of complex SQL issues persists as a significant bottleneck in real- world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL. translation, have not been rigorously evaluated on the more challenging task of debugging SQL issues. In order address this gap, we introduce BIRD-CRITIC, a new SQL issue debugging benchmark comprising 530 carefully curated PostgreSQL tasks (BIRD-CRITIC-PG and 570 multi-dialect tasks WHIRD-CRITIC-MULEE replayed within new environments to facilitate rigorous and contamination-Fie evaluation. Baseline evaluations on BIRD-CRITI underscore the task's 's complex- ity, with the leading reasoning model O3-MINI achieving only 38.87% success rate on realizing apen-source models for database tasks crucial which can empower local development while safeguarding data privacy. Therefore, we present SIX-GYM (SqI-fTX-Gry -Gmm), training environment for elevating the capabilities open-source models specifically for SQL. issue debugging. This environment leverages SQL- Rewind strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory- based fine-tuning methods do not explore substantial supervisory signals. We further propose f-Plan Boosting, which extracts high-level debugging plans auto- matically from SQL solutions, enabling the teacher LLMs to harvest and produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, BIRD-FINER. Based on Qwen-2.5-Coder 14B, BIRD- FIXER raises its success rate 38.11 13.11% % BIRD-CRRITIC-GG and 29.65% surpassing many leading proprietary models such as Claude-3. 3.7noeet and 1PT-4.1, marking significant step toward democratizing sophisticated SQL-debugging capabilities for both research and industry.
which are distilled from authentic user issues and
and 33.33% on BIRD-CRITIC-MER
Mcanwhile,
Introduction
Relational Databases (RDBs) serve as the bedrock for data storage and information retrieval across countless modern applications, ranging from financial systems to web services and scientific research
"Equal contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Issue
Solution SQL
hove canti see den
Output Formet
VITH RECURSIVE RODES AS
rep anns type of card with uriqu dentitor toge rumes andlarence The dan diartied vay cants
RECURSINT
Recarsive Join
SELECT
uuid,
Child Card Aas
card_nase AS
poieer treachee Neearity letsiyy rangee to crate recutivv quary dud.
gruped
WHERE sarent_uuld NULL
FROM card_type
I AS parents
LLMb
true (iuAEe chEldren
tel
AS card, type
emume each dlong with coreedly the lst parant wack sould priter to hace atrecsree ouppu where each For eompls word coneet the ne at all diectt child cark togeterr forning too lie staauur. would hel me itsurios sarrccyy oralies
formet daiis.
WEERE parent, uuit T5 NILL
UNION ALL
SELECT
EEXESISSSELECT FROM card_tyse
c.ouid, niee, FROM nedes.parents c.ouid
Databases
SPLECT root uuld uuid, rost arrmy wnn chi litren) AS chi Idren
10IN ON nodes.euis c.parent_ uoiid
FROM NOEES folse
canks with each parort cadd having an
aide
afhivv the
SERECT
HUN modax;
MyNta
GROUP BY reot uvid, root_naec ORDER BY uuot_uuid
foreromaton using SOUJ
Figure I: Illustration of the SQL issue debugging process in BIRD-CRITIC. It should start with a user issue query (left) and issue SQL query (center-left), L.LMs will produce corrected SQL solution (right) based on reasoning and interaction with the environment.
platforms [8, 34, 19, 35]. Structured Query Language (SQL), as the standard language for interacting with these systems, is thus a critical interface for data manipulation, querying, and administration [3, 2]. Despite its widespread adoption and apparent simplicity for basic operations, mastering SQL and troubleshooting complex queries or unexpected behaviors remains a significant challenge for users of all experience levels. The complexity of query semanties, diverse behaviors across different SQL operations (Create, Read, Update, Delete), evolving database features, and the need to understand underlying data schemas contribute to a steep learning curve and frequent user issues.
Resolving these SQL issues often demands considerable manual efforts, domain expertise, and time, representing a significant bottleneck in data-driven workflows and software development cycles 25, 12, 40, 13]. Support forums, Q& A sites, and internal helpdesks, such StackOverflow, are replete with user requests seeking assistance in debugging faulty queries, optimizing performance, or understanding why a query generates unexpected results. Therefore, automating this process holds huge value in improving productivity and reducing reliance on specialized human experts.
Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and code generation [6. 53, 7, 38, 50], notably achieving impressive results in converting natural language descriptions into SQL queries (text-to-SQL) [24, 45, 29, 22]. However, diagnosing and fixing existing incorrect or suboptimal SQL code presents more complex challenges. As shown in Figure 1, debugging such issues requires not only understanding the user's intent, often in a verbose and long-context description, but also analyzing the query logic underneath, identifying subtle errors, and intensively interacting with the database schema. Despite the practical importance of this task, the capabilities of current LLMs in SQL issue resolution have not been systematically investigated.
In this work, we are targeting to bridge this critical gap by two primary contributions. First, we present BIRD-CRITIC, a carefully curated benchmark built from authentic StackOverflow bug-fix fix threads. comes in two subsets: (1) BIRD-CRITIC-PG with 530 PostgreSQL-only tasks, and (2) BIRD- CRITIC-MULTI, whose 570 tasks are distributed across 4 major dialects: PostgreSQL and MySQL as open-source databases, SQL Server and Oracle as community-frient
cloud-based platforms with free developer editions. Each task undergoes rigorous reconstruction where the underlying knowledge structures and debugging heuristics are extracted, and the scenario is reproduced within a controlled sandbox environment by new RDBs and conditions. This process ensures that tasks remain relevant while minimizing potential exposure to pre-training data. Furthermore, execution accuracy (EX) in standard text-to-SQL is inadequate for the diverse types of issues in BIRD-CRITIC, frequently leading to false negatives. Specifically, tasks involving database state changes, i.e., via Data Manipulation Language (DML) or Data Definition Language operations (DDL), frequently permit multiple functionally equivalent solutions that may differ syntactically or include non-impacting elements [52, 4]; reliance on strict EX matching would incorrectly penalize such valid SQL solutions. Therefore, each task is augmented with custom evaluation scripts containing specific test cases designed to evaluate functional correctness, enabling precise calculation of task success rates. Our bascline evaluations on BIRD-CRITIC underscore the complexity of SQL issue debugging, in which

---

oivr lone Qaty
unee curta then the
Duoh
areer pordlas your aeyoree
Adaaud apai Loaat aoserd dide
withi resd catti
Decamens Achellen
Phonnel
Luttt BELL LEf
HEDS rosact meserts momer
2
Drop logs
[
Schacce Qhery
Solution Sais
with
Figure 2: Example task structure within the BIRD-CRITIC benchmark, demonstrating the transfor- mation from a user-reported issue and error SQL to a revised SQL solution.
even advanced reasoning models, 03-mini, only achieves a 38.87% success rate BIRD-CRITIC- PG and 33.33% on BIRD-CRITTC-MUUTT.
Second, inspired by prior work on code generation environments [28], we propose SIX-GYM (YL--FIX-GYM), a training environment designed to enhance the SQL. debugging capabilities of open-source models. A core innovation within SIX-GYM is the SQL-Rewind strategy, an automated methodology for generating large-scale, executable issue-solution datasets. This strategy operates by taking verified, correct SQL queries and systematically introducing plausible errors, effectively engineeengin realistic debugging scenarios. A common practice [28, 16] of such environments involves using advanced teacher LLM to generate successful task execution trajectories for fine- tuning student smaller models. However, we find that this approach underutilizes the guidance available from ground-truth reference solutions, potentially limiting the quantity and diversity of effective training trajectories. To address this, we introduce the Functional Plan (f-plan) Boosting strategy. This method first infers the underlying debugging logic by comparing the problematic SQL and the correct solution, representing this logic as a step-by-step pseudo-functional Afterwards, guided by this f-plan, a teacher LLM employs our designed agent scaffold, SQL-ACT, to execute the debugging task within the environment. This plan-guided approach generates a significant 73.7% % increase in more successful trajectories, providing richer data for fine-tuning open-source models, particularly smaller ones, to effectively interact with the database environment and debug complex SQL issues. The agent fine-tuned using this f-plan boosted data is termed BIRD-FIXER.
functional code plan.
Our experiments demonstrate that BIRD-FIXIE significantly enhances the performance of open- source models from various families. Notably, BIRD-FIXER fine-tuned 2wwn-2.5-ccan 5-Coder-14B achieves a 38.11% 38% Success Rate (SR) on MULTI, surpassing the performance of the highly capable models such as Claude-3.7-Somin and G.T-4.1. This result marks a significant advancement towards democratizing sophisticated SQL debugging capabilities for both research and practical industry applications.
and 29.65% on BIRD-CRITIC-
2 Problem Definition
In this paper, we introduce a more complex but realistic task of SQL issue resolution. This task starts with user- psoviprovided issue SQL query Tissu, a natural language problem description P detailing the issue and intent, and the database schema S. The goal is to generate revised SQL query (opoaa that corrects the fault while preserving the user's intent. This mapping is:
Opeed = fo SP,SSacm d.sue).
The desired output pped must satisfy the user's underlying intentions as inferred from the triplet (P. S, d. In BIRD-CRITIC, we annotated referenced ground-truth solution SQLs as a² and develop tailored evaluation scripts (detailed in Section 3) for each task, enabling precise evaluation of the functional correctness of predicted solution SQLs.
3

---

Grouping ng and (
Table 1: Data Characteristics
Recursive querles (74)
Window functions (284)
STATISTIC
PG
MULTT
JSON operations
Total Issue
530
570
query- like
291
Date and time operations (107)
of management iscues
# of personalization isenes
151
Subqueries (225)
Appregation functions (276)
user length (mesn/max)
162 162.98/1046
165. 75/105
SQL length (mean/max
133, 238.29/1222
125.86/1254
SQL longth (mean/max))
112.64/853
117.46/859
Figure 3: Distribution of issue categories in all BIRD- CRITIC, derived from an analysis of SQL usage in the real-world database applications. A detailed distribution in Appendix E.
distinct test cases
365
317
d of prepruceess SQLs a of clean SQU
643
571
287
262
agreement
3 BIRD-CRITIC Benchmark
Annotator Group. BIRD-CRITIC is developed via multi-stage annotation converting raw user issues into executable, verifiable tasks. This involves two annotation groups: 1) 10 qualified database/SQL annotators, who pass strict entry test detailed in Appendix B.1 and systematic training shown in Appendix B.2 to promise the quality of annotation; 2) 3 senior database ex- perts/scientists for final data collection decision. This process is visually outlined in Figure 2.
Environment Setup. We leverage relational databases from the BIRD-SQL development set [24] chosen for its domain diversity across real data-science tasks (California Schools, Financial, Superheroes) and its permissive license. We migrate their original SQLite schemas to PostgreSQI. MySQL, SQL Server, and Oracle, four widely used production-gres -ariiinnggreeeeeee dialects. During migration, go beyond direct dialect translation by refining table and column names. We adjust data types and introduce guarded alterations to schema components to reduce potential information leakage (see Appendix A.2) To pair these databases with realistic debugging scenarios, we collect SQL issue queries from Stack Overflow, following a strict protocol shown in Appendix A.3.
Issue Reproduction. Following the initial collection of candidate issues, we start reproducing them in our environment in following produces illustrated in Figure (1) Distilling Intent and Error: Precisely identifying the user's underlying goal and the specific reason of the issue exhibited by Fisuee The core reason of the issue documented. (2) Schema Mapping: Assigning the issue to one of the adapted BIRD-SQL database schemas (S) that provides suitable context for the problem. (3) Reproducibility Verification: We adapt and execute oismee against the chosen database, verifying through execution logs that the error appears as expected. This entire process transforms potentially ambiguous web forum post into standardized, reproducible problem instance (P,S, Fissue) ready for solution annotation.
Solution SQL & Evaluation Script Annotation. Annotators carefully review the reproduced issue and craft user's objective as inferred from P and the context of disue- Also, to ensure robust evaluation, each task is annotated with evaluation scripts consisting of specific test cases written by Python and SQLs, Details can be found in Appendix C. We report the Success Rate (SR %), considering a task solved only when peed successfully passes all test cases in its evaluation script.
σ". This annotation requires ensuring that a accurately fulfill the
Validation. After annotation, BIRD-CRITIC undergoes cross-validation, with annotators exchang- ing data for review. This verification involves three steps: (1) enhancing test case functions with additional test cases for robust SQL code validation; (2) red teaming the SQL by introducing errors make sure evaluation scripts can flag these errors. (3) Annotators first attempt to resolve disagree- ments through discussion. Persistent issues are escalated to the expert team for final determination, which may involve modification or rejection of the disputed annotation.
Benchmark Statistics. Table 1 summarizes the key properties of the BIRD-CRITIC benchmark, and Figure visualizes the distribution of underlying knowledge categories. The distribution of benchmark, is detailed in Appendix E. A side-by-side comparison with standard text-to-SQL. benchmarks (Table in Appendix E. exposes three distinctive challenges introduced by BIRD- CRITIC: non-query-lic problems, multi- complexities, and the most verbose but authentic user

---

queries. As far as we know, BIRD-CRITIC is the first debugging benchmark for SQL applications. These aspects establish HIRD-CRITIC as crucial benchmark for rigorously evaluating LLM proficiency in solving authentic SQL issues.
4 SIX-GYM: An Automated SQL Debugging Environment for LLMs
This section introduces SIx- GYM, a dedicated training environment for enhancing the SQL debugging capabilities of LLMs. This environment is built upon SQL-Rewind, which is responsible for the automated generation comprehensive suite of SQL issue instances.
Overview. GYM-like datasets have proven effective for training LLMs as agents for complex tasks [28]. However, manually collecting and annotating these datasets is labor-intensive and difficult to scale, especially for debugging tasks. Thus, we introduce SQL-Rewind, which addresses this by inverting the debugging paradigm: starting with correct SQL. queries (a*) and systematically introducing realistic issues to generate issue SQLs (Øssue) and user issue query P. This approach enables efficient creation large-scale training data without human annotation. The pscudo-algorithm is shown in Appendix H.I.
Solution SQL Collection. We begin with raw StackOverflow issue data and enforce two principles against data overlap: (i) any issue used construct BIRD-CRITIC tasks is excluded from SIX-GYM, and (ii) SQL-Rewind operates only on the 12 databases in the training databases of BIRD-SQL, while BIRD-CRITI evaluation confined to databases drawn solely from the BIRD-SQL dev set. We mine new candidate SQLs via rulc-based regular expressions, then leverage Gemini-2 O-Flash to align table and column references to 12 databases SIX-GYM, while preserving the original SQL's logical structure. To validate these adapted SQL queries as ground truth solutions, each was executed against its target database; only those queries that completed without error and yielded a non-null result were accepted into our final corpus of solution SQLs (a").
Synthetic Issue Generation and Automated Verification. We employ Gemini Demmini-22 22 0-Flash to automate the entire process of issue reproduction and verification. Initially, the model summarizes issue reasons (Tissue) and modifies solution SQL (a") ) to create issue SQL (Tissue) guided by I' rissue- Concurrently, it generates evaluation scripts T comprising test cases designed to be passed by solution SQLs but failed by issue SQLs. The model then automatically validates whether the logic triplet (Sissue Tissue o²) is coherent and whether the evaluation script accurately identifies errors while allowing solution SQLs to pass. This validation process undergoes 3 iterative refinements; the components are deemed compatible, the data is added to our collection.
User Issue Query Generation. Finally, we employ Gemini- 10aiini-2.0FFFF. -.FFFssh again to simulate a realistic user issue description P. The generated P includes the user intent, issue description, and requirements. Each P must be logically consistent with (S, Tissue, T,, a²). It undergoes up to 3 rounds of optimization by the model to reduce hallucinations. The resulting tuples are collected as final data. Using this SQL-Rewind strategy, we successfully generate approximately 3,301 high-quality synthetic data instances, forming training environment we term SIX-GyM.
BIRD-FIXER: Elevating Open-Source LLMs to an SQL Issue Fixer
5.1 Agent Scaffold: SQL-AcT
ReAct [43] interleaves internal reasoning (thoughts 4)), external actions (a:), and observations (o:). and has proved highly effective for state-of-the-art code agents [28, 38, 39]. Building upon this paradigm, we introduce SQL-ACT, a specialized agent scaffold tailored for SQL tasks, particularly targeting challenges presented in benchmarks like BIRD-CRITIC. Unlike tool-based agents whose action space is restricted to a finite, hand-crafted set of operations, SQL-ACT treats arbitrary SQL commands as actions, dramatically enlarging the space of possible manipulations and enabling richer, more flexible debugging strategies.
At each step the agent emits tuple (ta, o, o.). where o; is the SQL statement executed at step i. The complete exccution trajectory is therefore 01,), (t2, 02, o2),... (In, on 0n)). As
5

---

Table 2: Success Rate (SR %) of different models GIRD-CRITIC-PG and MULTI, grouped by each issue and dialect categories. Bold numbers indicate the highest score in each column, and underlined numbers indicate the second highest. "Quer." = query-like issues, "Mana." data-management issues, "Pers." personalized-funetino issues. "PG." PostgreSQL, "My." MySQL. "Server"= SQL-Server.
BIRD-CRITIC-PG
BIRD-CRITIC-MULT
Model
Quer.
Mana.
Pers.
Overall
PG.
My.
Server
Oracle
Overall
General- Models
Meta-Llama-3.1-88
18.21
22.73
11.26
16.98
13.04
13.27
21.43
3.06
12.81
Phi-4
30.24
37.50
25.83
30.19
25.72
27.55
23.47
8.16
22.63
Deepseek-V3
25.09
35.23
28.48
27.74
27.17
26.53
21.43
14.29
23.86
Gemini-2..0-Hlash
27.84
44.32
29.14
30.94
27.54
22.45
31.63
7.14
23.86
Meta-Llamma-33-700
27.84
32.95
27.81
28.68
26.81
22.45
28.57
14.29
24.21
Owen2.5-Coder-33
31.62
38.64
24.50
30.75
28.26
24.49
30.61
9.18
24.74
Claude-3.7-Sonne
27.15
43.18
35.10
32.08
32.61
30.61
21.43
18.37
27.89
GPT-4.1
31.27
55.68
38.41
37.36
36.23
28.57
29.59
9.18
29.12
Reasoning Models
Gemini-2 2.0-ni--200PP- Flash-
27.15
53.41
33.11
33.21
28.99
35.71
37.76
19.39
30.00
Claude-3.7 Sonnet-Thinking
29.55
45.45
35.76
33.96
35.51
31.63
27.55
15.31
30.00
01 Pre-iiiwww222-009
29.90
53.41
37.09
35.85
40.94
33.67
33.67
11.22
33,33
03-Mini-2025-01--1
32.30
57.95
40.40
38.87
41.30
26.53
32.65
18.37
33.33
shown in Section 6.2, SQL-ACT is not only simpler to implement than TOOL-ACT but also delivers consistently higher accuracy in SQL issues solutions.
5.2 Trajectory Collection and Agent Fine- Tine-Tuning
f-Plan Boosting. The standard "gym-style" practice involves a strong teacher LLM on the environ- ment and logs only those trajectories that reach the reference solution. In our experiments, running Gemini- nin2 0- Flash with SQL-ACT on SIX-GYM produces just 1,254 successful trajectories, which just utilizes 38.0% of the data.
To augment successful trajectories, we introduceffPPie Boosting, a two-phase seff-ditiillation loop:
(1) Backward inference. Given the problem (P,S. Jissue) and its corrected query a², the teacher annotates step-by sep-by-stee symbolic functional plan F (fi..... where each f; represents an abstract debugging operation that maps Jissue toward a². Since such plan contains few tokens yet exhibits more structured format, is especially amenable execution by LLMs [6, 18].
Using only the context (P, S, Fisme) and the candidate plan F, the teacher LLM regenerates solution by SQL-ACT. The plan is accepted iff the regenerated solution SQL passes every test cases in T, producing reliable pair <(P, S, Tissue) F). After rollout we discard F and retain only the executable trace r' = ((t1, (111,0.00 01. 01),................... (tn, ou 0n))-
(2) Forward validation.
single pass of f-Plan Boosting produces total 2,178 successful trajectories, increase of 73.7% % over the vanilla collection pipeline, which we then use to fine-tune the open-source models via Low-Rank Adaptation (LoRA) [14].
Generative Thought Mode (GTM).. The generalization of the agent can degrade when it predicts thoughts and actions jointly, because the model tends to overfit to the SQL patterns seen during fine-tuning. To counter this problem, we introduce a Generative Thought Mode (GTM), which explicitly decouples the two predictions, akin how Skip-gram gram in Word2Vec separates target and context words [26]. Let Mo be the fine-tuned model, MB the original base model, and H;-1 = ((1, σi, .i),..,,.................. ....,, (4--1, -; ,;-1, the interaction history. During the inference step i, the fine- tuned model first proposes thought-action pair (4,, ) (oHH;-)), from which only the thought extracted. The SQL. action is then generated by the base model, a; W 4.), leveraging
its wide-coverage knowledge of diverse SQL dialects. GTM preserves the specialized debugging

---

SQLACT
ToolACT
Base Model
Success Rate (SR %)
45.47
43.96
35.66 36.04
37.36
38.87
32.08
30.94
daude-3.7-somee Semini-220-fiase
gpt-4.1
03-mini
Figure 4: LLM agent performance for SIRD-CRITIC-PG. ToOACT employs constrained toolkit as actions, while SQLACT executes SQLs as actions.
logic learned by Mo. fully taking advantage of generative features of auto-regressive models [49], while mitigating overfitting of SQL patterns during training.
Experiments
6.1 SetUp
Models. We evaluate the performance several popular and strong LLMs across two primary categories, including general-purpsse models: Gemini-2. 0-2lash.. GPT-4.1, Claude-3. 7-Sonnet, Qwen-2. 5-Coder-32B. Meta-LLama-3.14 3.1-88 Meta-LLlma-333-1 Phi-4 and DeepSeek-V3. The second category consists of models specifically renowned for their advanced reasoning capabilities: 03 03-mini, 01 01-preview, Gemini-2, O-Flash-Thi ing, and Claude-33 7----nnnettThinkiin The implemen- tation details are in Appendix G.2.
Advanced Agentic Methods. Agentic workflows have shown considerable promise for addressing complex tasks. Accordingly, we also benchmark LLM agent performance on BIRD-CRITIC. Broadly, agentic systems be classified into two main categories based on their action types. The first, which we term TOOL-ACT, involves agents employing pre-defined tools tailored to specific tasks. We implement Tool-Act guided by SOTA agents Spider-Agent [21] and InterCode [42] in SQL tasks. The second category, CODE-ACT [38], allows for more flexible, free-form actions where LLMs generate code perform operations. In the context of this research, implement specific variant called SQL-ACT, where the LLMs generate SQL queries as their actions as introduced in Section 5.1.
6.2 Main Results
Baseline Results. An evaluation of mainstream Large Language Models (LLMs) on BIRD-CRITIC detailed Table 2. We can observe that:
(1) Superior Performance of Reasoning-Orient
Models. A clear performance advantage is evident for reasoning-orienee oriented LLMs. These models surpass general-pure purposs counterparts by an average Success Rate (SR) of 6.13 % on PostgreSQL issues and 8.03 % on multi-dialect issues. This disparity underscores the computationally intensive, reasoning-driven nature of SQL-issue debugging, task that demonstrably benefits from models capable of intermediate inferential steps.
(2) Persistent Challenge Posed by SQL Issue Debugging, Despite ongoing advancements LLM capabilities, BIRD-CRITIC continue to present a considerable challenge. The top-performing model, 03 Mini-2022 01131, achieves overall SR of only 38.5 38.87% on PostgreSQL. issues and 33.33% on multi-diales issues, leaving large head-room room for future research.
(3) Heterogeneous Difficulty Across Issue Categories.
An analysis performance across distinct SQL issue categories reveals clear differences in dif- ficulty. Issues related to data management, such DML. operations: insertions, deletions, up- dates, and DDL operations like schema modifications, are found to relatively more manage- able. On average, reasoning models achieved a 52.6% SR and general-purposs models 38.8%

---

Table Detailed comparison of BIRD-FIXER with other strong baselines on and BIRD- BRRD-CRITIC-MULT. A shows relative improvement of BIRD-FIXER compared to base model.
BIRD-CRITIC-PG (SR %,
BIRD-CRITIC-MULT (SR %, f)
Model
Base
SQL-ACT BIRD-FIXER A
Base
SQL-ACT BIRD-FIXER A(%)
Llama-3.1-8B
16.98
16.42
24.34
+ +43.34
12.81
13.64
18.25
+42.466
Oiwen2.5--oode--
23.40
26.60
31.32
+33.84
17.89
17.19
21.58
+20.58
Qwen-2.5-Coder--ee
31.32
31.13
38.11
+21.68
24.04
23.33
29.65
+23.36
Phi-4
30.19
29.43
38.11
+26.23
22.63
19.80
27.89
+20.58
SR in data management. Issues associated with Personalized functions also demonstrate mod- erate success rates. In contrast, Query-like issues present the greatest challenge for all LLMs. These issues require understanding of logi- cal flaws within complex SELECT statements, particularly those involving joins, subqueries, aggregations, and conditional filtering. Unlike more standardized data management operations, SELECT queries exhibit remarkable diversity in their logic, structure, and intent, mostly re- flected by the wide variety of underlying busi- ness requirements they serve, making their error patterns significantly harder to predict and cor- rect. As evidenced Figure Query-like issues contain the most diverse functions, leading to the lowest performance of both general-purpose and reasoning models, which presents strong negative correlation.
Success Diversity Category)
$2.0%
26.6%
Succeen Bate
Quiry-Stt
Figure 5: Success Rate Query Diversity (by Issue Category). It shows a strong negative correlation (r -0.89) between n-gram of tokens and model performance after normalization.
(4) Dialect-Specific Performance Variations. Model effectiveness exhibits notable depen- dency on the specific SQL dialect, as observed within the BHRD-CRITIC-MULTI Specifically, Gemini-2 O--lash-Thinkiine iiniiiii demonstrates the lower performance on PostgreSQL with 28.99% SR. contrast, becomes the most proficient for SQL Server (37.76% SR), with clear margin over other evaluated models in that dialect. Such variations are plausibly attributable to differential distributions of SQL dialects within the respective training corpora of these models, suggesting that the composition of training data significantly influences dialect-specifi debugging capabilitics.
(5) Agentic Workflow Performance. Figure 4 compares the performance of different ILM-based agents on BIRD- BRDD-CRITCC-PG The results show that agentic workflows markedly boost LLM accuracy on issue debugging tasks, which benefits from iterative interaction with its environment. Additionally, the SQL-ACT agent mostly outperforms the TooL-ACT agent, suggesting that the richer, more flexible action space offered by SQL-ACT better equips LLMs to address the diverse and uncertain challenges encountered during debugging.
6.3 Performance Analysis of BIRD-FIXER
Overall Performance of BIRD-FIXER. Table 3 reports the performance gains achieved by BIRD- FIXER across three model families: Llama, Qwen, and Phi, which range from roughly 7B 14B parameters. For each model, BIRD-FIXER delivers substantial improvements, demonstrating that the benefits of SQL-ACT + f-plan and SIx-GyM architecture-agnosicc and scalable. The table also exposes limitation of small language models (SLMs) in agentic workflow only by inference: on several models, agent performance actually declines, suggesting that long, complicated interaction histories can overwhelm SLMs. By contrast, our methods equip these compact models with richer interaction capabilities, enabling them to navigate complex environments more effectively. This benefit is especially valuable for privacy-sensitive SQL workloads: running 7-14B parameter agent locally avoids any exposure of proprietary data to cloud services. Notably, BIRD-FIXER based on 14B base models, e.g., Qwen-2 5-Coder- 4B, BIRD-FIXER presents competitive performance to

---

Table Trajectory Generation Efficiency Comparison. Baseline: Standard SQL-ACT rollout with a single attempt (temperature=O). f-Plan (Ours): A single rollout guided by functional plans extracted from issue-solution pairs (temperature=0) Rejection Sampling: Up to trials per instance (temperature=0.8)) with early stopping when successful trajectory is obtained. Reject + + Combination of rejection sampling (up trials) with f-Plan guidance.
Method
Max Tries
Successful Traj.
Avg Tries
DB Time
Cost ($)
(min)
Baseline
1
1,254
1.0
306
8.47
f-Plan
2,178
1.0
324
27.44
Rejection Sampling
5
1,910
4.2
1,377
108.05
Reject + f-Plan
5
2,560
1.7
810
41.16
03-mini and outperforms the Claude-3, 7-Sonnet agent on BIRD-CRITIC-PG, suggesting a promising path toward this goal of privacy while keeping effectiveness.
Generalization to Multi-Dialect SQL Issue Debugging. Although BIRD-FIXER is fine-tuned only on PostgreSQL trajectories within SIx-GYM it generalizes robustly other SQL dialects, as evidenced by the multi-dialect results in Table 3. That is because GTM elicits each model to produce reusable debugging strategy trained SIX-GyM while keeping pretrained knowledge dialect variation. In conclusion, BIRD-FIXER exhibits strong cross-dialect reasoning without any extra data collection or further training, underscoring its practicality for heterogeneous database stacks.
6.4 Trajectory Sampling Comparison
To better illustrate the efficiency and effectiveness of f-Plan Boosting, we compare it against widely used trajectory augmentation approaches. We evaluate four strategies for trajectory generation, using Gemini- 2 O-F Flash as the teacher model on SIX-GYM.
As shown in Table 4, f-Pla Boosting yields 73.7% more successful trajectories than the baseline while maintaining similar runtime and overhead. By contrast, rejection sampling increases success rates modestly but at the cost of 4.2x more attempts and 4.5x longer execution time. When combined, rejection sampling and a f-Plan achieve the best overall trade-off, generating 2,560 trajectories with reduced average attempts (1.7) and 62% reduction in cost relative rejection sampling alone. These results demonstrate f-Plan provides an effective and efficient approach to trajectory augmentation during rollout in complex environments. Other detailed comparison be found in Appendix F.3.
6.5
Ablation Study of BIRD-FIXER
Figure shows the ablation study BIRD-FIXER, highlighting:
dubitiion day STRD POEEE
GTM (Generative Thought Mode): Removing GTM causes the fine-tuned model Mo predict both thought and SQL action directly. The performance drop to 33.33% indicates that GTM effectively leverages the base model MB for SQL generation guided by Mo's 's thought, mitigating overfitting to SQL patterns and better utilizing MB broad SQL. knowledge.
f-Plan Boosting: Using only trajectories from the vanilla collection pipeline reduces performance to 32.45% in BIRD- CRITIC-PG, highlighting f f-Plan Boosting's 's importance in generating diverse, high-quality training trajectories crucial for complex reasoning tasks.
Figure 6: Ablation study of compo- nents in BIRD-FIXER.
6.6 Error Analysis
To understand how far current L.LM-based agents still are from fully resolving user-reported SQL issues, we sample 100 failed tasks from 4.1, Claude-3.7 3.7-eee,,-S and BIRD-FIXER. can concluded that current agents exhibit four distinct error modes reflecting different levels of reasoning deficiency: Projection Mismatch errors
by 4 agents based on: O3-mini, GPT-

---

(26.9%), where models misinterpret output requirements by, for instance, adding unexpected columns or misapplying aggregations, suggesting limitations in semantic understanding of user intent and schema alignment; Chain of Errors (27.3%), characterized by cascading failures due to partial problem resolution that overlooks dependent issues such as sequence updates accompanying primary key modifications, revealing difficulties in multi-step causal reasoning and consistency maintenance; The database engine only reports the most superficial issue, masking a deeper, dependent error that is the true root cause. For instance, type mismatch error might be reported, but the underlying problem could be an incorrect join that brought together the wrong columns in the first place. Incorrect Logic (44.5%), the most prevalent, highlighting fundamental misunderstandings of data structures or transformation methodologies, particularly in complex operations like JSON array manipulation, leading to syntactically plausible but semantically flawed SQL; and Syntax Errors (29.3%), indicating technical implementation flaws such as type mismatches (e.g., DATE versus TIMESTAMP) or improperly formatted intervals, especially in specialized SQL contexts like recursive queries. The detailed examples for each category are in Figure 8. These findings highlight that future improvements should emphasize logical and schema-aware reasoning, cross-step dependency tracking, and dialect-robust SQL generation rather than mere syntactic refinement.
7 Related Work
Large Language Models for Text-to-SQL. The automated conversion of natural language queries into Structured Query Language (SQL), known as Text-to-SQL. has garnered significant attention
due to its practical utility in the era of big data [47, 41, 31. 15]. The advent of LLMs has notably advanced the capabilities in this domain. For instance, DIN-SQL [29], DAIL-SQL [9]. TA-SQL [32], and Chase-SQL [30] have demonstrated SOTA performance on standard benchmarks like Spider [45] and BIRD [24], primarily by leveraging in-context learning with powerful foundation models like
GPT-4. Also Supervised fine-tuning can fuel smaller LLMs towards stronger text-to-SOC totttooLLLno parsers as evidenced by XiYanSQL[10), Ar8, OmniSQL [23], CodeS [22] , and SHARE [33]. Beyond direct generation, agentic workflows such as MAC-SQL [37], InterCode [42], which empowers LLMs to interact with database environments and gather contextual information, are pushing the boundaries of LLM cognition in handling complex and previously unseen databases. Concurrently, the field is evolving towards addressing more sophisticated, rellsany Text-to-SQL challenges. Initiatives like Beaver [5] and the Spider 2.0 [21] signify a shift from end-user focused queries to tasks requiring deeper BI knowledge and handling of larger schemas. This progression naturally leads to a critical, but underexplored, question: Can LLMS effectively diagnose and resolve issues within existing, psov-drovided SQL queries?
LLMS for Program Repair. Program repair provides a complementary lens through which to evaluate and enhance the reasoning abilities of LLMs. At the function level, DEBUGBENCH [36] offers a multi-language suite that stresses fundamental programming logic. Repository-scale efforts such SWE-BENCH [17] move closer realistic software engineering, while follow-up studies, including SWE-LANCE [27] and MULTI-SWE [46] highlight the limitations of even sophisticated LLM-driven agents on complex, multi-language projects (c.g. Python, Java). Despite this rapid progress in general-purpose code fixing, SQL-specific debugging remains largely unexplored. even though databases are the backbone of most data-centric applications. To the best of our knowledge, our work is the first to formally cast SQL issue repair as a benchmark task, and to propose methods that adapt and augment open-source LLMs for automated SQL debugging.
Conclusion
We introduced BIRD-CRITIC, the first benchmark for SQL issue debugging tasks. Experiments show that SOTA LLMs solve fewer than 40% SR, underscoring the challenge. We also create SIX- an automated training environment which can produce thousands of high-quality agent trajectories without human annotation. Built on top of these trajectories, we proposed SQL-Aet, a lightweight agent scaffold, and applied trajectory-level augmentation (f-plan) fine-tune open-source LLMs, leading to the Bird-Fixer. Despite using only 7-14 B parameter backbones, BIRD- -IXER outperforms larger proprietary models and generalizes across four SQL dialects without additional training. Our research charts a path toward robust, real-world SQL issue debugging assistants.
10

---

Acknowledgments
We thank the anonymous reviewers and committees for their helpful comments, suggestions and organizations. We thank John Yang for early discussion and suggestions. Reynold Cheng, Jinyang Li, Xiaolong Li, Ge Qu, Nan Huo, Xiaohan Xu, and Ziwei Tang were supported by the Research Grant Council of Hong Kong (RGC Project HKU 17202325). the University of Hong Kong (Project 2409100399), and the HKU Faculty Exchange Award 2024 (Faculty Engineering).
References
[1] Alireza Ahadi, Julia Prior, Vahid Behbood, and Raymond Lister. Students' syntactic mistakes in writing seven different types of sql queries and its application to predicting students' success. In Proceedings of the 47th ACM Technical Symposium on Compating Science Education, pages 401-406. ACM, 2016.
[2] Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, and Matei Zaharia. Spark SQL: relational data processing in spark. In Timos K. Sellis, Susan B. Davidson, and Zachary G. Ives, editors, Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, May 31 June 4, 2015, pages 1383-1394. ACM, 2015.
[3] Donald D. Chamberlin and Raymond F. Boyce. SEQUEL: structured english query language. In Proceedings of the 1974 ACM SIGMOD Workshop Data Description, Access and Control, pages 249-264, ACM, 1974.
[4] Bikash Chandra, Ananyo Banerjee, Udbhas Hazra, Mathew Joseph, and S. Sudarshan. Auto- mated grading of sql queries. In 35th IEEE International Conference on Data Engineering (ICDE), pages 1630-1633. IEEE, 2019.
[5] Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çagatay Demiralp, and Michael Stonebraker. Beaver: an enterprise benchmark for text-to-sql. arXiv preprint arXiv:2409.02038. 2024.
[6] Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[7] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, In The Twelfth International Conference on Learning Representations, 2024.
[8] C. J. Date. An Introduction to Database Systems. Addison-Wesley Pearson, 8th edition, 2003.
[9] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint vrXiv:2308.15300 2023.
[10] Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, and Yu Li. A preview of xiyan-sql: A multi-generator ensemble framework for text-to-sql. arXiv preprint arXiv:2411.0885 2024. URL https: http/aaxiivvii.ae/eaa wtt//bs/////ooooooooo.../o...eee 2 2211. 08599.
[11) Zhipeng Gao, Xin Xia, David Lo, John C. Grundy, Xindong Zhang, and Zhenchang Xing. I know what you are searching for: Code snippet recommendation from stack overflow posts. ACM Trans. Softw. Eng. Methodol., (2(3):80:1-80:4) 2023.
[12] Sneha Gathani, Peter Lim, and Leilani Battle. Debugging database queries: A survey of tools, techniques, and users. In CHI '20: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, April 25-30, 2020, pages 1-16. ACM, 2020.
[13) Sabaat Haroon, Chris Brown, and Muhammad Ali Gulzar. Desql: Interactive debugging of SQL. in data-intensive scalable computing. Proc. ACM Softw: Eng., 1(FSE):767-788 2024.
11

---

[14] Edward J. Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhie Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[15] Nan Huo, Jinyang Li, Bowen Qin, Ge Qu Xiaolong Li, Xiaodong Li, Chenhao Ma, and Reynold Cheng. Micro-act: Mitigate knowledge conflict in question answering via actionable self-reasoning. arXiv preprint arXiv:2506. 05278, 2025.
[16] Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and lon Stoica. R2e- gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. arXiv preprint arXiv:2504.07114, 2025.
[17] Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
[18] Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, and Graham Neubig. Program-aided reasoners (better) know what they know. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[19] Sean Kandel, Andreas Paepcke, Joseph M. Hellerstein, and Jeffrey Heer. Enterprise data analysis and visualization: An interview study. IEEE Trans. Vis. Compul. Graph. 18(12): 2917-2926, 2012.
[20] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang. Ruiqi Zhong, Luke Zettlemoyer, Wen- Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. DS- 1000: A natural and reliable benchmark for data science code generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 Proceedings of Machine Learning Research, pages 18319-18345. PMLR, 2023.
[21] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on rerl-world enterprise text-to-SQL. workflows. In The Thirteenth International Conference on Learning Representations, 2025.
[22] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data, 2(3):1-28. 2024.
[23] Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang. Tieying Zhang, Jianjun Chen, Rui Shi, et al. Omnisql: Synthesizing high-quality text-to-sql data at scale. arXiv preprint orxiv:2503: 203:::2503.02240, 2025.
[24] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve database interface? big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024.
[25] Floris Miedema, Janet Spacco, and Raymond Lister. Patterns of SQL mistakes among novice programmers: An exploratory study. In Proc. 26th ACM Conf. on Innovation and Technology in Computer Science Education (ITiCSE), pages 55-61. ACM, 2021. doi: 10.1145/345656.. 3456622.
[26] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances neural information processing systems, 26, 2013.
12

---

[27] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier Ilms earn $1 million from real-world freelance software engineering? arXiv preprini arXiv:2502. 12, 2025.
[28] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024. URL https: //rxivv org/ abs//bb//////// 20np/24422 2412.2113
[29] Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36: 36339-36348, 2023.
[30] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gau- rav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik. CHASE- SQL: Multi-path reasoning and preference optimized candidate selection text-to-SQL. In The Thirteenth International Conference on Learning Representations, 2025. URL
https: / openreview net/ wtt////ormmmmieeeeee www.rrmiiicvvggggoo.......
[31] Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, Fei Huang, and Yongbin Li. A survey text-to-sql parsing: Concepts, methods, and future directions. arXiv:2208. 13629, 2022.
[32] Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. Before generation, align it! novel and effective strategy for mitigating hallucinations in text-to-SQL generation. In Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics, August 2024.
[33] Ge Qu, Jinyang Li, Bowen Qin, Xiaolong Li, Nan Huo, Chenhao Ma, and Reynold Cheng. SHARE: An SLM-based hierarchical action CorREction assistant for text-to-SQL. Association for Computational Linguistics, 2025.
[34] Margo Seltzer, Keith Bostic, Michael Stonebraker, and Joseph M. Hellerstein, editors. Read- ings in Database Systems, 4th Edition. MIT Press, Cambridge, MA, 2005,
[35] Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Anthony, Hao Liu, and Raghotham Murthy. Hive a petabyte scale data warehouse using hadoop. In Proceedings of the 26th International Conference on Data Engineering, ICDE 2010, March 1-6, 2010, Long Beach, California, USA, pages 996-1005. IEEE Computer Society, 2010.
[36] Runchu Tian, Yining Yc, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, and Maosong Sun. DebugBench: Evaluating debugging capability of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1173-4198, aug 2024.
[37] Bing Wang. Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wer Wen Zhang, Yin, Xing Sun, et al. Mac-sql: multi-agent collaborative framework for text-to-sql. In Proceedings of the 31st International Conference Computational Linguistics, pages 540-557, 2025.
[38] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better IIm agents. In ICMI., 2024.
[39] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui. Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for Al software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Zuozhi Wang. Avinash Kumar, Shengquan Ni, and Chen Li. Demonstration of interactive runtime debugging of distributed dataflows in texera. Proc. VLDB Endow, 13(12):253-22 -956. 2020.
13

---

[41] Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(0OPSLA):
1 1-26, 2017.
[42] John Yang, Akshara Prabhakar, Karthik R Narasimhan, and Shunyu Yao. Intercode: Standardiz- ing and benchmarking interactive coding with execution feedback. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[43] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[44] Ziyu Yao, Daniel S. Weld, Wei-Peng Chen, and Huan Sun. Staqe: A systematically mined question-code dataset from stack overflow. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1693- 1703. ACM, 2018.
[45] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross- semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, page 3911-3921, 2018.
[46] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, al. Multi-swe-bench: multilingual benchmark for issue resolving. arXiv preprini 2504v:250..02655. 2025.
[47] John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Conference on Innovative Applications of Artificial Intelligence, pages 1050- 1055, 1996.
[48] Bohan Zhai, Canwen Xu, Yuxiong He, and Zhewei Yao. Excot: Optimizing reasoning for text-to-sql with execution feedback. arXiv preprint arXiv:2503. 19988, 2025.
[49] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, 2025.
[50] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xucling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodelnterpreterr Integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12834-12859 Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[51] Yaowei Zheng, Richong Zhang. Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Association for Computational Linguistics, 2024.
[52] Qi Zhou, Joy Arulraj, Shamkant B. Navathe, William Harris, and Dong Xu. Automated verification of query equivalence using satisfiability modulo theories. Proc. VLDB Endowment, 12(11):1276-122 2019.
[53] Terry Yue Zhuo Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateck Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang. David Lo, Binyuan Hui, Niklas Muemnighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning petcts://oppeae Representations, 2025. URL https: / /pwpneeveeeeeeeee///e/e/e net/ wwtp///uuuriiie wtr.m? d=
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions and scope, aligning well with the methodology and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of the work in Appendix I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

This paper does not include theoretical results, and therefore does not contain formal theorems or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix G, we provide the alias of the LLMs used, model implementation details, and agent design specifics. Additionally, in Appendix K, we include all the prompts utilized the experiments. We include the code and data in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The code and data used to reproduce the main experimental results are provided in the supplementary material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /tt///nsssc.o cc/
public/ guides / SuodeSuomi asionPolicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips cc/ http/pulicc/// ////uudiio meides//a Code Submi ss cesionPolla piiccy))) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide detailed information on the experimental setup, including model specifications, training procedures, and testing configurations in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All the results shown in the paper are based on experiments run three times for each model and report the average results. This approach ensures that the results reflect consistent performance across multiple runs, accounting for any variability
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix G, provide detailed compute resources required for both the training and experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: /ttps:////urip.... httppp/blcc/ttttttiooi.ooooo
Answer: [Yes]
Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics. We have reviewed the guidelines and ensured that the work aligns with the ethical standards outlined, including considerations related to data privacy and model fairness.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the Appendix J.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Appendix A.3, we include the licenses of the existing assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, papers swwiswwi.. thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed documentation for our new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects. All annotators involved in data collection and task creation authors of the work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects, so IRB approval is not required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology of the research does not involve L.MMs as an important, original non-standard component, declaration is required.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: : //nuuriussppp/ www.c//////neceeesseeeececessece 2025/ LLM) for what should or should not be described.
21

---

Appendix Contents
Environment Setup Details
23
A.1 SQL Dialects Implementation
23
A.2 Databases Migration & Modifications
23
A.3 Issue Collection Protocol
B Annotator Qualification
24
B.1 Annotator Entrance Test
24
B.2 Training Tutorial
24
B.3 Qualification Test
24
Evaluation Script Details
25
D Evaluation Metrics
25
More Statisties
26
E.1
Comparison of BIRD-CRITIC with other conversational Text-to-SQL benchmarks
26
E.2 Detailed Statistics of BIRD-CRITIC-MULT
26
E.3
Quality Validation of S1X-GyM
26
Error Analysis Details
27
Experiment Details
28
G.I Alias LLMs
28
G.2 Model Implementation Details
28
G.3 Agent Implementation Details
28
H Algorithm
29
H.1 SQL Rewind Algorithm
29
H.2 BIRD-FIXER Algorithm
29
I Limitation And Future Work
30
Broader Impact
30
K Prompt
31
BIRD-Fixer Example
35
M f-Plan Example
36
22

---

A Environment Setup Details
A.1 SQL Dialects Implementation
For the implementation SQL dialects, we set up sandbox environment using Docker containers. This environment consists of four database containers and one evaluation container, all managed via "docker-compose.ym configuration. The databases used in this setup include:
Table 5: SQL Dialects used in BIRD-CRITIC.
Dialect
Version
URL
PostgreSQL
14.12 8.4 Community Edition
http://wwwpotstssssssooo org//
MySQL
Microsofi SQL Server Oracle
2022 19.3.0 Developer Edition
https: http::/wwwwiii.oooooo com/ sqi-server
https://Nwwt www.::wwwwrcc......o com wtw.://eee..o/aaaaaaeeee.e dttp/astabbbseeeeeaa
Each of these databases is deployed in its container, ensuring isolation and compatibility with the respective SQL. dialects. The containers are connected through Docker Compose, allowing seamless interaction between the databases and the evaluation environment.
A.2 Databases Migration & Modifications
Our initial setup begins with the BIRD-SQL development database, which is based on SQLite. The migration process carried out using Navicat', powerful database management tool. This tool is used to migrate the original SQLite databases to the four SQL dialects mentioned above.
After the migration, the schema structures of the databases are manually verified to ensure that they reflect the correct translations between different dialects. SQL queries, such as 'SELECT * FROM <table>', are executed to check data consistency and ensure that the migration retains the integrity of the original data. This step ensures that the translated databases can be used reliably for testing and evaluating SQL queries in the BIRD-CRITIC framework.
A.3 Issue Collection Protocol
StackOverflow, a prominent online Q&A platform for software development under -rrierch--riendly license (CC B BY-SA 4.0), is frequently utilized primary data source for code-related evaluation research, [20, 44, 11]. To ensure the issue quality, we pre-define a rigorous protocol based on 4 criteria: 1) presence of executable SQL code with identifiable errors inefficiencies, 2) representation of significant database concepts from academic literature or real- world debugging practice, 3) appropriate complexity (queries exceeding 100 tokens incorporating non- trivial function usage) and 4) sufficient contextual information to prevent ambiguity. We incorporate candidate issues that fulfilled at least 3 criteria, thereby assembling representative collection of SQL challenges that authentically reflect the obstacles encountered in professional database application environments.
User Issue Query Collection.
Annotators meticulously review the reproduced issue (P,S. ossue) and craft a new a". This annotation requires ensuring that o": (1) Correctly Implements Intent: Accurately fulfills the user's objective as inferred from P and the context of Dissue- (2) Resolves the Error: Explicitly fixes the identified flaw(s) in Ousuee (3) Is Functionally Correct: Executes successfully the target database instance D (conforming to S) within the specified dialect and produces the expected, correct results. (4) Adheres Best Practices: Solution SQLs should present reasonably efficient and well-formed query. As shown in Figure 1, this results in a curated "Solution Query" (a*) paired with the user query and issue SQLs. Finally, ensure robust evaluation, we annotate each task with evaluation scripts consisting of specific test cases written by Python and SQLs. Details can be found in Appendix C.
Phttps://www wwwp://wwwwww..o.ee/ com
23

---

Docker Setup
Data Annotation Instruction
SQL Debugging Entry Exam BIRD-CRITIC
Whis taterial yoww lltam now regrocuue the Stack Over Annotators
1. This tutorial puides through cetting Docker on
ERITIC Envtarmment
2. kote It area Mac uses you can scip section 862
This tu tutorial ds ded into Bractione
Purpese Structure
3 Thichuteeial sudiid into sectoon
Pytton Lttl
enll leiuu ane timent maurnesi euscsih culiii deleggi toraulll
thus sulks Lamoy wiie
WSLL
Git Bish
SELECT Case
Environment Setup Requirements
Docker
UPDATE Case
Downlccd from Google drive
the poadded Deciereniinn vimomm canaaaeee evall
fek enturions can ru bas nataiii
Pytton Eny
Stock Overllow Group
Gib-ub saue Group
AUTER Care
Roc undun fundiion suag,, sSON, optimiraaioon
Exom Structure
differen
in VScaae
YOU Docker EMy WORKING ON
qaery bebagging Inoransag)
VSCcde for python sode
get
project
Task arur proneetd
Use VSCade code, iythen Squ
Sohemn nuberns Eippr derl
12
Lost modifed zorr 2005 91-21
You beeedly futhnearyy and you
Figure 7: Examples of training materials by screenshots for BIRD-CRITIC annotators. Left: Docker setup instructions for creating the standardized annotation environment. Middle: Data annotation tutorials with detailed procedures for reproducing SQL issues. Right: Entry examination outline evaluate annotator proficiency across various SQL debugging challenges.
used
B Annotator Qualification
B.1
Annotator Entrance Test
To ensure high-quality annotations for the BIRD-CRITIC benchmark, we implemented a rigorous training process for all annotators. Each potential annotator underwent a comprehensive training program before contributing to the benchmark creation.
B.2 Training Tutorial
Annotators participated in an intensive tutorial program covering essential aspects of SQL issue debugging. including:
Database environment setup
Database schema analysis and comprehension
SQL error identification patterns and common debugging approaches
Systematic issue reproduction techniques
Solution validation and evaluation script development
Best practices for creating test cases across different SQL dialects (PostgreSQL, MySQL, Oracle, and SQL Server)
The training materials included detailed documentation, practical examples, and hands-on exercises that mirrored the complexity and diversity of real-world SQL issues. Annotators introduced to the specific annotation workflow required for BIRD-CRITIC benchmark creation.
B.3 Qualification Test
Following the week-long training phase, each candidate annotator required to complete qualification test consisting of ten representative SQL issue debugging tasks.
For each task, candidates had to:
1. Correctly identify the underlying issue in the problematic SQL
2. Reproduce the issue in the controlled environment
3. Develop solution SQL that resolved the identified problems
4. Create comprehensive test cases validate solution correctness
5. Document their reasoning and approach

---

Only candidates who successfully completed all ten tasks with satisfactory quality were approved as annotators for the BIRD-CRITIC benchmark. This stringent qualification process ensured that all annotators met the high standards required for creating robust and trustworthy benchmark.
The qualification test success rate was approximately 90%, indicating the effectiveness of tutorial materials and instruction program in preparing candidates for SQL issue debugging tasks. All annotators who contributed to the final BIRD-CRITIC benchmark successfully passed this qualification process.
C Evaluation Script Details
To rigorously evaluate the correctness and suitability of generated SQL solutions (oer)): particularly in the context of issue resolution, evaluation methodologies must extend beyond superficial syntactic checks or simple result set comparisons. We annotate each task with specific test case functions, which encompass four categories of SQL issue types in BIRD-CRITIC:
Query-like Issues: Predominantly for conventional SELECT queries. Given that BIRD- CRITIC already provides issue SQLs that deliver original user intents, the solution SQLs must preserve these intentions while addressing identified problems. This protocol assesses correctness by executing pped and the ground-truth o² on the database instance and verifying the semantic equivalence of their result sets, typically accommodating variations in tuple ordering unless explicitly constrained by the task specifications.
Management Issues: Essential for tasks involving Data Manipulation Language (DML: OPDATE, INSERT, DELETE), Data Definition Language (DDL: CREATE, ALTER), Data Control Language (DCL: GRANT, REVOKE), complex multi-step procedures. For these cases, domain experts manually design test cases to ensure that the results executed by pred fulfill the specified user requirements.
Personalization Issues:
For tasks imposing specific syntactic or semantic constraints on the solution (c.g., mandatory use of certain SQL features, avoidance of others, derived from the problem description P), this category extends the test case functions of the previous two categories while enforcing additional compliance criteria.
D Evaluation Metrics
In HIRD-CRITIC we adopt the Task Resolution Success Rate (SR %) as metric. This metric measures the percentage of tasks for which a model generates a SQL solution pred that successfully passes the all curated test cases in the evaluation script. Formally, let N be the total number of tasks in the evaluation set, and let T; represent the dedicated evaluation script designed for task i. A generated solution o pred for task is considered successful if and only if (()peed)) returns passing outcome (returns True). The overall Success Rate is then calculated as:
N ((Tr(p)eed.) = True)
SR
N
where 11()) denotes the indicator function, evaluating 1 the condition true and 0 otherwise. This metric directly leverages the outcomes of our comprehensive, category-aware test case framework. Since each test function T, is tailored to the specific nature of the user's issue, evaluating semantic equivalence of results (Soft EX), correctness of database state transitions, adherence to explicit constraints via parsing as appropriate, the SR provides a holistic measure of model's capability. assesses the model's 's ability to generate solutions that are not merely executable, but are functionally correct and contextually appropriate for resolving the specific problem presented in the task instance (P, S, dissme). We argue that this success rate provides more rigorous and practically relevant assessment of SQL issue resolution capabilities compared to metrics focused solely on execution or partial component matching.
25

---

Table Data statistics of features in WERD-CRITIC compared to related benchmarks. [f]: Results taken from public available Spider 2.0 Lite Gold SQL. EM refers to the Exact Match, EX refers to Execution Accuracy, and PCM-F1 refers Partial Component Match FI.
Dataset
Eval
Toks. QQ
# Toks. / SQL
Evaluation Metric
Non Query-like
Multi-Dialect
Spider 1.0
1,034
14.28
30.18
EMIEX
x
SEDE
PCM-FI
X
x
BIRD-SQL
1,543
18.36
50.01
EX
Spider 2.06
547
61.93
412.37
X
BEAVER
203
59.27
538.13
X
x
BIRD-CRITIC PG
307.35
111.47
Test Cases
X
BIRD-CRITIC MULTI
570
296.27
112.64
Test Cases
E
More Statistics
E.1
Comparison of BIRD-CRITIC with other conversational Text-to-SQL. benchmarks
This section compares
with other benchmarks, highlighting its advantages in handling significantly longer user queries and supporting non-query-Iin qukee--ikke SQL statements (e.g., DMI,, DDL), which present additional challenges. Additionally, the custom- custom-designed test cases ensure a faithful evaluation of SQL solutions, while the multi-dialect support enables comprehensive evaluation across diverse environments
E.2 Detailed Statisties of BIRD-CRITIC-MULN
This section focuses the detailed statistics of the BIRID-CRITIC-MUII dataset, emphasizing its support for multiple SQL dialects and showcasing the distribution query types, SQL issues, and test cases across diverse dialects.
Table 7: Statistics grouped by Category and Dialect
Count
Query
Issue SQL
Solution SQL
Test Cases
Category
Mean
Max
Mean
Max
Mean
Max
Mean
Max
Query
304
179.12 12
1058
168.20
1262
126.72
853
80.29
134
Management
104
141.44
519
68.80
267
102.76
578
189.53
733
Personalization
162
146.43
528
100.21
1073
113.01
778
160.50
517
Dialect
PostgreSQL
276
152.61
1058
78.17
1073
103.45
578
151.30
733
MySQL
98
152.86
435
65.12
230
93.40
778
93.34
281
Oracle
98
171.52
421
265.36
1262
155.92
853
93,95
342
SQLServer
98
192.17
403
214.31
798
145.89
542
95.57
459
E.3
Quality Validation of SIX-GYM
This section validates the quality of synthetic data generated by SQL-Rewind by comparing Six- GYM with the manually curated HIRD-CRITIC-PG benchmark. Table 8 demonstrates that synthetic dataset exhibits comparable complexity and diversity across multiple dimensions, including similar distribution of complex operations, higher SQL diversity ratio, and comparable query lengths to comman-annote challenging data.
Table 9 further breaks down performance by SQL complexity. The benefits of f-Plan scale with difficulty: while gains over rejection sampling are modest on simple queries (+7.5 points), they grow dramatically complex queries with 5+ clauses (+29.2 points). f-Plan also resolves instances unsolved by either baseline or rejection sampling, particularly those with high keyword diversity and nested operations. These results highlight that f-Plan narrows the search space through structured
26

---

Table 8:
Data Statistics Comparison between SIX-GYM and BIRD-CRITIC-PG [i]: Diversity
Ratio =
Unique 3-grams/ / Total 3-grams.
Dimension
BIRD-CRITIC-PG
SIX-GYM
User Query Length (mean/max)
162.98/1046
171.1/882
Issue SQL Length (mean/max)
133.29/1262
110.2/1089
Solution SQL Length (mean/max)
112.64/853
94,8/772
SQL. Keywords Coverage
165
157
Complex Operations (%)
54.5
54.3
Multi-clause Queries (%)
59.4
61.2
SQL Diversity Ratio
0.728
0.750
debugging plans, providing explicit guidance that is especially valuable when random exploration becomes ineffective.
Table 9: Success Rate by SQL Complexity
Issue SQL Complexity
Baseline
Reject
f-Plan
Simple (1-2 clauses)
52.3%
62.8%
70.3%
Medium (3-4 clauses)
38.7%
58.8%
69.4%
Complex (5+ clauses)
19.4%
37.1%
66.3%
High Keyword Diversity (10+)
24.1%
35.6%
54.3%
Nested Operations (2+ levels)
21.8%
36.8%
49.2%
F Error Analysis Details
Figure 8 shows examples for each error type, along with an analysis of why the LLM-generated SQL failed the issue SQL query resolution.
Projection (5)
(23,5%)
Qaorkee: Stariag Ove upoukd
upilii My wegit
deaffic.
Trom SOL
maliraate decce
spaata 20000
Salatle
CONSERALIE ONoRleca, ADD CORGEJALIE 1-1%, whete
ECURTIT ement Dewel wattenetti SGl ihitt boand chcre wo cndionati ael daces aY
STUATT saSarata 18, CRBS Oh0NEO UMA AM Boone censt, to XM
CaCo #ecelone Skc sogge -RRgzE Naaaa Cods JEE and aOEO ABU
11T 1E2 Seagie
Eecicino a da dion w for fick accataga comple noe highttrr des Bharre last
to in
N01
SERTORY
castag)
staa per
saggel toa moc pectidd Tome pasiod torae.
SROATS taRiE Soodig 3ET contagueation HRRRECT Jauce sag a0-a2 2004
superted foner
pectod in peart_pectod.
E8 IOY TREEL NODE c. confeloueeccee Be 111"09:121117
Tyl Tivveereen, 1ar1 se the ane and comtery
guailiee 1580 FROR S402E acEny alenects wow.ioooeatatsoo 2225 COSI0NAT1
Hith 1000e0iv pe bod feeae Eoach SELacd and pertodd Lo the saam pectod, cho 1907 peeeaid donart
shye
Figure 8: Detailed Error Analysis
27

---

G Experiment Details
G.1 Alias of LLMs
The following aliases are used for the models in this work:
Claude-3.7-Sonnee claude -attee-----onnn--2225555
Claude-3.7-Someet 7-Soknntt-Tiinking: refers to claude-3-7-somme 3-7-sonnet 3-7-sonnet-22225223 20250219 with ex- tended thinking
O3-Mini: - 1eani-200 2000-01-32-11111
O1-Preview: 01 01-Prreveew222244000011
GPT-4.1:
Gemini-2.0- 2.0-FFash: gemini-2.0-flas.
Think--hin oat.
deepseek-v3: deepseek-char
deepseck-rl: deepseek reasoner
All open-source models are downloaded from Hugging Face":
Llama: Meta-LLane
Qwen-Coder: - Instruct,, Meta Litt-Llamms 3. Instrrettt - Instrutt,
5- Instrucct
Phi: Phi-4
G.2 Model Implementation Details
For inference with proprietary models, we use official API providers, including OpenAl (https: /openaii. com/). Anthropic (https: /www. anthropic com/), Google (https: // gemini gemine. com/), and Deepseek (https: /wwp///wwww////////////////// wwwpsee.....e...mmmmmmmmmmm...e. com/). The total API cost for proprietary models is around $200 USD.
For open-source models, we fine-tune all our models using the LlaMa-Factory library [51] (version 0.9.2) https: htt///gitttii com/hi http/hyyyuaaaa ments are conducted on 8xH100 GPU with 80GB memory. We set the low-rank dimensions as 8, the learning rate 55 and the batch size 4. The specific training hours for each backbone model are shown in Table 10. We use VLLM (version 0.6.4.postl) to perform inference. We set the temperature as 0.1, the top p as 0.95, and the maximum input token length as 8000. We report the experimental results as the average of five repeated trials. The total GPU hours spent on inference are approximately 20 hours.
with LoRA [14]. All our experi-
Table 10: GPU hours spent to train each backbone model.
Model
GPU Hours
Meta-Llama-3.1-88
24.88
Qwen2.5-Coder-7B
22.00
Qwen2.5-Coder-
35.93
Phi-4
31.42
G.3 Agent Implementation Details
All agent designs follow the ReAct framework [43], which uses interleaving Thought, Action, Observation steps. Specifically:
hhtpe:://hugiinggfccc.oo
28

---

SQL-ACT:
The action is the freedom to execute any executable SQL query.
Tool-ACT: Actions are predefined and include:
Schema Inspection: Reveals table/column information.
Sample Data: Previews example rows from a table.
Solution Query: The final, correct SQL query that resolves the user's issue.
H Algorithm
H.1 SQL Rewind Algorithm
We formalize the end-to-end SQL-Rewinn pipeline in Algorithm 1, outlining each stage from raw post extraction the construction of high-quality training tuples.
Algorithm Automatic construction of SIx-Gyn training instances with SQL-Rewind.
Require: Drw (Stack Overflow posts), W (training databases); largel_size; max iler
Ensure: 9 2 target_stt size
procedure [QL_REWIND
collected training tuples
for each post in Druw do if OVERLAP WITH WITH_WITTD_D BIRD_IRDDDCCC (RIITCC(CC(())t then continue
end if
(OTTRACTSQQL(pos))
D regex extraction
for each sgl in do for each db in W do
sol_ sql + ADAPT ADAPT_SSCEEAL SCHEMA(sQL, db)
if EXEC EXEE__O(Xsl_____ Oo(soloss(LL)))
d6) then
issue synthesis and verification
for i to max iter do
(a Nissue; T) GEN ISSUE(sOl sql, db)
VALIDATE(O Fisuee T, sol col_sql, db) then
break
end end for if validation failed then continue end if
user query generation
for + 1 to max iter do
P +- GEN USER QUERY(OIsSUC) Tissues T, db) if' CONSSISTENN(P,, Tisues T, sol_sql) then break
end end for if consistency failed then continue end if
G < G (ddbsS, rissue T, sol__aq)))
if [ 2 target__sie size then
break all loops
end
end if
end for
end for
end for
return G end procedure
H.2 BIRD-FIXER FIXER Algorithm
29

---

Algorithm BIRD-FIXER:
Functional planning, backward inference, and forward validation for
SQL issue fixing.
Require: P. S. Osssei 0², T; F= fk)
Ensure: Trajectory T'
On. on))
Function: BIRD-FIXER
procedure FUNCTIONALPLAN
Annotate symbolic functional plan F (fi,..., from teacher L.M for each f in F do
f. represents an abstract debugging operation mapping iisuue to a end for end procedure procedure BACKWARDINTERENCE
Given the problem (P, S, ussue) and the corrected query o* Generate step-by-step functional plan F (fi....: .(. F is annotated by the teacher LLM to map disuee to o² end procedure
procedure FORWARDVALIDATION
Using (P,S, Oissue) ) and candidate plan F Regenerate solution using SQL-ACT with teacher LLM if Regenerated SQL passes all test cases in ' then Accept F
Retain executable trace 7' = ((t), (111,0100) (In, on, 0n)) else
Discard plan F
end if
end procedure
Limitation And Future Work
Our work primarily focuses on SQL content and knowledge by simplifying the impact of external workflows through containerized Docker environments. Workflow operations such as file reading and editing represent important considerations for future development in BIRD-CRITIC 1.5. Actually, We conducted preliminary experiments on models performing workflow iinterr-ttee content-baat based tasks, where L.MSs only check and revise SQL issues but also results files. This integration resulted in substantial performance drop, with success rates dropping from approximately 30% to 10%. However, prioritize SQL knowledge improvement in this work since significant opportunities for advancement remain in this domain.
Similar to most complex task evaluations [53], BIRD-CRITIC employs single-turn evaluation while striving to make task descriptions as clear as possible. However, real-world applications typically require crucial interaction between users and agents since most cannot articulate their intents or queries with complete clarity and may need multi-turn interactions for clarification or additional information processing. Our recent work, -ete-Itteaa,',, evaluates text-to-SQL performance of LLM agents through dynamic interaction by multi-turn conversational and agentic interactions. Future work will extend BIRD-CRITH to incorporate dynamic user-SQL debugging processes, better simulating the complexity of real-world agent-human interactions.
J Broader Impact
Our work presents an approach to training open-source models specifically designed for debugging SQL issues. Additionally, we introduce a workflow for constructing robust benchmarks from diverse open platforms, such as StackOverflow, through a reproducible loop to mitigate potential data leakage. Furthermore, our research primarily targets technical SQL knowledge within the programming domain. Thus, it does not directly engage with or pose risks concerning broader societal issucs.
http://iidddintteattg..ii.
30

---

K
Prompt
Baseline Prompt for resolving SQL issues with an LLM
are SQL. assistant. Your task is to understand user issue and correct their problematic SQL. given the datahase schema. Please wrap your corrected SQL with sql \n [Your Fixed SQI.] > tags in your response.
Database Schema:
[SCHEMA]
User issue: {USER_ISSUE)
Problematic SQL: [ISSUE_SQL)
Corrected SQL:
31

---

Prompt used to generate Thought
Interact with the [db_id) database using PostgreSQI. solve the issue. You will be given
following information:
1. Database schema: complete CREATE TABLE DDL.
2. User Issue: natural language description of the desired outcome or the current bug.
3. Problematic SQL: the query (or queries) that presently fail meet the requirement,
Use interlcaving Thought, Action, Observation steps.
Thought can reason about the possible errors or other information you think you need for debugging about the current situation. For instance, it could be:
Diagnosis bug you see the current query.
Hypotheses want to confirm (e.g.. Maybe the join is missing date filter).
Reasoning that led you the next SQL step (checking row counts, inspecting NULLs, etc.). A brief plan for what you will try next.
Action can only be the executable Pos SQL.. Observation would execution results feedback from the environment.
Wrap your thought in the <thought> Your Thought /</thoghht> tag and your action in <action> Executable SQL </action>.
The input for you follows: Database Schema
[SCHEMA]
User Issue [USER_ISSUE)
Problematic SQL [
Important Rules:
Wrap your thought in the <thought> [Your Thought </thought> lag and your action in the <action> [Executable SQL] </action> tag.
MOST IMPORTANT:
The action inside the <a <action></acoe inn> tags must be pure PostgreSQL. statements that be executed directly, without any comments needs for additional post-precessing
Now generate the thought and action of the next round given the trajectory history and the input. You still have [turn] turns left. React [history]
<thought>
32

---

Prompt used to generate Action
Interact with the [db_id) database using PostgreSQI. solve the issue. You will be given
following information:
1. Database schema: complete CREATE TABLE DDL.
2. User Issue: natural language description of the desired outcome or the current bug.
3. Problematic SQL: the query (or queries) that presently fails to meet the requirement.
Use interlcaving Thought, Action, Observation steps.
Thought can reason about the possible errors or other information you need for debugging about the current situation. For instance, could be:
Diagnosis bug you see the current query.
Hypotheses want to confirm (e.g.. Maybe the join is missing date filter).
Reasoning that led you the next SQL step (checking row counts, inspecting NULLs, etc.). A brief plan for what you will try next.
Action can only be executable PostgreSQI. SQI. according to the corresponding thought. The Observation would be the execution results feedback from the environment.
Your task generate the action for the current round thought given the react history. Wrap your action <a <action> [Executable SQL] a/actiic ion>, If you think the debugging process done, just output sant [DONE] </ /ttctiiiinn/ooooocooooeoeeeeeeot ion> as the action.
The input for is as follows: Database Schema
(SCHEMA)
User Issue {USER ISSUE)
Prohlematic SQL [ISSUF_SQL.
Important Rules:
MOST IMPORTANT: SQL]</actt </action>
Wrap your
action
in
<action> [Executable
The action inside the <act soctions</cctinns tags must be pure PostgreSQL statements that can be executed directly, without any comments or needs for additional post-processing.
If you believe the debugging process is finished, output <action> [DONE] < </action> the action for this turn.
Now generate the action of this round given the trajectory history and current thought. Generating multiple rounds once NOT ALLOWED! You still have (turn) turns left. React
(history) )
<action>
33

---

Prompt used to generate Corrected SQL
You text-to-SQI. expert. You will be given the following information:
Database schema: complete CREATE TABLE
DDI..
User Issue: natural language description of the desired outcome or the current bug.
3. Problematic SQL: the query (or queries) that presently fails to meet the requirement.
React Thought Chain: A history of your prior debugging iterations, formatted as sequence of thought >> action >> observation tuples. Each tuple is separated from next by a blank line (
n).
Thought Your reasoning: hypotheses about errors, assumptions, or additional data require-
ments.
Action A pure PostgreSQI. statement executed to test correct the issue.
Observation The exccution result returned by the database engine.
Your task is to understand all these contents and generate the final PostgreSQL that could accurately solve the
input is as follows:
Datahase Schemn
[SCHEMA]
User Issue {USER_ISSUE)
Problematic SQL sql list (ISSUE_SQL)
React Thought Chain [HISTORY)
Now generate final PostgreSQL that could accurately solve the user issue and could be directly executed. Wrap your answer the "sql n [Your Answer] " tag. Do not give me extra explanations or comments.
34

---

L BIRD-Fixer Example
Training Instance
Step
User Issue
Thought
I'm identiffy publishers with significant variability in title advances our PostgreSQI database. Specifically,
query is using 'ydd_ and also has a threshald of 2, which is incorrect. need to correct the query to use 'advance" and the correct threshold of 12000. First, will check the available columns in the 'titles' table to confirm the existence of
'advance"
distinct publishers published at least two titles where the difference between highest lowest advance payments exceeds 12000. I've tried writing
to find this information, but I'm getting incorrect results. The issue with I'm filtering the calculating the differences. Could someone help me fix fallowing
properly identify publishers meeting these criteria?
Action
column, name
FROM information, informtion__shema.comm ssouseaaccccsusss WHERE table nome 'titles;;
Problematic SQL
SELECT pub jub_id, MAXiytd_ sales) AS max_sales, MIN/ytd, sales) AS
Observation
pub_id id HAVING COUNT(pub_ (d) > 1 AND [MAXlytd saless MiN(ytdd sales]) 2;"]
status:
execution results:
Solution SQL
column
SELECT DISTINCT pub jd FROM titles GROUP BY pub id) [MAX(avvance)-
pubdate
price
MIN[advance)] 12000;
adivance
Step 2
Step 3
Thought
Thought
The 'advance' column exists. Now! modify the query to use 'advance instead ytd. sales' and the correct threshold of 12000. will also select only the "pub_i'" id" as requested by user.
The query now returns the correct "pub_r id' based on the condition specifled by the distinct ppub jd', will add 'DISTINCT' ensure that each "pub, id' returned only once, even it satisfies the condition multiple times.
the asked
query to
Action
Action
SELECT pub, id FROM titles GROUP BY pub HAVING COUNT[title jd) (MAX]advance)-
SELECT DISTINCT FROM titles GRDUP pub_ Id HAVING CoUNT(title (d) AND (MAX(advance)) MIN[advance)) 12000;
MIN[advancel] 12000;
Observation
Observation
execution status: success
execution status:
execution results:
execution results:
pub_id
pub_
0877
0877
Figure 9: BIRD-Fixes Example.
35

---

M
f-Plan Example
f-Plan
User Issue
I'm trying to identify publishers with significant variability title advances Specifically, need to find distinct publisshers that have publisshed at least two titles where the difference highest exceeds 12000. I've tried writing query to find this information,
PostgreSOL database.
<f alans
Backward Inference
verify column_ column_existttteee www.ssittable=elleeee calanc="advaanc"
payments
validate, aggrepation by= id",
one" prr"sdvanc"" allias"mmax_.nn "nvancaaeeece Tunclion="MAX" per puslisher coneccly aggregtted.
getting incorrect results, The seems be with how I'm filtering the data or calculating the differences. Could someone help fix properly identify publishers meeting these criteria?
validate, aggrogation "ttics',, by-"pub id', count Lel_on""dddrre "advance"" allase"min "min advaanc""" function"
following
comectly aggregeied.
check, select sal. logidquesstion, sq). table="titles")
Problematic SQL
Chesk functions and fitter concitions in SELECT-baps SOL.
sales) AS max sales, pub id HAVING COUNT(pub id) AND (MAXIjtd, sales) MMNilytd sales))
SELECT pub id, sales) MIN(ytd,
</T_plan>
GROUP
Reasoning Trajectory (Refer previous pages)
Solution SQL
SELECT DISTINCT pub, id FROM titles GROUP BY pub _i HAVING COUNT(ttle id) AND (MAX)advance) MIN(advancel) 12000;
Thought
Thought
Thought
Action
considered
User Issue
I'm trying to identify publisheer with significant variabiliity title advances
Reasoning Trajectory
PostgreSau database.
Specifically, need to find distinct publishers that have published least two titles where the difference between their highest and
Thought
Thought
Thought
Action
Action
advance query
Observation
information, but I'm getting incorrect results. The issue with how filtering the someone
calculating differences. the following ssue SOL to properly icentifly publihers meeting these criteria?
Forword Validation
x
Problematic SQL
Output
SELECT pub, jd, MAXIydd (olllyd_sales)aoo max, max_sales sales, MIN(ytd_ saless AS min, sales FROM utles GROUP BY
pub_id
HAVING COUNT]iile_s (id) >1 AND (MAXjdvaacee)
COUNT(pub_ jil IMAXlytd_saler sales) MIN(ysd saless) 2,"
MilN(ddvacce) 12000;
Solution SOL
SELECT DISTINCT pub Id FROM titles GROUP BY pub, HAVING COUNT(Hlle_e 1 AND (MVAiaavancc))
MIN(sdvvnoee)
Figure 10: f-Plan Example.
36


Reviews:
[{"review_id": "0JlOIiXpwl", "reviewer": "Reviewer_HbCB", "summary": "The contributions of this paper are fourfold: (1) BIRD-CRITIC Benchmark: A comprehensive benchmark for SQL issue debugging, which includes real-world tasks and custom evaluation scripts. (2) SIX-GYM Environment: An automated training environment that generates large-scale, executable issue-solution datasets by employing the SQL-Rewind strategy. (3) f-Plan Boosting: A method that extracts high-level debugging plans from SQL solutions, thereby significantly increasing the number of successful training trajectories. (4) BIRD-FIXER Agent: An open-source agent that leverages SQL-ACT and f-Plan Boosting, achieving state-of-the-art performance on SQL debugging tasks.\nBy providing a method for constructing training data, this paper reduces the consumption of human and material resources in data construction. Additionally, the proposed benchmark allows for the evaluation of LLMs' capabilities in SQL issue debugging, thereby contributing to the advancement of debugging capabilities in this area.", "questions": "1. Whether the training data for effective reasoning model?\n2. 2. BIRD-FIXER was only fine-tuned on PostgreSQL trajectories. If fine-tuned on multi-language trajectories, could the model capabilities be further enhanced?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Current research in the Text-to-SQL field mainly focuses on evaluating the ability of large language models (LLMs) to accurately convert natural language queries into SQL statements. However, there is still a lack of corresponding datasets for the evaluation of LLMs' SQL debugging capabilities. The dataset proposed in this paper fills the gap in this field in this aspect.\n2. This dataset is reasonably constructed and based on the relational database in the BIRD-SQL development set. This database has a complex structure and can simulate the real-world database environment to a certain extent. The annotation process was accomplished through the collaboration of two professional teams, and the annotation quality was ensured through cross-validation. The manual annotation method was adopted to further enhance the accuracy and reliability of the data.\n3. The set task is highly challenging. Despite the continuous improvement of LLMs capabilities, the BIRD-CRITIC dataset still poses significant challenges to existing models. For instance, the best-performing model O3-Mini-2025-01-31 has a success rate of only 38.87% in the PostgreSQL subtask and 33.33% in the multi-dialect subtask, indicating considerable room for improvement. \n4. Furthermore, the article also proposes a new method for automatically constructing training data annotations, and through experiments, it is verified that this method has a significant effect in improving the SQL debugging ability of LLMs.\n\nWeakness：\n1. The experimental section could be more robust. Although this paper claims that the proposed training method is model-architecture-agnostic, it only trains four models, and the training does not cover the thinking model. This is not sufficient to demonstrate the effectiveness of the training data.\n2. The paper mentions the differential distributions of SQL dialects within the respective training corpora of these models, which has an impact on the models' performance across different dialects. However, in the paragraph at line 300, it is stated that the model is only trained on PostgreSQL and is extensible, but there are no experiments to show whether training on other dialects would lead to improvements.", "was_revised": true, "final_justification": "Taking all the discussions into consideration, I agree to accept this paper.", "rebuttal": "We sincerely thanks for your appreciation of our work. And are grateful for your insightful questions. Let us answer them one by one:\n\n> Concern 1: Although this paper claims that the proposed training method is model-architecture-agnostic, it only trains four models, and the training does not cover the thinking model. Whether the training data for effective reasoning model?\n\n**A1:** We appreciate your attention to the breadth of our experimental details! \nRegarding model architecture diversity, our selection was strategically designed to represent key and popular architectural paradigms in the current LLM landscape. We evaluated models across different parameter scales (7B-14B), training methodologies (general pre-training vs. code-specialized), and architectural families (Llama, Qwen, Phi) with their different distributions of pre-trained data and model structures. The consistent improvements observed across these diverse architectures provide evidence for the architecture-agnostic nature of our approach.\n\nRegarding reasoning model, actually, the open-weights appear very recently, i.e., Qwen 3 with reasoning choice is in 14th May, and Phi-4-Reasoning appears on 30th April, which are closer to submission DDL. However, we appreciate your suggestions for this, and we tried training with the same trajectory data for Qwen-3, with reasoning enabled during inference. The result shows:\n| Category | Query  | Management  | Personalization | Overall |\n|----------|-------------|-----------------|----------------------|---------|\n| Qwen3-8B-Reason | 27.49 | 39.77 | 21.85 | 27.92 |\n| +BIRD-FIXER | **28.87** | **52.27** | **26.49** | **32.08** |\n\nIt shows our method can be also effective to recent reasoning-based open-weights model, and we thanks for your suggestion to make our statement more rigorous and robust.\n\n> Question: BIRD-FIXER was only fine-tuned on PostgreSQL trajectories. If fine-tuned on multi-language trajectories, could the model capabilities be further enhanced?\n\n**A2:** Thank you for raising this important point. Database agent training paradigm presents unique challenges due to its heterogeneous and long-context nature. Each trajectory requires extensive interaction with database environments, making data collection particularly time-consuming, especially when working with large databases. Additionally, while executable SQLs after processing are available on platforms like StackOverflow, reconstructing their corresponding database environments remains technically challenging. These constraints motivated our design of methods that account for such low-resource scenarios while demonstrating generalized capabilities. However, we agree with your suggestion which can further strengthen our statement. Within limited time and budget during rebuttal, we conducted preliminary experiments by collecting 2,216, 1,432, and 1,986 multi-dialect SIX-GYM instances for MySQL, Oracle, and SQL Server respectively, following the methods described in Sections 4 and 5 to fine-tune Qwen-2.5-Coder-7B specifically for each dialect:\n\n| Model | MySQL | Oracle | SQL Server |\n|-------|-------|--------|------------|\n| Base | 18.37 | 7.14 | 20.41 |\n| + BIRD-FIXER | **27.55** | **12.24** | **29.59** |\n\nThese results demonstrate obvious improvements with dialect-specific training. Notably, we observe that our previous generalization approach primarily addressed syntax and logic errors leveraging shared database knowledge across dialects. However, dialect-specific training enables the solution of environment-specific errors. For example, for SQL Server specifically, this includes handling T-SQL-specific constructs such as error handling with TRY...CATCH blocks. These dialect-specific nuances require targeted training to master more effectively.\n\nThis additional investigation validates your hypothesis and represents a promising direction for future work. We appreciate your insightful question."}, {"review_id": "wVMl7jHrZA", "reviewer": "Reviewer_Zyg2", "summary": "This paper presents first a benchmark (BIRD-CRITIC) which deals with repairing and debugging SQL-based programming problems. The authors manually curated problems from real user issues after applying them on a controlled environment. The authors also created a synthetic training dataset by synthetically generating new problems and solution using an LLM. The paper performs several rounds of evaluation and demonstrate that prior popular LLMs still struggle on the task of SQL-based debugging and shows the effect of fine-tuning on the synethtic training dataset has on performance", "questions": "1. Did the authors manually evaluate the quality of the trajectories/problems in the synthetic benchmark and see if they are of similar quality to the collected benchmark?\n2. What exactly are the benefits of f-Plan Boosting (i.e., Can it be used to generated very difficult trajectories from problems that the basic approach cannot?). Furthermore, what is the trajectory generation setting and does it use some level of rejection sampling scheme?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall the paper is well-written and the amount of contribution is strong\n\nstrengths \n- extensive description of how the benchmark is constructed\n- human-validated dataset construction with cross-validation to ensure high accuracy of the benchmark\n- the high number of testcases per each problem is also highly encouraging\n- proposed additional synthetic training dataset that can be further adopted and used by future work\n- strong amount of evaluation to analyze across different problem types and across many different models and model/agent setups\n\nweaknesses\n- its bit unclear what the exact quality of the synthetic dataset generated to create SIX-GYM is. For example, it would benefit the work if the authors selected several samples from the synthetic dataset and manually verified the quality of the benchmark\n- \"In our experiments, running Gemini-2.0-Flash with SQL-ACT on SIX-GYM produces just 1,254 successful trajectories\" from this sentence it is very unclear if the authors performed some level of rejection-sampling or just simple sample the model once?\n- The previous point also follows my next question: what exactly are the benefits of f-Plan Boosting? For example the authors claims that without f-Planning not all the trajectories can be successfully generated in the synthetic benchmark. However, is this due to the low number of sampled trajectories (i.e., lack of rejection sampling). I think a better benefit would be to show that f-Planning can arrive at a successful trajectory much sooner than regularly generating a trajectory. However this analysis is not shown in the paper", "was_revised": true, "final_justification": "In my opinion I think the paper does extensive work not only to create an environment that supports training SQL agents (generating synthetic training data) but additionally purpose a simple method to improve the probability and success rate of training trajectries. Furthermore the basic agent setup proposed in the paper can also serve as a good baseline for future work to compare with. Additionally, the authors also answered my concerns and questions with additioanly experiments and text that I hope they will include in the final version. As such I remain positive on this work and would recommend acceptance.", "rebuttal": "Thanks for your appreciation of our work and thoughtful suggestions and feedbacks. Here are our responses:\n\n> Concern 1: It's bit unclear what the exact quality of the synthetic dataset generated to create SIX-GYM is. For example, it would benefit the work if the authors selected several samples from the synthetic dataset and manually verified the quality of the benchmark.\n\n**A1:** Thank you for raising this. We acknowledge that demonstrating the quality of SIX-GYM is crucial for establishing the credibility of our approach.\n\nOriginally, The downstream performance improvements achieved by models trained on SIX-GYM serve as a strong indicator of dataset quality. BIRD-FIXER demonstrates consistent gains across different model architectures, suggesting that our synthetic data captures meaningful debugging patterns that transfer effectively to real-world scenarios.\n\nBeyond that, we also present detailed statistics comparing SIX-GYM with manually curated BIRD-CRITIC to demonstrate comparable data characteristics below. It shows that our synthesized GYM has similar quality and complexity with human annotated data:\n\n| Dimension | BIRD-CRITIC-PG (Benchmark) | SIX-GYM (SQL-Rewind) |\n|-----------|-------------|---------|\n| User Query Length (mean/max) | 162.98/1046 | 171.1/882 | \n| Issue SQL Length (mean/max) | 133.29/1262  | 110.2/1089 | \n| Solution SQL Length (mean/max) |112.64/853 | 94.8/772 | \n| SQL Keywords Coverage | 165 | 157 | Better feature coverage |\n| Complex Operations (JOINs, Window Funcs) % | 54.5 | 54.3 | \n| Multi-clause Queries % | 59.4 | 61.2 |\n| SQL Diversity Ratio | 0.728 | 0.750 |\n* Diversity Ratio = Unique 3-grams / Total 3-grams of SQLs; The closer the ratio is to 1.0, the higher the diversity.\n\nAlso, we sampled and dispatched 100 medium-difficult tasks to 5 annotators to manually evaluate synthesized data quality as you suggested:\n- **Issue-Solution Logical Consistency (88%)**: This measures whether the synthetic issue SQL can logically lead to the provided solution SQL. 0 means the transformation is arbitrary or nonsensical or it contains multi-hop gaps (not only one type of errors), while 1 means the debugging path is logically sound and semantically coherent.\n- **Realistic Error Patterns (94%)**: This evaluates whether the introduced errors mirror authentic debugging scenarios commonly encountered in their experience of understanding real-world user issues during testing annotation. 0 means the errors are obviously artificial or not meaningful, while 1 means they reflect genuine mistakes developers make in practice.\n- **Executable Code Quality (100%)**: This verifies that the solution SQL executes successfully against the target database schemas. 0 means the solution SQL contains syntax errors or runtime failures, while 1 means both queries execute without errors. \n- **User Intent Coherence (86%)**: This measures whether the generated user issue description accurately reflects the debugging objective, provides sufficient context, and has less ambiguity. 0 means the description is unclear or misaligned with the SQL changes, while 1 means it clearly conveys the intended debugging goal.\n\nThus, we think the combination of strong empirical results, comparable statistical distributions to manual benchmarks, demonstrates that SIX-GYM achieves high-quality synthetic data generation suitable for training effective SQL debugging agents. Thanks for your question!\n\n\n\n\n\n> Concern 2: What exactly are the benefits of f-Plan Boosting (i.e., Can it be used to generated very difficult trajectories from problems that the basic approach cannot?). Furthermore, what is the trajectory generation setting and does it use some level of rejection sampling scheme?\n\n**A2:** Thank you for this excellent question. Your inquiry touches on a critical aspect of our contribution that deserves detailed clarification. \n\nDatabase agent training paradigm presents unique challenges due to its heterogeneous and long-context nature. Each trajectory requires extensive interaction with database environments, making data collection particularly time-consuming, especially when working with large databases\n\nTo better answer your question, we conducted additional experiments comparing different trajectory generation strategies:\n\n| Method | Max Tries | Successful Trajectories | BIRD-FIXER (7B) | Avg Tries | DB Execution Time (min) | Cost (USD) |\n|--------|-----------|------------------------|-----------------|-----------|-------------------------|------------|\n| Baseline | 1 | 1,254 | 25.85% | 1.0 | 306 | 8.47 |\n| f-plan (Ours) | 1 | 2,178 | 31.32% | 1.0 | 324 | 27.44 |\n| Reject Sampling | 5 | 1,910 | 29.06% | 4.2 | 1377 | 108.05 |\n| Reject Sampling + f-plan | 5 | 2,560 | 33.02% | 1.7 | 810 | 41.16 |\n\n**Trajectory Sampling Strategy:**\n- **Baseline**: Standard SQL-ACT rollout with single attempt\n- **f-plan (Ours)**: Our f-plan approach (single attempt)\n- **Reject Sampling**: Multiple attempts (max_tries=5) with SQL-ACT by setting temperature=0.8, and early stop if the trajectory is successful.\n- **Reject Sampling + f-plan**: Combined approach of f-plan boosting with rejection sampling (max_tries=5).\n\n**Complex Issue SQL Resolution Analysis:**\n\nWe analyzed instances where f-plan succeeds but baseline and rejection sampling fail, finding that f-plan particularly excels on complex SQL structures:\n\n| Issue SQL Complexity | Baseline Success Rate | Reject Sampling Success Rate | f-plan Success Rate |\n|----------------|----------------------|----------------------------|-------------------|\n| Simple (1-2 clauses) | 52.3% | 62.8% | 70.3% |\n| Medium (3-4 clauses) | 38.7% | 58.8% | 69.4% | \n| Complex (5+ clauses) | 19.4% | 37.1% | 66.3% |\n| High Keyword Diversity (10+ unique) | 24.1% | 35.6% | 54.3% | \n| Nested Operations (2+ levels) | 21.8% |  36.8% | 49.2% | \n\nIn summary, we can conclude the benefits of f-plan:\n\n1. It's a **cost-efficient** trajectory-augmentation strategy leading to better agent performance in complex DB environments with less time and token/money cost.\n\n2. **Execution Efficiency**: Reduces average tries from 4.2 to 1.7 when combined with rejection sampling, significantly reducing database execution time and computational overhead. And this prove your guess that our f-plan boosting can make rejection sampling reach success sooner, we would include this into paper if we get a chance, thank you!\n\n3. **Complex Problem Resolution**: We observe that f-plan can resolve GYM instances that both rejection sampling and baseline cannot handle. F-plan helps teacher LLMs resolve instances with complex issue SQLs, showing obvious improvement when SQL contains more clauses and diversity of keywords (representing more diverse user intents).\n\n4. **Plug-and-Play Compatibility**: F-plan is orthogonal to traditional trajectory-augmentation strategies like rejection sampling, providing additional 34.0% successful trajectory improvement when combined while maintaining efficiency gains.\n\nThanks for your insightful question. After answering with additional experiments as evidence, it further demonstrates the benefits of our method for complex database debugging scenarios. We would include this analysis into paper to make readers better understand its benefits beyond downstream performance"}, {"review_id": "cMn3aTAQvK", "reviewer": "Reviewer_6yZr", "summary": "This paper introduces a new and challenging SQL benchmark named BIRD-CRITIC, which covers a wide range of complex SQL tasks. It is designed to evaluate the ability of large language models (LLMs) to decompose, analyze, and reason over intricate instructions in order to generate correct SQL queries. To address common issues such as formatting or logical errors that arise from insufficient SQL understanding in current models, the authors also propose an auxiliary agent called BIRD-FIXER. This agent enhances the model’s SQL comprehension and performs post-hoc corrections on generated outputs.", "questions": "1. Since your work aims to address the limitations of current models on complex SQL tasks, what do you think is the main reason these models struggle? Is it due to a lack of SQL-specific knowledge, or is it because these tasks resemble mathematical reasoning and require more advanced compositional abilities? If a model were equipped with strong SQL background knowledge, would your benchmark still offer meaningful challenges and insights for further improvement?\n\n2. Many of the tested models may not have been specifically optimized for SQL tasks. In general, SQL can be seen as one branch of code-related tasks. From your perspective, what makes SQL benchmarks necessary compared to other code benchmarks? What specific capabilities of LLMs do SQL tasks reveal that might be overlooked by more general code-oriented benchmarks?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed benchmark, BIRD-CRITIC, includes more complex and realistic SQL tasks derived from real user queries. Current mainstream reasoning models (e.g., O3-mini) struggle with these tasks, making this benchmark a valuable resource for the community to evaluate and improve LLMs on challenging SQL problems.\n\n2. The design of the BIRD-FIXER agent effectively corrects errors in the SQL generation process, including both syntactic and logical issues. This design makes it easier for future researchers to diagnose and improve model behavior in a more interpretable and modular way.\n\n3. The paper is well-structured and comprehensive. It includes detailed examples of benchmark tasks, data collection methodology, dataset statistics, baseline results from popular models, ablation studies on BIRD-FIXER, and a thoughtful categorization of failure cases. The analysis is thorough and informative.\n\n**Weaknesses:**\n\nThe approach used in BIRD-FIXER is conceptually similar to techniques already explored in other areas of LLM research, and thus lacks strong novelty. However, the adaptation of this idea to the SQL domain is still a meaningful contribution, especially given the practical challenges of SQL generation.", "was_revised": true, "final_justification": "I believe this is an excellent paper that not only introduces a benchmark but also offers deeper insights and innovations in the field, which contribute meaningfully to advancing the capabilities of large language models.", "rebuttal": "> Concern 1: The approach used in BIRD-FIXER is conceptually similar to techniques already explored in other areas of LLM research, and thus lacks strong novelty. However, the adaptation of this idea to the SQL domain is still a meaningful contribution, especially given the practical challenges of SQL generation.\n\n**A1:** Thank you for raising this point. We appreciate the opportunity to clarify the novel contributions of our work.\n\nAs we cited in the paper, the motivation of GYM-like dataset construction is inspired by SWE-GYM but we present an automatic framework to synthesize high-quality query-issue pairs with complex DB environment, BIRD-FIXER introduces several technical innovations:\n\n1. Our f-Plan Boosting represents a fundamentally new approach to enhancing trajectory quality in GYM-like environments. Unlike generic fine-tuning approaches, this method leverages the logical structure inherent in SQL solutions to create richer supervisory signals. The two-phase process (backward inference → forward validation) with automatic functional plan extraction is specifically designed for SQL debugging scenarios and generated 73.7% more successful training trajectories than baseline approaches.\n\n2. Beyond performance improvements, our method significantly enhances the efficiency of trajectory collection, as shown in our analysis:\n\n| Method | Max Tries | Successful Trajectories | BIRD-FIXER (7B) | Avg Tries | DB Execution Time (min) | Cost (USD) |\n|--------|-----------|------------------------|-----------------|-----------|-------------------------|------------|\n| Baseline | 1 | 1,254 | 25.85% | 1.0 | 306 | 8.47 |\n| f-plan (Ours) | 1 | 2,178 | 31.32% | 1.0 | 324 | 27.44 |\n| Reject Sampling | 5 | 1,910 | 29.06% | 4.2 | 1377 | 108.05 |\n| Reject Sampling + f-plan | 5 | 2,560 | 33.02% | 1.7 | 810 | 41.16 |\n\n**Trajectory Sampling Strategy:**\n- **Baseline**: Standard SQL-ACT rollout with single attempt\n- **f-plan (Ours)**: Our f-plan approach (single attempt)\n- **Reject Sampling**: Multiple attempts (max_tries=5) with SQL-ACT by setting temperature=0.8, and early stop if the trajectory is successful.\n- **Reject Sampling + f-plan**: Combined approach of f-plan boosting with rejection sampling.\n\nOur approach achieves more successful trajectories generated by dramatically lower computational cost compared to standard rejection sampling methods. Also, it shows f-Planning can arrive at a successful trajectory much sooner than regular and popular trajectory augmentation strategy as rejection sampling as Reviewer Zyg2 mentioned. To our best knowledge, this is the first work to propose such a strategy for enhancing utilization of GYM-like datasets. We think this would open important directions for better utilization of GYM-like datasets and hope it will inspire further research in efficient trajectory generation methods for specialized domains.\n\n3. Our SQL-ACT framework treats arbitrary SQL commands as actions rather than restricting to predefined tools, fundamentally expanding the action space beyond conventional tool-based agents. Experimental results demonstrate that SQL-driven agents can handle more complex debugging scenarios through this flexible exploration space, which is essential for addressing the diverse and unpredictable nature of user errors and varied database logic requirements.\n4. We also present a An innovative decoupling mechanism, **Generative Thought Mode (GTM)**, that separates debugging reasoning from SQL code generation, addressing overfitting issues while maintaining the model's native understanding of diverse SQL dialects.\n\nThanks for your question and suggestion.\n\n\n> Question 1: what do you think is the main reason these models struggle? Is it due to a lack of SQL-specific knowledge, or is it because these tasks resemble mathematical reasoning and require more advanced compositional abilities?\n\n**A2:** We sincerely thank you for proposing such deep questions. Actually, we agree that the primary difficulty for Large Language Models (LLMs) in complex SQL tasks stems more from limitations in their reasoning and compositional abilities than from a mere lack of SQL-specific knowledge.\n\n1. First, as we can observe, these models perform well on popular text-to-SQL benchmarks such as BIRD-SQL, Spider, etc. For example, Qwen-2.5-Coder can achieve 58.4% EX (which is a more strict metric) on BIRD without any fine-tuning and prompting techniques, but 30.75% success rate (by a more flexible test case evaluation) in BIRD-CRITIC-PG and 24.74% in BIRD-CRITIC-MULTI, which shows even though they have a good knowledge about SQLs, actually, SQL debugging logics maybe still a new research problem.\n2. The process of debugging SQL is inherently a \"reasoning-driven\" task. It requires not only understanding the user's intent from a verbose description but also analyzing the underlying query logic, identifying subtle errors, and interacting intensively with the database schema in Line 46-49. For example, an analysis of the errors made by these models reveals that \"Incorrect Logic\" is the most frequent failure category. This points to fundamental misunderstandings of data structures or how to perform complex transformations. Therefore, merely equipping a model with more SQL background knowledge would not be sufficient but a pre-requisite. SQL debugging requires a multi-dimension combination of capabilities: domain-specific knowledge (including mathematical computation and specialized fields like chemistry when working with scientific databases), deep understanding of data relationships and schema semantics, and sophisticated agentic debugging reasoning similar to what we observe in SWE-Bench. We will elaborate on these interconnected requirements in the following detailed response.\n\n> Question 2: In general, SQL can be seen as one branch of code-related tasks. From your perspective, what makes SQL benchmarks necessary compared to other code benchmarks? What specific capabilities of LLMs do SQL tasks reveal that might be overlooked by more general code-oriented benchmarks?\n\n**A2:** Thanks for this deep question. We think the special challenges by SQL debugging would be:\n\n\n1. A key challenge that SQL benchmarks illuminate is the capability of debugging **declarative** code. Unlike procedural languages like Python, where logic is often presented in a step-by-step manner, SQL's declarative nature nests complex reasoning within its clauses. This makes the user's intent and the query's logical flow difficult to trace from the code alone in a linear thinking. Furthermore, the feedback loop in database systems is often **less informative** than in general programming. For example, Oracle always return very concise error feedback such as `ORA-xxx: SQL command not properly ended`. This error confirms a syntax problem but gives no clue as to *where* or *why*, forcing the model to deduce the issue is a non-standard keyword making it a hard way of error location. This lack of explicit guidance means the model cannot simply follow an error stack trace; it must adopt an **agentic, exploratory approach** to form and test hypotheses against the database. While CTEs in SQL can break down queries step-by-step, they sacrifice precise Query Execution Plan generation, preventing accurate efficiency analysis. As we know that the SQL efficiency is important concern in DB field. Therefore, many user issue SQLs stem from declarative constructs that inherently hide useful reasoning and diagnostic clues.\n\n2. As we showed in error analysis, Chain-of-Errors matter in SQL debugging which requires agent capable of deep, causal reasoning in the face of masked errors. To be specific, the database engine reports the most superficial issue, masking a deeper, dependent error that is the true root cause. For instance, a type mismatch error might be reported, but the underlying problem could be an incorrect join that brought together the wrong columns in the first place. This contrasts with many general code benchmarks where an error trace can provide a clearer path to the initial point of failure.\n\n3. As usual Text-to SQL tasks, SQL queries are inherently business-driven, requiring extensive domain-specific knowledge across various fields ( in our benchmark it can be from financial calculations to chemical compound relationships). Models must not only possess this external knowledge but also map it accurately to specific database representations involving **precise** tables, columns, and data types. This semantic bridging between domain concepts and database schema is a unique challenge rarely found in general programming tasks. The first example in Figure 8 demonstrates how domain understanding directly impacts query correctness.\n\nIn conclusion we can consider SQL debugging a specialized complex tasks that are dictated by particularly their requirements for unique nature of databases, particularly their requirements for privacy, efficiency, data integrity, and domain knowledge mapping. An LLM cannot succeed by simply converting text; it must operate as a capable of exploration, hypothesis, inference, and adaptation within this constrained, stateful environment."}, {"review_id": "jTmdE8XXzm", "reviewer": "Reviewer_ixMH", "summary": "The authors proposed a new benchmark called BIRD-CRITIC for evaluating the ability of LLMs to fix issues in SQL queries. The benchmark was constructed by sourcing real questions on SQL overflow. Human annotators then adapted and reimplemented these issues using the set of reference databases provided by the pre-existing BIRD-SQL dataset. The benchmark is challenging for recent LLMs. The authors also proposed a related method called SIX-GYM for synthetically generating fine-tuning data for this task by taking correct SQL queries, introducing errors, and generating debugging trajectories. They also developed BIRD-FIXER, an agent using an open-weights model fine-tuned using SIX-GYM, which performs on par with recent proprietary reasoning models.", "questions": "- Could you comment on how SQL debugging is different from Text-to-SQL generation from scratch? In particular, if you were to run an evaluation using BIRD- FIXER where the models were only given the user issue description but not the issue SQL, would the models still perform as well?\n- I would expect the SQL-Act agent to struggle more on instances that require database schema migrations or destructive write operations, because the agent isn’t able to “try out” possible queries by taking actions (which would mutate the database) as it would with read-only SELECT queries. Did you observe this in your experiments?\n- Did you try some of these possible alternatives for fine-tuning BIRD-FIXER?\n    - Fine-tuning on a small train split of BIRD-CRITIC instead of on the SIX-GYM synthetic dataset\n    - Retaining the generated functional plan in SIX-GYM instead of discarding it", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n- The dataset has strong ecological validity because they are derived from real StackOverflow posts. It is also likely to be high quality due to the involvement of human annotators and experts. (Quality)\n- The dataset is reimplemented to work on the reference datasets from the pre-existing BIRD-SQL dataset, which makes it easy for a user to set up the databases in their own environment for running the benchmark. (Quality)\n- A number of different SQL dialects are included, and the results between them are compared. (Quality)\n- Generally, the writing is clear and well-structured. (Clarity)\n- The design of SIX-GYM is original and interesting. (Quality, Originality)\n- SQL-fixing is a useful task with many useful applications (e.g. data scientist copilot). While a number of Text-to-SQL benchmarks already exist, there are no notable SQL-fixing benchmarks to the best of my knowledge. (Significance, Originality)\n\n### Weaknesses\n\n- It is not clear to me that SQL debugging is a significantly different task from Text-to-SQL generation from scratch. The analogy to code debugging is weak. While code debugging requires understanding a large amount of pre-existing code to fix a bug and then making a small change, SQL queries are typically very short, and an incorrect query can simply be rewritten from scratch. (Originality, Significance)\n- Microsoft SQL Server and Oracle are proprietary databases, unlike Postgres and MySQL which are open source databases. Including SQL Server and Oracle has benefits for industry users because these are widely used databases in industry, but it may make it more difficult for academic users to run the benchmark as they may not have access to these database platforms. The paper also incorrectly claims in Section 1 Introduction that SQL Server and Oracle are open source databases. (Quality, Significance)\n- I felt that the description of the SQL-Act agent in Section 5.1 is not clear enough. My understanding is that the agent’s actions are sending SQL queries, and the observations are the results of executing the queries. (Clarity)\n- It would be useful to include an example of SIX-GYM in the appendix, including a function plan F and the corresponding generated solution. (Clarity)\n- There are a number of minor language and typographical issues. A partial list is below. (Clarity)\n    - “SQLs” should be “SQL queries” in multiple places\n    - Appendix E.2 - “SQLServer” should be “SQL Server”\n    - Section 5.1 SetUp - “SetUp” should be “Setup”\n    - Section 6.2 Main Results - Remove the line break after “more standardized data management operations,”\n    - Section 6.5 Error Analysis - The long sentence starting with “It can be concluded” should be broken into multiple sentences.\n    - Section 6 Related Work and Appendix I Related Work have some repeated material.", "was_revised": true, "final_justification": "This paper is presents a benchmark for a task that lacks existing coverage (SQL debugging) and a novel method for generating synthetic data for training models for this task. These ideas are novel, tested and practical, and overall this seems like a valuable contribution to the field.\n\nThe authors have addressed my primary concern - that SQL debugging is not significantly different from Text-to-SQL generation from scratch. The authors have ran additional experiments that show that the success rate increases significantly when the user's incorrect query is included. This demonstrates that SQL debugging is significantly different from Text-to-SQL generation from scratch, because the user's incorrect query contains important information on the user's intent. My additional questions have been sufficiently addressed as well.", "rebuttal": "Thanks for your insightful and feedbacks, we will revise typo errors as you mentioned and answer your questions as below:\n\n> W1 & Q1: Could you comment on how SQL debugging is different from Text-to-SQL generation from scratch? Does BIRD-FIXER still perform more effectively without issue SQLs?\n\n**A1:** Thank you for this insightful comment. We agree that SQL queries can be short, but we think this doesn't make SQL debugging equivalent to generation from scratch or that the analogy to code debugging is weak. Our research demonstrates that SQL issue resolution is a distinct and more complex task.\n\nWe summarize reasons as:\n\n1.  **Debugging is Diagnostic, Not Just Generative:** The core difference lies in the inputs.\n    * **Text-to-SQL generation** takes a natural language description ($\\mathcal{P}$) and a schema ($\\mathcal{S}$) to produce a query ($\\sigma_{pred}$).\n    * **Our SQL issue resolution task**, as formally defined in Section 2, takes three inputs: the description ($\\mathcal{P}$), the schema ($\\mathcal{S}$), and the user's own incorrect query ($\\sigma_{issue}$). \n    * The presence of $\\sigma_{issue}$ fundamentally changes the task. The model is not just generating a solution; it must first **diagnose the flaw** in the user's provided logic. Many issues in our BIRD-CRITIC benchmark are not syntax errors but logical or semantic flaws in which the provided SQL is often executable but produces an incorrect result. A model that simply tries to generate a query \"from scratch\" based on the description might easily repeat the same logical error since it hasn't understood *why* the user's initial attempt was wrong. The incorrect query provides crucial negative evidence about a plausible but flawed reasoning path. Also, we observe that most of user queries will not mention too many details about semantics but about their main aims since they think the provided SQLs could provide enough reference and only focus on error description such as which types of JOINs, or which color of cards that they construct for such recursive tree structure since issue SQL already provide these basic information.\n\n2.  **SQL is declarative, which means Logical Complexity is High:** As we answered for Reviewer 6yZr, one of the biggest challenges in SQL debugging is due to less context, partially caused by nested and short SQL presentation. Unlike procedural languages like Python, where logic is often presented in a step-by-step manner, SQL's declarative nature nests complex reasoning within its clauses. This makes the user's intent and the query's logical flow difficult to trace from the SQL alone in a linear thinking. Furthermore, the feedback loop in database systems is often **less informative** than in general programming. For example, Oracle always return very concise error feedback such as `ORA-xxx: SQL command not properly ended`. Those in real-world applications and our benchmark often contain deeply nested logic that is not trivial to rewrite.\n    - Rewriting such a query \"from scratch\" is often more difficult than debugging it. The incorrect query, even with its flaws, serves as a vital blueprint of the user's intent and structural approach. Discarding it would mean losing valuable context.\n\n3. We also conduct experiments to show performance of baselines, BIRD-FIXER for Qwen-2.5-Coder 14B and the performance of O3-mini for reference.\n\n| Model | PG w/o issue sqls | PG w/ issue sqls | MULTI w/o issue sqls | MULTI w/ issue sqls |\n|-------|-------------------------------|----------------------------------|-----------------------------------|-------------------------------------|\n| Qwen-2.5-Coder-14B | 17.74 | 31.32 | 13.33| 24.04 |\n| + BIRD-FIXER (ours) | 24.34 | 38.11 | 16.67 | 29.65 |\n| O3-mini | 26.42 | 38.87 | 15.96 | 33.33 |\n\nThe sharp drop from the setting with issue SQLs to without issue SQLs demonstrate that the issue_sql is an indispensable context that allows the model to ground its reasoning and diagnose the user's specific logical error. And our method can still perform well in such setting. But of course, we could manually convert the GT SQLs into conventional text-to-SQL tasks, it may require efforts of humans or experts to make sure every detail would be contained in user queries though. We would consider this as future advanced text-to-SQL tasks. Thanks!\n\n> W2: Microsoft SQL Server and Oracle are proprietary databases, and it may make it more difficult for academic users to run the benchmark as they may not have access to these database platforms. \n\n\n**A2:** Thank you for pointing out this important clarification regarding database statement.\n\nWe feel sorry for the inaccurate description in Section 1 Introduction. We will revise this to accurately describe them as community-friendly cloud-based dialect.\n\nAnd we want to provide more contexts here to address your concerns of Accessibility of these dialects:\n\n1. Both SQL Server and Oracle officially provide free editions suitable for research and educational purposes. And we provide comprehensive Docker containerization for all four database environments in our submission codes. Our `docker-compose.yml` configuration automatically handles the deployment and setup, making it straightforward for both academic and industry users to reproduce our experimental environment locally, keeping special features of each dialect, such as T-SQL for SQL Server and Oracle's strict coupling between users and schemas. This eliminates the complex setup burden for researchers.\n\n6. Researchers can still conduct meaningful evaluations using our BIRD-CRITIC-PG (PostgreSQL-only) subset while having the option to expand to multi-dialect evaluation when resources permit.\n\nIn conclusion, we believe our setup provides comprehensive research value while maintaining accessibility through free developer editions and containerized deployment. Aside scientific problems that we proposed and mitigated in the paper, we also did much engineering work to make the evaluation and agent rollout faster and smooth as possible as we can by tacking many technical barriers.\n\n> W3: I felt that the description of the SQL-Act agent in Section 5.1 is not clear enough. My understanding is that the agent’s actions are sending SQL queries, and the observations are the results of executing the queries.\n\n**A3**: Yes, your understanding is correct. And as we described in the paper, BIRD-FIXER is built on SQL-ACT, so the Figure 9 in Appendix M can reveal how SQL-ACT works in an simple example. We feel sorry that we cannot include this example into main content due to limited space. But we will enhance indexing there. Thanks!\n\n> Suggestion: It would be useful to include an example of SIX-GYM in the appendix, including a function plan F and the corresponding generated solution. \n\n**A4:** Thanks, we will include these examples in Appendix if we have a chance for revision. Actually, the example of SIX-GYM is similar to real-world issue solution instances just with statistic different. We feel grateful if you can refer to first table of A1 that we answered for Reviewer Zyg2 due to space limit. We presented a comparison between tasks generated by SIX-GYM and manually annotated benchmark.\n\n> Q2: I would expect the SQL-Act agent to struggle more on instances that require database schema migrations or destructive write operations, because the agent isn’t able to “try out” possible queries by taking actions as it would with read-only SELECT queries. Did you observe this in your experiments?\n\n**A6:** Thank you for this insightful question. It's about a critical aspect of agentic design in state-altering environments.\n\nCounter-intuitively, our experiments show that the agent struggles less on instances requiring database schema migrations or destructive write operations. In fact, these \"Data Management\" tasks, were found to be \"relatively more manageable\" than complex, read-only `SELECT` queries. Our quantitative results from Figure 5 support this finding directly. \n\nThe primary reason for this is the **comparative lack of diversity in the solution space**. That makes management issues while high-stakes, are \"more standardized\". The agent can learn these structured patterns more effectively via training techniques.\n\nHowever, your intuition about the agent's process is correct. We did observe that \"thinking JUMP\" which means agents would make abrupt leaps to final management SQL solutions without intermediate queries. This suggests the agent struggles with the inability to safely test destructive operations leading to a reasoning gap or guess. An advanced version of the agent in the future work could be equipped with a \"sandbox\" mode, allowing it \"try out\" destructive logic against reproduced databases before committing to a more accurate final solution for the actual database.\n\n> Q3:Did you try some of these possible alternatives for fine-tuning BIRD-FIXER?\n\n**A7:** Thank you for this question. We explored several alternative fine-tuning approaches:\n\nWe avoided using splits of our test data for training to prevent data leakage, because BIRD-CRITIC is designed as an evaluation benchmark. So we didn't try that to make evaluation set totally unknown for training technique.\n\nWe attempted with including plan generation during training smaller models to generate plans first before database interaction. However, this approach underperformed our current method for several reasons by our analysis:\n\n1. Smaller models tend to strictly follow their initial plans even when the immediate observations suggest other directions of exploration.\n2. Flawed initial plans lead to cascading errors throughout the debugging process.  \n\nWhile some cases were fixed correctly through explicit planning, the overall performance decrease outweighed these successes. We believe planning remains crucial for SQL agents, but developing more robust planning mechanisms that can adapt to real-time database feedback represents an important research direction."}]

Author rebuttal:

Meta-review: The contributions are: (1)  A comprehensive benchmark for SQL issue debugging, which includes real-world tasks and custom evaluation scripts,  (2)  an automated training environment that generates large-scale, executable issue-solution datasets, (3) a method that extracts high-level debugging plans from SQL solutions and increases the number of successful training trajectories, and  (4) an open-source agent that achieves state-of-the-art performance on SQL debugging tasks.

The authors addressed all concerns of the reviewers in their rebuttal and all reviewers unanimously voted for acceptance. The additional arguments and experiments presented in the rebuttal should be included in the revised version of the manuscript.
Decision: Accept (poster)