Title: SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
Venue: NeurIPS 2025
Accepted: yes
Keywords: text-to-SQL

LLM

SQL issues
Abstract: Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging on SQL issues. In order to address this gap, we introduce **BIRD-CRITIC**, a new SQL issue debugging benchmark comprising 530 carefully curated PostgreSQL tasks (**BIRD-CRITIC-PG**) and 570 multi-dialect tasks (**BIRD-CRITIC-Multi**), which are distilled from authentic user issues and replayed within new environments to facilitate rigorous and contamination-free evaluation. Baseline evaluations on BIRD-CRITIC underscore the task's complexity, with the leading reasoning model **O3-Mini** achieving only 38.87% success rate on **BIRD-CRITIC-PG** and 33.33% on **BIRD-CRITIC-Multi**. Meanwhile, realizing open-source models for database tasks is crucial which can empower local development while safeguarding data privacy. Therefore, we present **Six-Gym** (**S**ql-f**IX**-Gym), a training environment for elevating the capabilities of open-source models specifically for SQL issue debugging. This environment leverages **SQL-Rewind** strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose *f*-Plan Boosting, which extracts high-level debugging plans automatically from SQL solutions, enabling the teacher LLMs to harvest and produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, **BIRD-Fixer**. Based on Qwen-2.5-Coder-14B, **BIRD-Fixer** raises its success rate to 38.11% on **BIRD-CRITIC-PG** and 29.65% on **BIRD-CRITIC-Multi**, surpassing many leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities for both research and industry.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a new benchmark called BIRD-CRITIC for evaluating the ability of LLMs to fix issues in SQL queries. The benchmark was constructed by sourcing real questions on SQL overflow. Human annotators then adapted and reimplemented these issues using the set of reference databases provided by the pre-existing BIRD-SQL dataset. The benchmark is challenging for recent LLMs. The authors also proposed a related method called SIX-GYM for synthetically generating fine-tuning data for this task by taking correct SQL queries, introducing errors, and generating debugging trajectories. They also developed BIRD-FIXER, an agent using an open-weights model fine-tuned using SIX-GYM, which performs on par with recent proprietary reasoning models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The dataset has strong ecological validity because they are derived from real StackOverflow posts. It is also likely to be high quality due to the involvement of human annotators and experts. (Quality)\n- The dataset is reimplemented to work on the reference datasets from the pre-existing BIRD-SQL dataset, which makes it easy for a user to set up the databases in their own environment for running the benchmark. (Quality)\n- A number of different SQL dialects are included, and the results between them are compared. (Quality)\n- Generally, the writing is clear and well-structured. (Clarity)\n- The design of SIX-GYM is original and interesting. (Quality, Originality)\n- SQL-fixing is a useful task with many useful applications (e.g. data scientist copilot). While a number of Text-to-SQL benchmarks already exist, there are no notable SQL-fixing benchmarks to the best of my knowledge. (Significance, Originality)\n\n### Weaknesses\n\n- It is not clear to me that SQL debugging is a significantly different task from Text-to-SQL generation from scratch. The analogy to code debugging is weak. While code debugging requires understanding a large amount of pre-existing code to fix a bug and then making a small change, SQL queries are typically very short, and an incorrect query can simply be rewritten from scratch. (Originality, Significance)\n- Microsoft SQL Server and Oracle are proprietary databases, unlike Postgres and MySQL which are open source databases. Including SQL Server and Oracle has benefits for industry users because these are widely used databases in industry, but it may make it more difficult for academic users to run the benchmark as they may not have access to these database platforms. The paper also incorrectly claims in Section 1 Introduction that SQL Server and Oracle are open source databases. (Quality, Significance)\n- I felt that the description of the SQL-Act agent in Section 5.1 is not clear enough. My understanding is that the agent’s actions are sending SQL queries, and the observations are the results of executing the queries. (Clarity)\n- It would be useful to include an example of SIX-GYM in the appendix, including a function plan F and the corresponding generated solution. (Clarity)\n- There are a number of minor language and typographical issues. A partial list is below. (Clarity)\n    - “SQLs” should be “SQL queries” in multiple places\n    - Appendix E.2 - “SQLServer” should be “SQL Server”\n    - Section 5.1 SetUp - “SetUp” should be “Setup”\n    - Section 6.2 Main Results - Remove the line break after “more standardized data management operations,”\n    - Section 6.5 Error Analysis - The long sentence starting with “It can be concluded” should be broken into multiple sentences.\n    - Section 6 Related Work and Appendix I Related Work have some repeated material.\n\nQUESTIONS:\n- Could you comment on how SQL debugging is different from Text-to-SQL generation from scratch? In particular, if you were to run an evaluation using BIRD- FIXER where the models were only given the user issue description but not the issue SQL, would the models still perform as well?\n- I would expect the SQL-Act agent to struggle more on instances that require database schema migrations or destructive write operations, because the agent isn’t able to “try out” possible queries by taking actions (which would mutate the database) as it would with read-only SELECT queries. Did you observe this in your experiments?\n- Did you try some of these possible alternatives for fine-tuning BIRD-FIXER?\n    - Fine-tuning on a small train split of BIRD-CRITIC instead of on the SIX-GYM synthetic dataset\n    - Retaining the generated functional plan in SIX-GYM instead of discarding it\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new and challenging SQL benchmark named BIRD-CRITIC, which covers a wide range of complex SQL tasks. It is designed to evaluate the ability of large language models (LLMs) to decompose, analyze, and reason over intricate instructions in order to generate correct SQL queries. To address common issues such as formatting or logical errors that arise from insufficient SQL understanding in current models, the authors also propose an auxiliary agent called BIRD-FIXER. This agent enhances the model’s SQL comprehension and performs post-hoc corrections on generated outputs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The proposed benchmark, BIRD-CRITIC, includes more complex and realistic SQL tasks derived from real user queries. Current mainstream reasoning models (e.g., O3-mini) struggle with these tasks, making this benchmark a valuable resource for the community to evaluate and improve LLMs on challenging SQL problems.\n\n2. The design of the BIRD-FIXER agent effectively corrects errors in the SQL generation process, including both syntactic and logical issues. This design makes it easier for future researchers to diagnose and improve model behavior in a more interpretable and modular way.\n\n3. The paper is well-structured and comprehensive. It includes detailed examples of benchmark tasks, data collection methodology, dataset statistics, baseline results from popular models, ablation studies on BIRD-FIXER, and a thoughtful categorization of failure cases. The analysis is thorough and informative.\n\n**Weaknesses:**\n\nThe approach used in BIRD-FIXER is conceptually similar to techniques already explored in other areas of LLM research, and thus lacks strong novelty. However, the adaptation of this idea to the SQL domain is still a meaningful contribution, especially given the practical challenges of SQL generation.\n\nQUESTIONS:\n1. Since your work aims to address the limitations of current models on complex SQL tasks, what do you think is the main reason these models struggle? Is it due to a lack of SQL-specific knowledge, or is it because these tasks resemble mathematical reasoning and require more advanced compositional abilities? If a model were equipped with strong SQL background knowledge, would your benchmark still offer meaningful challenges and insights for further improvement?\n\n2. Many of the tested models may not have been specifically optimized for SQL tasks. In general, SQL can be seen as one branch of code-related tasks. From your perspective, what makes SQL benchmarks necessary compared to other code benchmarks? What specific capabilities of LLMs do SQL tasks reveal that might be overlooked by more general code-oriented benchmarks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents first a benchmark (BIRD-CRITIC) which deals with repairing and debugging SQL-based programming problems. The authors manually curated problems from real user issues after applying them on a controlled environment. The authors also created a synthetic training dataset by synthetically generating new problems and solution using an LLM. The paper performs several rounds of evaluation and demonstrate that prior popular LLMs still struggle on the task of SQL-based debugging and shows the effect of fine-tuning on the synethtic training dataset has on performance\n\nSTRENGTHS AND WEAKNESSES:\nOverall the paper is well-written and the amount of contribution is strong\n\nstrengths \n- extensive description of how the benchmark is constructed\n- human-validated dataset construction with cross-validation to ensure high accuracy of the benchmark\n- the high number of testcases per each problem is also highly encouraging\n- proposed additional synthetic training dataset that can be further adopted and used by future work\n- strong amount of evaluation to analyze across different problem types and across many different models and model/agent setups\n\nweaknesses\n- its bit unclear what the exact quality of the synthetic dataset generated to create SIX-GYM is. For example, it would benefit the work if the authors selected several samples from the synthetic dataset and manually verified the quality of the benchmark\n- \"In our experiments, running Gemini-2.0-Flash with SQL-ACT on SIX-GYM produces just 1,254 successful trajectories\" from this sentence it is very unclear if the authors performed some level of rejection-sampling or just simple sample the model once?\n- The previous point also follows my next question: what exactly are the benefits of f-Plan Boosting? For example the authors claims that without f-Planning not all the trajectories can be successfully generated in the synthetic benchmark. However, is this due to the low number of sampled trajectories (i.e., lack of rejection sampling). I think a better benefit would be to show that f-Planning can arrive at a successful trajectory much sooner than regularly generating a trajectory. However this analysis is not shown in the paper\n\nQUESTIONS:\n1. Did the authors manually evaluate the quality of the trajectories/problems in the synthetic benchmark and see if they are of similar quality to the collected benchmark?\n2. What exactly are the benefits of f-Plan Boosting (i.e., Can it be used to generated very difficult trajectories from problems that the basic approach cannot?). Furthermore, what is the trajectory generation setting and does it use some level of rejection sampling scheme?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe contributions of this paper are fourfold: (1) BIRD-CRITIC Benchmark: A comprehensive benchmark for SQL issue debugging, which includes real-world tasks and custom evaluation scripts. (2) SIX-GYM Environment: An automated training environment that generates large-scale, executable issue-solution datasets by employing the SQL-Rewind strategy. (3) f-Plan Boosting: A method that extracts high-level debugging plans from SQL solutions, thereby significantly increasing the number of successful training trajectories. (4) BIRD-FIXER Agent: An open-source agent that leverages SQL-ACT and f-Plan Boosting, achieving state-of-the-art performance on SQL debugging tasks.\nBy providing a method for constructing training data, this paper reduces the consumption of human and material resources in data construction. Additionally, the proposed benchmark allows for the evaluation of LLMs' capabilities in SQL issue debugging, thereby contributing to the advancement of debugging capabilities in this area.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. Current research in the Text-to-SQL field mainly focuses on evaluating the ability of large language models (LLMs) to accurately convert natural language queries into SQL statements. However, there is still a lack of corresponding datasets for the evaluation of LLMs' SQL debugging capabilities. The dataset proposed in this paper fills the gap in this field in this aspect.\n2. This dataset is reasonably constructed and based on the relational database in the BIRD-SQL development set. This database has a complex structure and can simulate the real-world database environment to a certain extent. The annotation process was accomplished through the collaboration of two professional teams, and the annotation quality was ensured through cross-validation. The manual annotation method was adopted to further enhance the accuracy and reliability of the data.\n3. The set task is highly challenging. Despite the continuous improvement of LLMs capabilities, the BIRD-CRITIC dataset still poses significant challenges to existing models. For instance, the best-performing model O3-Mini-2025-01-31 has a success rate of only 38.87% in the PostgreSQL subtask and 33.33% in the multi-dialect subtask, indicating considerable room for improvement. \n4. Furthermore, the article also proposes a new method for automatically constructing training data annotations, and through experiments, it is verified that this method has a significant effect in improving the SQL debugging ability of LLMs.\n\nWeakness：\n1. The experimental section could be more robust. Although this paper claims that the proposed training method is model-architecture-agnostic, it only trains four models, and the training does not cover the thinking model. This is not sufficient to demonstrate the effectiveness of the training data.\n2. The paper mentions the differential distributions of SQL dialects within the respective training corpora of these models, which has an impact on the models' performance across different dialects. However, in the paragraph at line 300, it is stated that the model is only trained on PostgreSQL and is extensible, but there are no experiments to show whether training on other dialects would lead to improvements.\n\nQUESTIONS:\n1. Whether the training data for effective reasoning model?\n2. 2. BIRD-FIXER was only fine-tuned on PostgreSQL trajectories. If fine-tuned on multi-language trajectories, could the model capabilities be further enhanced?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}