Title: Beyond Text-to-SQL: Can LLMs Really Debug Enterprise SQL?
Primary area: datasets and benchmarks
Keywords: SWE, LLM, benchmark, SQL, Bugfixing, Agent

Abstract: SQL is the core of data analysis and engineering across industries, powering large-scale workflows for data extraction, transformation, and loading. However, in enterprise-level scenarios, it is challenging to generate fully correct SQL code in a single attempt—even for experienced developers or advanced \ttsql LLMs. Multiple iterations of debugging are usually required, yet LLMs often get lost in multi-turn correction.
To address this gap, we introduce \textbf{\ourbench}, the first benchmark designed for enterprise-level SQL reasoning and debugging. Our benchmark is built upon two key innovations: (1) an \textbf{automated construction workflow} that employs reverse engineering to systematically inject realistic bugs into large-scale SQL code, enabling scalable and diverse benchmark generation; and (2) an \textbf{execution-free evaluation framework} tailored for enterprise settings, providing fast, accurate, and resource-efficient assessment.
%
\ourbench comprises $469$ \ourbenchsyn queries featuring syntax errors with explicit error messages, and $516$ \ourbenchsem queries targeting semantic errors where SQL fails to meet the requirement. The SQLs are substantially complex, averaging over $140$ lines with abstract syntax trees of high complexity (average width $>11$, depth $>8.7$).
%
We evaluate nearly $30$ LLMs on \ourbench. Even state-of-the-art reasoning models struggle: Claude-4-Sonnet achieves only 36.46\% success on \ourbenchsyn and 32.17\% on \ourbenchsem. Most models fail to reach 20\% success, underscoring the significant gap between current LLM capabilities and the demands of enterprise SQL debugging.
%
To bridge this gap, we systematically \textbf{explore four potential solution strategies and conduct extensive experiments} to evaluate and compare their effectiveness.  Our experiments not only highlight the challenges but also shed light on effective strategies for advancing SQL debugging with LLMs.