# Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models
## 1 Introduction  
The field of Text-to-SQL, which enables natural language querying of databases, has evolved significantly, propelled by neural architectures and large language models (LLMs). Early systems relied on inflexible rule-based or template-based methods, succeeded by neural approaches like Seq2 SQL [1] and SQLNet [2], which introduced reinforcement learning and sequence-to-sequence models to improve SQL generation [3]. Modern solutions, such as DAIL-SQL [4] and SQL-PaLM [5], leverage LLMs with pretraining on vast text corpora and dynamic context learning, achieving high accuracy on benchmarks. While SQL's structured format enhances LLM efficiency over unstructured data [6], challenges persist in handling complex queries, ensuring cross-domain generalization--where performance consistency falters across diverse domains--and maintaining accuracy amid ambiguous inputs [7].  
Zero-shot prompting, where LLMs perform tasks without task-specific training or examples, has gained traction in Text-to-SQL. [PERSON] et al. [8] demonstrated GPT-3's capability, a method now vital for complex queries with models like GPT-4 [9] and LLaMA [10]. Studies like [PERSON] et al. [11] further highlight LLMs' zero-shot Text-to-SQL potential, though limitations in logical reasoning persist. However, no benchmark specifically targets zero-shot scenarios robustly. The BIRD dataset [12], with 12,751 question-SQL pairs across95 databases and 37 domains, is a complex option, but approximately 30% of its queries suffer from logical misalignments (e.g., mismatched timestamps as shown in Figure 1), data inconsistencies, or grammatical errors [12], impairing evaluation. Such 'dirty' data--common in real-life benchmarks--hamper LLM reasoning evaluation, as logical clarity is critical in zero-shot settings where fine-tuning is absent [13]. Current Text-to-SQL benchmarks exhibit growing query ambiguity [14], rendering them less suitable for testing LLM logic. For instance, Spider's [14] structured questions lack the complexity of real-world multi-table joins [14], a gap widened as LLMs master simpler datasets [15]. Conversational benchmarks like SParC [16] and CoSQL [17] focus on multi-turn interactions but neglect broader zero-shot reasoning assessment.  
This study addresses the critical need for a reliable benchmark to evaluate LLM zero-shot reasoning in Text-to-SQL. We refine the BIRD dataset [12] by meticulously selecting 146 complex, logically sound question-SQL pairs across 11 domains. Our refinement process involves correcting SQL errors (e.g., fixing timestamp mismatches), aligning questions with SQL outputs, and enriching table schemas with contextual information (e.g., data ranges, typical values) to minimize ambiguity and reliance on external knowledge. This curated benchmark facilitates a rigorous assessment of LLM zero-shot reasoning on complex, real-world-like queries while eliminating the noise present in the original data. We evaluate six state-of-the-art LLMs (GPT-4o, GPT-4o-Mini, Qwen-2.5-Instruct, Llama 370b, DPSK-v3, O1-Preview) using this refined benchmark and specific prompt strategies, achieving average accuracies ranging from 41.65% to 78.08%. This work provides a robust foundation for evaluating and advancing reliable Text-to-SQL systems, supporting practical NLP-driven database applications.