# Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models
## 5 Results
### Ablation Studies  
To quantify contributions of logical alignment, schema enhancement, and prompt guidelines, we conducted ablation studies across models and prompts on our refined dataset (146 pairs), measuring accuracy and error counts. Starting from full refinement, we incrementally removed components down to the original BIRD subset baseline. Table 2 consolidates accuracy and error distributions. Subsequent analyses explore impacts on reasoning, generalization, and error reduction. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{5}{c}{**Refined Dataset (New Prompt)**} & \multicolumn{5}{c}{**Refined Dataset (Old Prompt)**} & \multicolumn{5}{c}{**Original BIRD**} \\ \cline{2-13}  & **Test 1** & **Test 2** & **Test 3** & **Test 4** & **Test 5** & **Avg** & **Test 1** & **Test 2** & **Test 3** & **Test 4** & **Test 5** & **Avg** & **Avg** & **Avg** \\ \hline \multirow{2}{*}{\begin{tabular}{c} GPT-do \\ Mini \\ \end{tabular} } & 42.47\% & 46.58\% & 39.73\% & 41.78\% & 37.67\% & 41.65\% & 40.41\% & 32.88\% & 39.73\% & 34.93\% & 36.30\% & 36.85\% & 34.12\% & 31.25\% \\ \hline \multirow{2}{*}{\begin{tabular}{c} GPT-do \\ Mini \\ \end{tabular} } & 47.94\% & 53.42\% & 51.36\% & 53.42\% & 50.00\% & 51.23\% & 50.00\% & 50.00\% & 52.05\% & 42.46\% & 52.74\% & 49.45\% & 43.68\% & 41.10\% \\ \hline \multirow{2}{*}{\begin{tabular}{c} Qwen-2.5- \\ Instruct \\ \end{tabular} } & 46.58\% & 42.47\% & 45.21\% & 41.10\% & 45.89\% & 44.25\% & 43.84\% & 43.84\% & 44.52\% & 46.57\% & 36.30\% & 43.01\% & 38.90\% & 37.12\% \\ \hline \multirow{2}{*}{\begin{tabular}{c} LLAMA-3 \\ \textbackslash{} DB \\ \end{tabular} } & 47.26\% & 50.00\% & 46.58\% & 45.21\% & 47.95\% & 47.80\% & 46.58\% & 47.95\% & 45.21\% & 44.52\% & 43.84\% & 45.62\% & 41.23\% & 39.80\% \\ \hline \multirow{2}{*}{\
\begin{tabular}{c} DFSK-\textbackslash{} \\ O1-Preview \\ \end{tabular} } & 48.63\% & 50.00\% & 51.37\% & 47.26\% & 48.63\% & 49.10\% & 47.26\% & 48.63\% & 50.00\% & 46.58\% & 46.58\% & 47.80\% & 42.80\% & 41.50\% \\ \hline \multirow{2}{*}{
\begin{tabular}{c} O1-Preview \\ \end{tabular} } & 77.40\% & 79.45\% & 76.03\% & 80.14\% & 78.08\% & 78.22\% & 75.34\% & 77.40\% & 74.66\% & 76.71\% & 75.34\% & 75.89\% & 69.50\% & 67.80\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Average accuracy of six models across five test rounds on the refined dataset (new and old prompts) compared to the original bird subset. The refined dataset consistently outperforms BIRD, with new prompt accuracies ranging from 41.65% (GPT-4o-mini) to 78.22% (o1-preview), and old prompt accuracies ranging from 36.85% (GPT-4o-mini) to 75.89% (o1-preview). \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Configuration**} & \multicolumn{5}{c}{**Accuracy (\%)**} & \multicolumn{5}{c}{**Errors by Type (GPT-to, New Prompt)**} \\ \cline{2-13}  & **GPT-4o (New)** & **GPT-4o (Old)** & **GPT-4o-Mini (New)** & **O1-Preview (New)** & **T1** & **T2** & **T3** & **T4** & **T5** & **T6** & **T7** \\ \hline Full Refinement & 51.23 & 49.45 & 41.65 & 78.08 & 12 & 5 & 5 & 10 & 24 & 8 & 3 \\ w/o Prompt Guidelines & 49.80 & 47.67 & 40.00 & 76.85 & 12 & 6 & 6 & 10 & 25 & 9 & 3 \\ w/o Schema Enhancement & 48.12 & 46.30 & 38.90 & 74.52 & 14 & 8 & 5 & 13 & 26 & 8 & 3 \\ w/o Logical Alignment & 45.06 & 43.15 & 36.25 & 70.89 & 15 & 10 & 6 & 12 & 32 & 9 & 4 \\ Original BIRD Subset & 43.68 & 41.10 & 34.12 & 68.49 & 16 & 12 & 7 & 14 & 35 & 10 & 4 \\ \hline \(\Delta\) from Full (GPT-to, New) & - & \(-\)1.78 & \(-\)9.58 & +26.85 & +4 & +7 & +2 & +4 & +11 & +2 & +1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Ablation study combining accuracy across models and prompt types with error counts by type for gpt-4o (new prompt). T1–T7 denote error types (T1 Misinterpretation, T2 Forgetting, T3 Syntax, T4 Data Processing, T5 Query Logic, T6 Edge Cases, T7 False Assumptions). Impact of Logical Alignment: Removing logical alignment caused the largest accuracy drop across all models (\(-6.17\%\) to \(-7.19\%\)).