# Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models
## 4 Methods
### Dataset Creation
To resolve the ambiguity, the description of data and the terms in question are transformed into more detailed forms (e.g., Question 1028), as shown in Figure 3.1  
Avoid Ambiguities in Questions Leading to Misaligned SQL:  
Vague phrasing and ambiguous terms like percent or rate are sometimes inconsistently addressed in SQL and can eventually cause misinterpretations. Avoid Overly Complex SQL for Simple Queries:  
Although simpler answers exist, SQL queries can sometimes be unnecessarily complex. The solution is to compare the SQL generated by LLMs and check if they work better. If the SQL generated by LLMs works better in execution speed, etc., replace the original SQL with the generated SQL, as presented before. Avoid Redundancies or Missing Logic:  
SQLs may contain redundant operations or omit key constraints, leading to incomplete or incorrect results (e.g., Question 835). Adding and removing appropriate terms from the SQL can solve such issues, as shown in Figure 3.2  
Avoid Inconsistent Use of Data Representations:  
SQL queries sometimes omit data inconsistencies or ambiguities, which may misalign with the questions' intent, as in Question 83 (presented before). To solve this issue, the data description document is carefully checked to ensure no ambiguous, wrong, or missing data descriptions appear in the description documents, as shown in Figure 3.3  
Avoid over-testing the same issue:  
Original questions sometimes examine the same point too frequently. This problem can be solved by reducing the proportion of test questions with the same test point, i.e., by selecting only some of them to be added to the new data set. Adding Information about Data into Table Schema:  
This benchmark focuses on the logic of the LLM, so ensuring the integrity of the data provided to the LLM before testing is critical. To this end, the data description file has been modified by adding two columns of data description, Data Range and Typical  
Figure 3.4 Example of omitted constraints in SQL from the BIRD dataset (Question 835). The question asks for the percentage of ‘good’ superheroes in Marvel Comics, but the original SQL’s INNER JOIN excludes heroes without alignment data. The new answer uses LEFT JOIN to include all heroes, aligning with the question’s intent, as explained in the reasoning. Data, to avoid problems caused by unknown data in the dataset to the LLM, as shown in Figure 3.5 Data ranges are obtained by capturing the maximum and minimum values in the corresponding columns to give the LLM an idea of the range of the data; typical data are obtained by randomly selecting 5 values in a field to simulate the behavior of a quick read of the database and to give the LLM a better understanding of the data. Assigning Difficulty Level Properly:  
Most test questions were assigned the same difficulty as the original to show fidelity to the original data. Only those test questions that evidently contradicted their own difficulty labels were modified, as shown in Question 322 in the 'Redundant and Unreasonable Difficulty Labels' section. Figure 3.6 Example of correcting data description inconsistency for Question 83 in the BIRD dataset. The original description specifies ’V’ and ’N’ for the ‘Magnet’ field (originally highlighted), but the actual data use ’0’ and ’1’. The new description aligns with the data, resolving the ambiguity to ensure SQL queries (e.g., T2. Magnet = 1) match the question’s intent without misinterpretation. Figure 3.7 Example of enhanced data description for the ‘CDSCode’ column in the ‘frmp’ table (California schools domain) from the refined BIRD dataset. The Data Range shows the minimum and maximum values (’01100170109835’ to ‘58727695838305’), while Typical Data provides five random samples, aiding LLMs in understanding data structure and range. This modification ensures data integrity for logical reasoning evaluation.