SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: The Execution Accuracy (EX) and Valid Efficiency Score (VES) are presented for both the ChatGPT model and its version with grounding (KG) for external knowledge evidence, taking into consideration development and testing datasets.
B.2 Annotation Entrance
Annotation Platform and Compensation. The data is collected from Alibaba-Appen $^{\S}$ , an internal version. Each Question annotator receives a $0.6 reward for each validated question, while SQL annotators earn $1 per SQL contribution. We also invite text-to-SQL experts and professors to join to check and annotate external knowledge evidence without compensation. There are ~1340 SQLs confirmed per week.
$\S$  https://appen.com/crowd-2/#crowd
Text-to-SQL Experts. The three full-time text-to-SQL experts in this project are: (1). A database research scientist who's published over 20 top DB conference papers (e.g., SIGMOD, VLDB). (2). A PhD student with research interests in text-to-SQL, who achieved state-of-the-art results on text-to-SQL open challenges. (3). A DBA engineer with more than 10 years of experience in text-to-SQL applications for both B2B and B2C businesses.
Question Annotation Entrance. We hire a group of native speakers of English with degrees above the bachelor's level and database-related knowledge to ask a variety of natural language questions regarding the values of databases. To fulfill this objective, we have adopted the following procedure: (1). ER diagrams and database description files are documented to assist the annotators in understanding the databases; (2). we present the annotators with three databases from different domains and require them to generate 10 questions for each database; (3). these questions are then assessed by 3 text-to-SQL experts applying predefined rules. Those questions earning at least two votes are marked as valid. Only annotators capable of generating no less than 8 valid questions per database are preserved. As a result, 11 native speakers contribute questions to BIRD.
SQL Annotation Entrance. With the purpose of enhancing the quality of our SQL queries, we assemble a team of skilled data engineers and database students. The team undergoes rigorous testing through the text-to-SQL evaluation process, which assesses their capability of generating SQL queries for a variety of questions facing different domains of databases. Each annotator is asked to answer 10 questions, and only those who score at least 9 out of 10 will be qualified to annotate SQL queries for BIRD.
B.3 Question Distribution
Figure 8 contains the detailed question types and their examples.
B.4 Experiment Details
FT-based Models. T5 is a strong and versatile pre-trained language model (PLM) for text-to-text generation that has achieved state-of-the-art performance in a variety of semantic parsing tasks, including text-to-SQL. We concatenate the question with serialized database schema as input [40, 49, 41]. And SQL can be fetched in an end-to-end fashion by easily fine-tuning. While seq2AST-based methods [43, 5] are also effective in text-to-SQL, actually their grammar rules utilized during decoding are constrained on specific datasets [25]. We implement our codes mainly based on the hugging-face transformers library  $\mathbb{Q}$ . We set the max input length as 1024, the generation max length as 512, and the batch size as 32. We also adopt Adafactor as our primary optimizer with a linear decayed learning rate of 5e-5. All experiments are conducted on one NVIDIA Tesla A100 80GB, which is available for most research centers. We set the random seed as 1 for all runs of FT-based models since 1 is an optimal seed proven by previous SOTA models [27, 49].
ICL-based Models. Codex (code-davinci-002) and ChatGPT (gpt-3.5-turbo) are popular and powerful large-scale pre-trained language models (LLMs) for code generation driven by ICL. They can produce multiple types of codes, including SQL, from human instructions without additional training. We employ programming-based prompts, as described in [39], to collect results by calling the API. Also, we choose the Azure OpenAI API to align the codes with other variants of LLMs. Given that models are not allowed access to unseen databases and ground-truth SQLs in the evaluation set, a zero-shot generation strategy is the most appropriate. Moreover, to investigate the impact of multi-step reasoning of LLMs on BIRD, we implement the Chain-Of-Thought (COT) technique [48] by easily adding the prompt sentence "Let's think step by step." before the generation of SQLs [21]. However, we find out the output of ChatGPT is too uncertain with many unexpected explanations, thus we provide a 1-shot pseudo example for ChatGPT to learn the procedure of thinking and output format. The detailed prompt design is shown in Figure 9. In order to minimize the randomness of results, we set the temperature as 0 to ensure reproduction.
Knowledge Fusion. In the baseline implementation, we naively concatenate the knowledge evidence sentences with questions and database schemas, but we can observe a significant improvement
$\mathbb{I}$  https://huggingface.co/
Question Type	Sub Type	Question / SQL	Percentage
Fundamental Type	Match-based	How many gas stations in CZE has Premium gas?	83.9 %
SELECT COUNT (GasStationID) FROM gasstations WHERE Country = 'CZE' AND Segment = 'Premium'
Ranking	What are the titles of the top 5 posts with the highest popularity?	20.3 %
SELECT Title FROM posts ORDER BY ViewCount DESC LIMIT 5
Comparison	How many color cards with no borders have been ranked higher than 12000 on EDHRec?	16.7 %
SELECT COUNT (id) FROM cards WHERE edhrecRank > 12000 AND borderColor = 'borderless'
Counting	How many of the members' hometowns are from Maryland state?	30.4 %
SELECT COUNT (T2(member_id) FROM zip_code AS T1 INNER JOIN member AS T2 ON T1.zip_code = T2.zip WHERE T1.state = 'Maryland'
Aggregation	What is the average height of the superheroes from Marvel Comics?	15.7 %
SELECT AVG (T1.height_cm) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1publisher_id = T2.id WHERE T2publisher_name = 'Marvel Comics'
Reasoning Type	Domain Knowledge	Name the ID and age of patient with two or more laboratory examinations which show their hematoclit level exceeded the normal range.	23.6 %
SELECT T1.ID, STRFTIME('%Y', CURRENT_TIMESTAMP) - STRFTIME('%Y', T1.Birthday) FROM Patient AS T1 INNER JOIN Laboratory AS T2 ON T1.ID = T2.ID WHERE T1.ID IN ( SELECT ID FROM Laboratory WHERE HCT > 52 GROUP BY ID HAVING COUNT (ID) >= 2 )
Numeric Computation	Among the posts with a score of over 20, what is the percentage of them being owned by an elder user?	24.5 %
SELECT CAST (SUM(IIF (T2.Age > 65, 1, 0)) AS REAL) * 100 / count (T1.Id) FROM posts AS T1 INNER JOIN users AS T2 ON T1.OwnerUserID = T2.Id WHERE T1.Score > 20
Synonym	How many clients opened their accounts in Jesenik branch were women ? (female)	7.2 %
SELECT COUNT (T1.client_id) FROM client AS T1 INNER JOIN district AS T2 ON T1district_id = T2.district_id WHERE T1.person = 'F' AND T2.A2 = 'Jesenik'
Value Illustration	Among the weekly issuance accounts, how many have a loan of under 200000?	70.1 %
SELECT COUNT (T1_account_id) FROM loan AS T1 INNER JOIN account AS T2 ON T1_account_id = T2_account_id WHERE T2 Frequency = 'POPLATEK TYDNE' AND T1.amount < 200000
Figure 8: Questions in the BIRD contain two main categories. The Fundamental Type of questions are comparable to other text-to-SQL benchmarks. The Reasoning Type of questions requires external knowledge grounding to answer.
by this easy method. A more complicated and effective strategy of knowledge grounding for ChatGPT and T5 would be an important future topic. The knowledge evidence sentences are concluded to the external knowledge provided by annotators as described in Section 3.3.
ICL Prompt Input
1-shot pseudo example
CREATE TABLE singer
singer_id TEXT not null
primarykey,
nation TEXT not null,
sname TEXT null,
dname TEXT null,
cname TEXT null,
age INTEGER not null.
year INTEGER not null.
birth year INTEGER null.
salary, REAL null
city TEXT null
phone number INTEGER null
External Knowledge: age = year - birth year;
-- Using valid SQLite and understaging External
Knowledge, answer the following questions for
the tables provided above.
The final SQL is: Let's think step by step.
1. referring to external knowledge, we need to filter singers 'by year' - 'birth_year' > 27; 2. we should find out the singers of step 1 in which nation = 'US', 3. use COUNT() to count how many singers. Finally the SQL is: SELECT COUNT(*) FROM singer WHERE year - birth_year > 27;
Create DDL Prompt
CREATE TABLE customers
CustomerID INTEGER UNIQUE not null
primary key,
segment TEXT null,
Currency TEXT pull
CREATE TABLE gasstations
GasStationID INTEGER UNIQUE not null
ChainID INTEGER null,
Country TEXT null
Segment TEXT null
CREATE TABLE yearmonth
CustomerID INTEGER not null,
Date INTEGER not null,
Consumption REAL null.
primary key (CustomerID, Date).
foreign key (CustomerID) references
customers(CustomerID)
Knowledge Prompt
-- External Knowledge: August of 2012 means
Date contains '201208' in the yearmonth.date
of the database; Price per unit of product =
Price / Amount
Task Instruction
-- Using valid SQLite and understading External
Question Prompt
-- For all the people who paid more than 29.00
per unit of product id No 5 Give their
consumption status in the August of 2012
1-shot COT Prompt
ChatGPT COT Steps
1. referring to external knowledge, we need
to calculate the price per unit of product id
No.5, which is Price / Amount: 2, we should
filter the people who paid more than 29.00
per unit of product id No.5: 3, we should find
out the consumption status of the people in
August of 2012
SQL Output
Finally the SQL is:
SELECT yearmonth.Consumption FROM transactions 1k JOIN yearmonth ON transactions 1k.CustomerID = yearmonth.CustomerID
AND transactions 1k.Date = yearmonth.Date WHERE transactions 1k.ProductID = 5 AND transactions 1k.Price /
transactions_1k.Amount > 29.00 AND yearmonth.Date = '201208';
Figure 9: The detailed prompt design for implementation of ChatGPT + KG + COT.
B.5 Efficiency Analysis Details
Two strategies for performing text-to-efficient-SQL are presented in Figure 10. Examples show that both two-stage optimization and embodied databases can help semantic parsings generate more efficient SQLs.
B.6 Error Analysis Details
Figure 11 presents a detailed analysis of errors made by ChatGPT.
B.7 Evaluation Details
During double-blind annotation in BIRD, we encountered numerous ambiguous issues that led to mismatches, predominantly due to unclear user intents. The most serious ambiguity is the use of "DISTINCT". Some annotators believe it should present only unique values, such as names, and cities, while others argue that it should be used only when questions explicitly mention "different" or "distinctive". Therefore, we use HashSet rather than List to compare final results since HashSet disregards row order and automatically filters repetitive rows to reduce this ambiguity. However, this may result in false positives for questions utilizing "ORDER BY." We identify three "ORDER BY" usage scenarios in BIRD: 1) Rank-based questions (e.g., "Show me the top 5 students according to their math scores"): The order is less important as long as the results contain the correct students. 2) Superlative questions: (e.g., "List the longest river in the USA"): The answer typically contains only one item (or tied results), so the impact is minimal. 3) Questions requiring a specific order (e.g., "Show me the top five students based on their math scores in descending order"): This scenario explicitly requires correct ordering and may lead to false positives. However, such instances are uncommon, accounting for less than  $1\%$  of BIRD.
Query Rewriting
Ex1.1 Question:List out the age of users who located in Vienna, Austria obtained the badge?
ChatGPT SQL:SELECT Age FROM users WHERE Location = 'Vienna, Austria' AND Id IN (SELECTUserId FROM badges)
Optimized SQL: (time-saving percentage: 99.92%)SELECT u.Age FROM users AS u INNER JOIN badges AS b ON u.Id = bUserID WHERE u.Location = 'Vienna, Austria'
Take Away:By applying a JOIN operation instead of a subquery with IN can improve efficiency, as the database may execute the JOIN and filtering processes concurrently in just one operation without the need to store the intermediate results to filter primary query.
Ex1.2 Question:How many of the members' hometowns are from Maryland state?
ChatGPT SQL:SELECT COUNT(*) FROM member INNER JOIN zip_code ON member.zip = zip_code.zip_code WHERE zip_code.state = 'Maryland'
Optimized SQL: (time-saving percentage: 67.93%)SELECT COUNT (member.member_id) FROM member INNER JOIN zip_code ON member.zip = zip_code.zip_code WHERE zip_code.state = 'Maryland'
Take Away:Utilizing the COUNT function on a NOT-NULL column, as opposed to COUNT(*), can increase time efficiency. This rewritten SQL enables the database to count NOT-NULL values within a single column, rather than compute all rows including those with NULL values. Usually, the primary key column is selected as this NOT-NULL column.
Ex1.3 Question:Who is the owner of the account with the largest loan amount?
ChatGPT SQL:SELECT c.client_id FROM client c INNER JOIN disp d ON c.client_id = d.client_id INNER JOIN loan 1 ON d_account_id = 1account_id ORDER BY 1.amount DESC LIMIT 1
Optimized SQL: (time-saving percentage: 62.39%)SELECT c.client_id FROM client c INNER JOIN disp d ON c.client_id = d.client_id INNER JOIN loan 1 ON d/account_id = 1account_id WHERE 1.amount = ( SELECT MAX(amount) FROM loan)
Take Away:In an unindexed environment, employing the MAX function can potentially yield faster results since it avoids the need for sorting, which could run against a large table.
Adding Indexes to Database
Ex2.1 Question:How many accounts are there in the district of ''Pisek''?
ChatGPT SQL:SELECT COUNT(*) FROM account a INNER JOIN district d ON adistrict_id = ddistrict_idWHERE d.A2 = 'Pisek'
Added Indexes: (time-saving percentage: 87.27%)CREATE INDEX account district_id index ON account (district_id);CREATE UNIQUE INDEX district district_id uindex ON district (district_id);
Take Away:Adding indexes into a database can significantly increase the speed of SQL queries because it creates a data structure that enables the database engine to quickly locate rows that match specific criteria instead of scanning the entire table.
Figure 10: Two possible solutions and explanations to improve efficiency are presented. The first batch of examples shows how to optimize SQL efficiency by rewriting SQL based on rules. The last example is to show that adding indexes to databases can also improve SQL efficiency without rewriting them.
B.8 VES Details
Regarding  $E$ , in our experiment, we consider time as the main metric to represent efficiency, where  $E \in (\epsilon, 30s)$ . Here,  $\epsilon$  is a small positive constant to prevent floating-point overflow. The single  $E$  is not stable due to machine status. Lower  $E$  refers to faster execution speed, which is more efficient.
Concerning  $R$ , it represents a normalized efficiency ratio between human-annotated SQL queries and predicted SQL queries to reduce the influence of machine status. The stability of this metric is ensured by running this computation 100 times for each example, filtering outliers, and subsequently
Wrong Schema Linking  $(41.6\%)$
Question:
List the names of schools with more than 30 differences in enrollments between K-12 and ages 5-17. Please also give the full street address of the schools.
Evidence:
Difference in enrollment = 'Enrollment (K-12)' - 'Enrollment (Ages 5-17)'
Ground Truth:
SELECT T1.School, T1.StreetAbr FROM schools AS T1 INNER JOIN frpm As T2 ON T1.CSDCode = T2.CSDCode WHERE T2.Enrollment (Ages 5-17) > 30
ChatGPT SQL:
SELECT s.School, s.Street, s.City, s.zip FROM schools s JOIN frpm s ON s.CSDCode = f.CSDCode WHERE f.Enrollment (K-12) - f.Enrollment (Ages 5-17) > 30	Question:
List out the accounts who have the earliest trading date in 1995.
Evidence:
False
Ground Truth:
SELECT account_id FROM trans WHERE STRTIME('6Y', date) = '1995' ORDER BY date ASC LIMIT 1
ChatGPT SQL:
SELECT account_id FROM account WHERE date = (SELECT MIN(date) FROM account WHERE date LIKE '1995')
Misunderstanding Knowledge Evidence (17.6%)
Question:
On average, how much did the Student_Club spend on
food for the typical event in the past?	Question:
Name elementary schools in Riverside which the average
of average math score for SAT is greater than 400, what is
the funding type of these schools?
Evidence:
DIVIDE(SUM(spent), COUNT(spent)) where category = 'Food'; event in the past' means event_status = 'Closed'	Evidence:
Average of average math = sum(average math scores) / count(schools)
Ground Truth:
SELECT T1.sname, T2. Charter Funding
Type' FROM sutcores AS T1 INNER JOIN
frpm... HAVING CAST(SUM(T1.AvgScrMath)
AS REAL) / COUNT(T1.cds) > 400
Ground Truth:
SELECT SUM(spent) / COUNT(spent) FROM budget WHERE category = 'Food' AND event_status = 'Closed'
ChatGPT SQL:
SELECT DIVIDE(SUM(spent), COUNT(spent)) FROM budget WHERE category = 'Food' AND event_status = 'Closed'	ChatGPT SQL:
SELECT s.School, s.FundingType FROM
schools s ... AND sc.rType = 'E' AND
sc.AvgScrMath > 400 GROUP BY s.School,
s.FundingType
Misunderstanding Database Content  $(40.8\%)$
Question:
What is the average math score of the school with the lowest average score for all subjects, and in which county is it located?
Evidence:
Average score for all subjects can be computed by AvgScrMath + AvgScrRead + AvgScrWrite
AvgScrMath:
SELECT T1. AvgScrMath, T2. County FROM satscores AS T1 INNER JOIN schools AS T2 ON T1.cds = T2.CDScode WHERE T1.AvgScrMath IS NOT NULL ORDER BY T1.AvgScrMath + T1.AvgScrRead + T1.AvgScrWrite ASC LIMIT 1
ChatGPT SQL:
SELECT cname, AvgScrMath FROM satscores
JOIN schools ON satscores.cds =
schools.CDScode WHERE satscores.type = 'S' ORDER BY (AvgScrMath + AvgScrRead + AvgScrWrite) ASC LIMIT 1	Question:
What is the average lap time in milliseconds of all the lap records set on the various circuits in Italy?
Evidence:
average = AVG(milliseconds)
Ground Truth:
SELECT ST(SUM(T1.milliseconds) AS
BEN1 / COUNT(T1.laps) FROM laptimes AS
T1 INNER JOIN races AS T2 ON T1.raceId
= T2.raceID INNER INNER JOIN circuits AS T3
on T2.circuitid = T3.circuitld WHERE
T3.country = 'Italy'
ChatGPT SQL:
SELECT AVG(milliseconds) FROM
lap records WHERE circuit IN ('Monza',
'Imola', 'Mugello', 'Misano')
Syntax Error  $(3.0\%)$
Question:
How many meeting events were held in 2020?	Question:
How much more was customer 7 consuming in April 2013 than customer 5?
Evidence:
meeting events refers to type = 'Meeting'; held in 2020 refers to YEAR(event_date) = 2020	Evidence:
April 2013 refers to '201304' in the yearmonth date
Ground Truth:
SELECT COUNT(event_id) FROM event WHERE type = 'Meeting' AND STRING('%Y', COLUMN) = '2020'	Ground Truth:
SELECT SUM(IF(CustomerID = 7, Consumption, 0)) - SUM(IF(CustomerID = 5, Consumption, 0)) FROM yearmonth WHERE Date = '201304'
ChatGPT SQL:
SELECT COUNT(*) FROM event WHERE type = 'Meeting' AND YEAR(event_date) = 2020	ChatGPT SQL:
SELECT(*), (SELECT Consumption FROM yearmonth WHERE CustomerID = 7 AND Date 1 = 201304) - (SELECT Consumption FROM yearmonth WHERE CustomerID = 5 AND Date = 201304)
Figure 11: 4 major types of error cases are presented. Some cases are shortcuts for better presentation.
computing the average. Considering the rapid advancement of technology, it is impractical to anticipate the fastest SQL performance. Currently, the range for the efficiency ratio,  $R$ , is defined as  $R \in (0, +\infty)$ .  $E(\hat{Y}_n)$  (efficiency of predicted SQL) is much lower than  $E(\hat{Y})$  (efficiency of ground truth SQL according to EX), then the relative efficiency score  $R$  will be increased. In short, higher  $R$  refers to higher efficiency.
When measuring VES, we run 100 times for each SQL in the same CPU and evaluate average results after dropping the outliers. The STD of VES on dev set and test set after 10 trials are 0.043 and 0.025 respectively. We detect outliers in the following procedures:
1. Compute the mean and standard deviation of the dataset.
2. Then calculate the lower threshold as mean  $-3 \times$  standard_deviation and the upper threshold as mean  $+3 \times$  standard_deviation.
3. Statistically, approximately  $99.7\%$  of the data points fall within 3 standard deviations of the mean.
B.9 Human Performance Collection
The procedure of collecting human performance is still rigorous. During the annotation, all data is divided into 10 batches for better management and error tracks by experts. The first 8 batches of data are the final training data and dev data for public use, and the remaining 2 batches of data are used for testing. We consider the annotation of the first 8 batches of data as a learning process for SQL annotators since their erroneous SQLs could be fixed by experts and learn how to generate good-quality SQLs for this task. Then their first scores on an examination, conducted by testing set from the final two batches can be viewed as human performance since we don't interrupt and assist them during the examination and all errors are preserved. After testing, we proceed with the following double-blind SQL annotation procedures as Section 3.4 to correct SQLs for these data by a discussion with experts. And SQLs after the second round of double-blind annotation are collected as ground truth.
B.10 Distribution of Open-source Databases
The databases in BIRD are all in accordance with one of following licenses:
Public Domain Public Domain Mark
A public domain license refers to a legal designation that allows intellectual property, such as creative works or inventions, to be freely used, shared, and built upon by anyone without restrictions. When a work is in the public domain, it is no longer protected by copyright, patent, or trademark laws.
CC-BY Creative Commons Attribution 4.0 International
This license is one of the open Creative Commons licenses and allows users to share and adapt the dataset so long as they give credit to the creator.
CC-BY-SA Creative Commons Attribution-ShareAlike 4.0 International
This license is one of the open Creative Commons licenses and allows users to share and adapt the dataset so long as they give credit to the creator and distribute any additions, transformations, or changes to the dataset under this license.
GPL General Public License
The GPL was created by the Free Software Foundation (FSF) and is also known as the GNU GPL, as it is used by the GNU Project. And it allows users to use, study, share, and modify the software under certain terms and conditions.
CPOL Code Project Open License
It is a software license that is often used for articles, tutorials, and sample code shared on The Code Project website. The CPOL is intended to be a more permissive license, allowing developers to use, modify, and distribute the software without many of the restrictions imposed by other licenses like the GPL.
CC0 Creative Commons Zero
It is a public domain dedication tool created by Creative Commons. It allows creators to waive all their copyright and related rights in a work, effectively placing it in the public domain. This means that anyone can freely use, share, modify, and build upon the work without seeking permission or providing attribution to the original creator.
B.11 SQL Function Taxonomy
SQL functions in BIRD mentioned in Table 1 span across multiple categories including:
- Window Functions, i.e., OVER()
- Date Functions, i.e., JULIANDAY ( )
- Conversion Functions, i.e., CAST ( )
- Math Functions, i.e., ROUND()
- String Functions, i.e., SUBSTR()
B.12 Keyword Statistic
We have conducted a comprehensive analysis of the keywords employed in the BIRD dataset and visualize the results in the form of a nice-looking word cloud, which can be found in Figure 12. We further classify keywords into 7 following categories:
Main Body Keywords • SELECT • FROM • WHERE • AND • OR • NOT • IN • EXISTS • IS • NULL • IIF • CASE • CASE WHEN.
Join Keywords ·INNER JOIN • LEFT JOIN • ON • AS.
Clause Keywords • BETWEEN • LIKE • LIMIT • ORDER BY • ASC • DESC • GROUP BY • HAVING • UNION • ALL • EXCEPT • PARTITION BY.
Figure 12: Keyword cloud presentation for SQLs in BIRD.
Aggregation Keywords • AVG • COUNT • MAX • MIN • ROUND • SUM.