Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Authors: Jinyang Li
Binyuan Hui
GE QU
Jiaxi Yang
Binhua Li
Bowen Li
Bailin Wang
Bowen Qin
Ruiying Geng
Nan Huo
Xuanhe Zhou
Chenhao Ma
Guoliang Li
Kevin Chang
Fei Huang
Reynold Cheng
Yongbin Li
Venue: NeurIPS 2023 Datasets and Benchmarks Spotlight (2023)
Keywords: Text-to-SQL Parsing
Large Language Models
Database Grounding
SQL Efficiency

Abstract: Text-to-SQL parsing, which aims at converting natural language instructions into executable SQLs, has gained increasing attention in recent years.  In particular, GPT-4 and Claude-2 have shown impressive results in this task. However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications. To mitigate this gap, we present BIRD, a BIg benchmark for laRge-scale Database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents,  and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL models must feature database value comprehension in addition to semantic parsing. The experimental results demonstrate the significance of database values in generating accurate text-to-SQLs for big databases. Furthermore, even the most popular and effective text-to-SQL models, i.e. GPT-4, only achieve 54.89% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand. We also provide an efficiency analysis to offer insights into generating text-to-efficient-SQLs that are beneficial to industries. 
We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research.
The leaderboard and source code are available: https://bird-bench.github.io/.

Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Jinyang Li Binyuan Hui Ge Qu Jiaxi Yang?, Binhua Li,, Bowen Li Bailin Wang Bowen Qin Ruiying Geng", Nan Huo', Xuanhe Zhou², Chenhao Ma".
Guoliang Li Kevin C.C. Chang" Fei Huang', Reynold Cheng', Yongbin Li
The University of Hong Kong 2 DAMO Academy, Alibaba Group Tsinghua University Shanghai AI Laboratory MIT CSAIL
6 The Chinese University of Hong Kong, Shenzhen University of Illinois at Urbana-Champaige {j10725, quge (aagee@ccoo @connect hku. hk, ckcheng& cs hhu. hk binyuan. hby ibaba inc. com
Abstract
Text-to-SQL parsing, which aims at converting natural language questions into executable SQLs, has gained increasing attention recent years. particular, GPT-4 and Claude-2 have shown impressive results in this task. However, most of the prevalent benchmarks, i... Spider, and WikiSQL, focus on database schema with few rows of database values Icaving the gap between academic study and real-world applications. To mitigate this gap. we present BIRD, a BIg bench for laRge-scale Database grounded in text-to-SQL tasks, containing 12,751 1 text-to- SQL pairs and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the challenges of dirty and noisy database values, external knowledge grounding between NL questions and database values, and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL. models must feature database value comprehension in addition semantic parsing. The experimental results demonstrate the significance of database values in generating accurate texl-to-SQLs for big databases. Furthermore, even the most effective text-to-SQL. models, i.e. GPT-4, only achieve 54.89% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand. We also provide an efficiency analysis to offer insights into generating are beneficial to industries. We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research. The leaderboard and source code are available: https: / /bb rd bench. gi github..i.... io/.
Soo-efficient-SOO
that
Introduction
Text-to-SQL parsing [55, 50, 51, 3, 52, 37], which focuses on converting natural language questions into SQL queries, has attracted significant research interests from both academia and industry. This attention stems from its potential empower non-expert data analysts automatically extracting desired information from ubiquitous relational databases using natural language. Recent advances in neural models, including those based on large language models (LLMs), have led an impressive performance existing benchmarks such as SPIDER [53] and WikiSQL. [58]. For instance, the execution accuracy of the top-performing model in the SPIDER leaderboard has increased from 53.5%
Equal contribution.
i Work done during an intern at Alibaba DAMO Academy,
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Large and Bealattic Database Valuee
External Knowledge Ressoning
Execution Efficiency
What the evrrae of the peweeett-pprreee partoring eragen
ryymbad kis
S0l, normal semontic porser Bun Hira 22.45
SUACY caucht RoM WHiSS AND pont
de 00
Soate Your and MMssgn
Dababaww
Employese
SOL cifficientt semontic parser Run time 4.0s
pert knoen ceerrs ge 1R000 coeeres couchert
STLECT COUNT"" HOM
DRDCR WULL) oRcch cauch port UIeT
Figure I: Examples of challenges in our BIRD benchmark. 1) databases contain values of noisy data types [14, 23, 19, 31J. In the left example, the average salary could be fetched by processing the data type from string (TEXT in SQLite) to float (REAL in SQLite) after deleting the special tokens, "US$" and ", ". 2) external knowledge and reasoning are required. In the middle example, models must handle that only " "OWNER" accounts are eligible for loans. 3) query execution efficiency needs to be considered. In the right example, the adoption of more efficient SQL. queries leads to significant gains in speed, which is of great value in industries.
[59] to 85.3% [35] over the past three years. The latest SOTA parser [35] in SPIDER benefits from the powerful understanding and coding capabilities of the large language model (LL.)), and such excellent performance leads us to ask a question: Can LLM already serve as a database interface ?
The answer is no, as previous benchmarks focus on database schema with few rows of database values leaving the gap between academic study and the real world. As shown in Figure 1, first, we discovered that current state-of-the-ar models still struggle to generalize to more realistic situations characterized by large database sizes and noisy values. Second, the growth in database sizes often results in much context compression, making it challenging to reveal the entire context [1]. Thus it requires external knowledge reasoning for comprehensive understanding. Third, existing benchmarks do not account for SQL. execution efficiency, which holds significant practical importance in real-life applications, notably in the case of large databases. Motivated by these observations, we aim develop a new text- to-SQL. benchmark that better represents real-life scenarios and narrows the gap between experimental and practical settings.
In this work, we propose BIRD, a Big Bench for LaRge-Scale Database Grounded in Text-to-SQLs for real-world applications. BIRD contains complex 12,751 examples of querying information over 95 big databases with total size of 33.4 GB spanning 37 professional domains. For training and development, we collected and modified 80 open-source relational databases from real analysis platforms (Kaggle, Relation.vit, To further avoid data leakage, we curated 15 additional relational databases for a hidden test set. Given these databases, we rely on crowdsourcing to collect natural language questions and the corresponding SQLs. Additionally, we propose new evaluation metric Valid Efficiency Score (VES) to evaluate the efficiency of generated SQL.s. To the best of our knowledge, BIRD is the first text-to-SQL benchmark to incorporate efficiency, promoting more efficient query methods within the context of massive and noisy database values.
We evaluate the performance of state-of-the-an text-to-SQL parsers using two popular methodologies: fine-tuning (FT) with T5 [38]. and in-context learning (ICL) with advanced large language models (LLMs) such as ChatGPT [33] (gpt-3 1eppt-3.5-tubbae 5-turbo), Claude-2 [2] (claude-2. 0), GPT-4 [32] (gpt-4-32k). Our experimental results demonstrate that the current models struggle to generalize well on BIRD. Specifically, even the GPT-4 only achieves 54.89% in execution accuracy. In comparison, the performance still lags far behind the human performance of 92.96%, proving that challenges still stand. Moreover, we perform a comprehensive analysis to provide insight and direction. We encourage further research by the NLP and DB communities to jointly address the more realistic settings presented in this benchmark.
2 Task Formulation & Annotations
Text-to-SQI. refers to the process of converting a natural language question Q into a SQL query Y capable of retrieving relevant data from database. The database can be represented as D (C,

---

Queniin
Collet
(acast CHEFIRRTIONA PEDY peods
2003
DESATET
experts
ouTuraAsA prile
Workflo
DeDaaaab Docsiption
Figure 2: An Overview of the BIRD Annotation Workflow in (a). This figure depicts a four-step procedure. (1) The workflow begins with specialists assembling and producing databases and description files. (2) Experts then teach and evaluate crowdsourcing people, keeping only those who pass the evaluation. (3) Question annotators create a corpus of questions using databases and their corresponding description files. (4) SQL annotators produce SQL files, equipped with databases, descriptions, and questions. (b) and (c) also depict the Double-blind annotation procedure and an example of database descriptions.
where C and T are columns and tables respectively. When dealing with complex database values, such as BIRD, it is erucial incorporate external knowledge evidence, denoted as K., to improve the models' understanding of database values. Finally, the text-to-SQL. could be formulated as:
Y f(Q, D,K 0),
(1)
where the function (+ () can represent model neural network with the parameter O.
3 Dataset Construction
3.1 Annotation Entrance
To deliver a high-quality benchmark, we administer thorough exams to all applicants and only hire those who pass these rigorous tests. Further information is available in the Appendix B.2.
3.2 Database Source
It is difficult to collect databases with complex schemas and sufficient value due privacy protection. Earlier works [45, 53] choose to self-design database schemas and value production. Nonetheless, the value distribution and schemas may differ from real-world scenarios in this way. In our work, we obtain and process databases from three different sources to enrich real-world attributes. 32% of databases sourced from Kaggle", platform renowned for holding data science competitions with difficult, noisy values and schemas. Another 48% come from CTU Prague Relational Learning Repository", open platform for machine learning research with multi-relational data. The remaining 20% are built by acquiring open tables, synthesizing and standardizing schemas, and generating database constraints. All of these databases contain real and large value distributions and are easily accessible with the appropriate licenses. Finally, we present 95 databases consisting of 69, 11, and 15 databases for training, development, and testing respectively. Our databases cover 37 professional domains, including blockchain, sports, health care, politics, etc. We anticipate that will be significant resource for researchers explore domain generalization in semantic parsing tasks with large database values.
"https://ww, htt.///wwwwkgl.....////// com/ 'nttps://relation fit evut.ez/ cz/
3

---

3.3 Question Annotation
Database Description File. The Database Description File a crucial resource designed to aid annotators in comprehending database values, thereby allowing them to ask insightful questions. It offers two primary pieces of information regarding the database. (1) Full schema names: database table and column names are frequently abbreviated, which are difficult to understand.
(2) Value description: this aspect is particularly useful when phrases or tokens in a question do not directly match values in the database.
External Knowledge Evidence. In our study of professional data analysis, we find that external knowledge evidence is required to map the natural language instructions into counterpart database values. Therefore, we collect and classify such evidence into four categories; (1) Numeric Reasoning Knowledge: this category refers to the mathematical computation required for certain SQL. operations. In our benchmark, we present 8 basic math operations, including 4 complex operations as [7]: MINUS, ADDITION, DIVISION, MULTIPIN, BIRD also contains compositional operations over basic ones, such as percentages, formulas, etc. (2) Domain Knowledge: this category consists of domain-specific knowledge that utilized to generate SQL operations [10, 57J. For instance, a business analyst in the banking business requires knowledge of financial indicators such as return on investment and net income in order to generate effective SQL queries. (3) Synonym Knowledge: this category includes words expressions that have the same similar meanings regardless of how they are phrased differently [11]. (4) Value Illustration: this category refers to detailed descriptions of database values, including value types, value categories, and the mapping combinations of columns and values that correspond to entities, for example: "center" can be represented by "pos = c" in the database professional basketball.
3.4 SQL Annotation
Double-Blind Annotation. As shown in Figure 2 (b), we employ double-blind approach [42] for SQL. annotation. This approach involves two independent SQL annotators who generate SQLs for the same question without discussion. The annotated SQLs are executed in databases, and those yielding identical results are gathered. Otherwise, the SQLs are checked with experts until a consensus is reached. Double-blind procedures can dramatically reduce the SQL annotation error rate, as there is a small probability for two skillful annotators to generate the same incorrect results when databases have large values. The more rrmantic-equivalle and efficient SQL selected by experts for each question is picked as ground truth SQL in BIRD, and the external knowledge evidence sentences are recorded for each SQL if utilized.
Examination. Experts evaluate each Eoxt-to-SQL pair to ensure the highest quality of data. The evaluation process includes two dimensions: SQL validness, and soxt-kowweede-SCe alignment. Firstly, the SQL validness will be confirmed that each SQL is executable and can return a valid result from the database. The "valid result" refers the set of results that not "NULLL" the executed result set is "NULL" ", experts will make slight changes to the conditions of the questions until the associated SQLs can provide valid result set. Secondly, involved to ensure that each SQL can be generated with the given texts and knowledge evidence. the evidence is insufficient to generate the SQL or contains errors, experts will be in charge of correcting them.
alignment
4 Data Statisties
Overall Statisties Table presents overview comparison between BIRD and other cross-domain text-to-SQL benchmarks. As the statistics demonstrate, BIRD a large-scale cross-domain bench- mark, covering complex SQL functions, knowledge reasoning, and efficiency evaluation.
Question Statisties Database values bring more challenges in text-to-SQLs. In order to underscore this, we classify questions into two macro-categories: Fundamental Type and Reasoning Type, and each contains 4-5 micro-categories in detail. The Fundamental Type of questions refers to those that can be answered without database value comprehension. It contains Match-based (83.9%), Ranking (20.3%), Comparison (16.7%), Counting (30.4%), Aggregation (15.7%). The

---

Table 1: An overview comparison between BIRD and other cross-domain text-to-SQL benchmarks. In SQL., Function on pertains the SQL functions (Appendix B.11). Knowledge refers to whether or not this dataset necessitates external knowledge reasoning from the model. Efficiency refers whether or this dataset takes into consideration execution efficiency.
Dataset
# Example
# DB
# Table/DB
Row/DB
Function
Knowledge
Efficiency
WikiSQL [58]
80,654
26,521
17
X
X
X
SPIDER [53]
10,181
200
X
X
X
KaggleDBQA [24]
272
8
2.3
280K
X
BIRD
12,751
95
7.3
549K
V
TEXT
REAL
40-
150
2000
5:6
dAy
31.3
43.4
Pourtags (S))
36.1
X.0
100
3xal
type
Figure 3:
This is a comprehensive database distribution in the BIRD. a) shows the domain and size distribution of each database. And b) shows the data type distribution of databases.
Reasoning Type entails questions that demand the external knowledge grounding on values, which is exclusive to BIRD. To be specific, the questions about Domain Knowl wdwge (23.6%) Numeric Computing (24.5%), Synonym (7.2%), Value Illustration (70.1%), are in- volved in BIRD. There are ample examples in Appendix B.3. addition, we observe that 70. 1% of the questions need value illustrations. This indicates that more real-world questions in text-to-SQL. applications demand thorough understanding of database values, which is consistent with our motivation for creating the BIRD benchmark.
Database Statistics In BIRD, we investigate the distribution of database domains, database size, and value types. Figure 3 (a) presents detailed distribution of domains and their counterpart databases in a sunburst diagram for both training and development sets. The area each semi-circle corresponds to the number of text-to-SQL, pairs in this database. Figure 3 (a) also shows the size distributions of databases. The darker color means larger size of databases, and vice versa. For example, database Donor is the largest database with 4.5 GB in this dataset. Furthermore, Figure 3 (b) that considerable proportion of BIRD'S data comprises date-related values. Considering that real-world applications often rely on üimessenstive data [25], the prevalence of such questions highlights the practical purposes.
observe from
SQL Statisties We provide the complexity and diversity of SQLs in BIRD. As illustrated in Figure 4, we present a comprehensive distribution analysis SQLs across four dimensions. No Toks / SQL and No. JOTNs / SQL demonstrate the intricacy of the SQLs in BIRD. No of Keyworde and No n- n-grams / SQL (n=3) serve as the support for the diverse patterns of SQLs since we decouple the question and SQL annotation procedures to make the situation more realistic [6].
5 Evaluation Metrics
In contexts of practical data analysis, text-to-SQL models are prioritized for delivering expected results accurately and efficiently. Thus we provide two metrics in BIRD, execution accuracy (EX)
5

---

WabSQL
Spicer
30
6,0
75.0
18-5
15
12.2
135
08
No. Toka SQL
No. of Keywords
No. n-grams/ (QL())3) (x-3)
No. JOINO/ON///0 /SQL.
Figure 4: A comparative statistical analysis of SQL queries in the BIRD dataset and other cross- domain text-to-SQL. benchmarks.
and valid efficiency score (VES) to evaluate text-to-SQL parsers confronted with large real-world database values.
Execution Accuracy (EX) EX is defined as the proportion of examples in the evaluation set for which the executed results both the predicted and ground-truth SQLs are identical, relative the overall number of SQLs [37]. Considering the result set as Vn executed by the nh ground-truth SQL Yn, and the result set V, executed by the predicted SQL Yn, EX can be computed by:
J.
1 1V,, V..)
EX
(2)
N
where 1() is indicator function, which be represented as:
1, V V
1(V,))
(3)
0. V
Valid Efficiency Score (VES) VES designed to measure the efficiency of valid SQLs generated by models. It is worth noting that the term "valid SQLs" refers to predicted SQL queries whose result sets align with those of the ground-truth SQLs. Any SQL queries that fail to fetch the correct values will be declared invalid since they are totally useless if they cannot fulfill the requests, regardless their efficiency. this case, the VES metric considers both the efficiency and accuracy of execution results, providing comprehensive evaluation of model's performance. Formally, the VES can be expressed as:
VES- DN I(V, V.) R(Y,,
B(Y.) E(Y.)
R(Y., Y.)
(4)
N
where R(-) denotes the relative execution efficiency of predicted SQL in comparison ground-trute -trut SQL, allowing for machine status-related uncertainty. E(-) is function to measure the absolute execution efficiency for each SQL in given environment". Furthermore, root function to minimize random instances that are abnormally faster or slower than the ground-truth SQLs. Here, efficiency can refer to running time, throughput, memory cost, merged metries. In BIRD, we consider the running time mainly at this time. Appendix B.8 provides a detailed description of the VES.
incorporate the square
Experiments
6.1 Baseline Models
We present the performance of two types of baseline models in BIRD. The first type of model is based on fine-tuning (FT) techniques, which outputs SQL by tuning all parameters of language models to learn the annotated train set. On the other hand, the second type of model based on
In BIRD evaluation, we run 100 times for each SQL in the same CPU and evaluate average results after dropping the outliers.

---

Advanced Text-to-SQL Models
Execulton Accurncy (E3))
Develqmmer Knowrldgge
Dovdigmerr kite
80-
Toeting weo knowledge
Toeting Knowsadge
50-
20.
T5-Large
ChiGPI-EDD
Valid Eeiceency Score (VES)
60
Developennt wis Kaowiedge
Devoleproer Kaowiooige
Scoowledge
Teoling Kaowledge
T5 Base
Large
T5:3B
Palm-2
Codex
CharGPT
ChiGPT-COT
Claude
GPT4
Figure 5: A bar chart provides clear visualization of the performance of advanced models on BIRD.
Table 2: The Execution Accuracy (EX) of advanced text-to-SQL. models in BIRD. The human performance also provided.
Development Data
Testing Data
Models
w/o knowledge
w/ knowledge FT-based
w/o knowledge
w/ knowledge
TS-Base
6.32
11.54 (+5.22)
7.06
12.89 (+5.83)
T5-Large
9.71
19.75 (+10.04)
10.38
20.94 (+10.56)
T5-3B
10.37
23.34 (+12.97)
11.17
24.05 (+12.88)
ICL-based
Palm-2
18.77
27.38 (+8.61)
24.71
33.04 (+8.33)
Codex
25.42
34.35 (48.93)
24.86
36.47 (+11.61)
ChatGPT
24.05
37.22 (+13.17)
26.77
39.30 (+12.53)
ChatGPT COT
25.88
36.64 (+10.76)
28.95
40.08 (+11.24)
Claude-2
28.29
42.70 (+14.11)
34.60
49.02 (+14.42)
GPT-4
30.90
46.35 (+15.45)
34.88
54.89 (+20.01)
GPT-4 + DIN-SQL
50.72
55.90
Human Performance
72.37
92.96 (+20.59)
in-context learning (ICL), can generate results without additional training. In FT models, we select T5 family [38] as the main baseline models. For ICL-based models, we provide zero-shot results of Codex (code seoode-daving davinci- 002), ChatGPT (gpt-3, -.. 3-w--3.rbo)t turbo), GPT-4 (gpt ----44-2222 -32k), Claude-2 (claude-22 0), Palm-2
-exttbisoonnoo 001). Additionally, we also implement a state-of-the-art (SOTA) model of SPIDER, DIN-SQL [35], to evaluate the challenges proposed by the BIRD dataset.
Table 2, Table and Figure present the overall results of advanced language models on BIRD.
Execution Accuracy Analysis
Table 2 and Figure presents stratified performances of various models in BIRD. GPT-4 surpasses all baseline language models. Claude-2 closely follows, demonstrating outstanding abilities in semantic parsing and knowledge reasoning. Further, the incorporation of a dedicated reasoning prompt by [35], enables DIN-SQL GPT-4 achieve a new state-of-the-an result on BIRD. It contains value sampling, few-shot demonstrations, and self-correction. Despite considerable advancements in Language Model Learning (LLMss) and prompt intelligence, the performance of these models lags obviously behind human capabilities. Not only does this gap highlight the complex nature of BIRD, but it also presents opportunities for uncovering more capable models or advanced reasoning prompt methods applicable rorl-world text-to-SQL scenarios.
7

---

Table 3: The Valid Efficiency Score (VES) of advanced text-to-SQL models in BIRD. The human performance also presented.
Development Data
Testing Data
Models
w/o knowledge
w/ knowledge
w/o knowledge
w/ knowledge
FT-hased
TS-Base
7.78
12.90 (+5.12)
8.97
14.71 (+5.74)
T5-Large
9.90
22.74 (+12.84)
12.25
25.00 (+12.75)
T5-3B
13.62
25.57 (+1195)
15.17
27.80 (+12.63)
ICL-based
Palm-2
20.82
28. (+7.82)
31.32
38.41 (+7.09)
Codex
33.37
43,41 (+10.04)
35.40
41.60 (+6.20)
ChatGPT
27.97
43.81 (+15.84)
36.68
51.40 (+14.72)
ChatGPT COT
32.33
42.30 (+9.97)
49.69
56.56 (+6.87)
Claude-2
32.75
45.28 (+12.53)
39.32
55.77 (+16.45)
GPT-4
34.60
49.77 (+15.17)
40.20
60.77 (+20.57)
GPT-4 + DIN-SQL
58.79
59.44
Human Performance
70.36
90.27 (+19.91)
6.3 Baseline Performance on Spider
SPIDER [53] is the most prevalent and complex cross-domain text-to-SQL benchmark. It mainly focuses on evaluating schema-relevant seman- tic parsing capabilities. To demonstrate the in- creasing difficulty of BIRD dataset due to its complex database schema and values, visualize the execution accuracy of the same baseline models on both the BIRD and SPIDER datasets. To ensure fair evaluation, all models furnished with knowledge about values, and the same programming prompt implemented for Language Models (LMs) across the two datasets. Figure shows the concentration on database values makes BIRD become the most challenging text-to-SQL. benehmark. This dis- parity in the performance of each model demon- strates the need for further research and develop- ment of models capable of handling complicated database schema and values.
SPIDER
BAD
50.7
72.4
ChatGPT
37,2
743
34.4
15:38
68
154arge
19.8
T5-Base
Figure 6: The EX results of the same baseline models on the SPIDER and BIRD dev set.
6.4 Efficiency Analysis
According to Table 3, we can observe that models with higher EX can more possibly achieve higher VES. This can be explained by the prerequisite that text-to-SQL. models must accurately predict results in order to attain higher VES, which fulfills the practical purpose.
Two-Stage Optimization. Intuitively, the goal of text-to-efficient-SQQ conversion can be de- composed into two sub-stages. Following previous text-to-SQQ tasks, the first sub-stage, semantic parsing, concentrates accurately converting questions into SQL queries. The second sub-stage involves optimizing the SQL queries, rewriting them to be more efficient while maintaining the same selected 10 random examples from
results [61]. To demonstrate the efficacy of this approach, the development set where ChatGPT accurately predicted the results. Then, our specialists optimize these queries based the established query optimization rules [28, 34, 62). We observe that the two-stage optimization leads to an average time-saving of 77.75% while keeping the same results.
Chat w/ Database. BIRD introduces the novel mode "Chat With Database". which enables models to be aware of data types and distributions by generating global queries that interact with databases. This approach lays the foundation for the development of more effective and efficient SQL queries. As observed in the experiment, the simesaviigg percentage of the SQL queries can

---

ChatGPT
GPT-4
Claude-2
ChatGPT
GPT-4
Claude-2
Overall
carge
Figure 7: The fine-grained categorical evaluation of advanced large language models on BIRD.
be reached 87.3% by configuring indexes within the database. The detailed efficiency analysis is presented in Appendix B.5.
6.5 Knowledge Evidence Analysis
We implement each baseline model for both two scenarios. The first is NOT provide the ground truth external knowledge evidence sentence (w/o knowledge) for each sample. The other testing bed provide such evidence (w/ knowledge) and make text-to-SQL models do knowledge grounding by themselves. As we discuss in Section 3.3, expert annotations on external knowledge evidence sentences are employed to enhance the model's comprehension of database values.
After being easily fed with the external knowledge evidence about the database values, all models have clear improvement across the different difficulty levels shown in Table 2 and Table 4. This indicates that external knowledge evidence in BIRD effective and instructive for models to better understand the database values. Also it illustrates that the database values are very important to text-to-SQL models when facing more real databases. Besides, ICL-based approaches have better self-knowledge grounding capability and pre-trained SQL knowledge than FT smaller models with less than 5B parameters. Equipped with COT, ChatGPT can perform better, since multi-step reasoning beneficial when the knowledge and data are resource.... Despite this, we observe a decline or limited improvements in performance for ChatGPT + external knowledge evidence for COT version. We hypothesize that the internal multi-step knowledge reasoning of LLMs is not compatible with the way of external knowledge (evidence) in this situation. Therefore, the development of methods that effectively combine the strong multi-step self-reasoning capabilities of L.MMs with external knowledge reasoning coherently presents a promising future direction [29].
6.6 More Analysis
Fine-grained Category Analysis. Figure 7 provides a detailed comparison of various dimensions of sub-capabilities of advanced LLMs on BIRD. The results indicate that GPT-4 exhibits superior performance against ChatGPT and Claude-2 in all areas. Nevertheless, there notable disparity in the performance of ranking and numerical computing (math) among all the models. This limitation may suggest the inadequacy of contemporary L.LMs for deep data science tasks because such tasks always incorporate mathematical computations and rankings within the context of vague user queries. Conversely, these models demonstrate relatively better performance in domain knowledge, synonym detection, and value illustration, which can be attributed to their adequate linguistic training and reasoning capabilities during the pretraining phases.
Human Performance. In order to activate the efforts of text-to-SQI. studies to achieve an application-level performance in real-world scenarios, we provide human performance BIRD. Table 2, Table 3 shows that there's 's still huge gap between even SOTA text-to-SQL models and human performance. The thorough introduction of procedures is Appendix B.9.
Error Analysis. ChatGPT is currently the most prevalent and cost-efficient LLM. Therefore, the performance of ChatGPT is concentrated in this error analysis. The detailed analysis in Appendix B.6. We observe 500 randomly sampled error cases, providing in-depth assessment in the following

---

categories. Wrong Schema Linking (41.6%) pertains to the scenario where ChatGPT can accurately comprehend the structure of the database but erroneously associates it with inappropriate columns and tables. This demonstrates that the task of schema linking [43, 57], even in intricate and practical situations, continues to be significant obstacle for models. Misunderstanding Database Content (40.8%) occurs when ChatGPT either fails to recall the correct database structure (e.g., rtype doesn't belong to the satscores table) or generates fake schema items (e.g., wwrp_lpppporrr records not appearing in the formula I database and many values predicted incorrectly) especially when the database is very large, In this case, how to make ChatGPT really understand database structure and values [27] is still pain point topic in LLMs. Misunderstanding Knowledge Evidence (17.6%) refers to cases in which the model does not accurately interpret human-annoturl evidence. An instance is that ChatGPT directly copies the formula DIVIDE (SUM (spent), , COUNT (spent)) ) This finding demonstrates that ChatGPT exhibits a lack of robustness in response to unfamiliar prompts or knowledge, causing it to directly replicate formulas without considering SQL syntax [15]. We also observe that ChatGPT occasionally employs incorrect keywords (e.g.. misusing the MySQL. Year () function instead of an SQLite function STRFT IME ) exhibits decoding errors.
7 Related Work
High-quality datasets are crucial for advancing various natural language processing tasks, including text-to-SQL. Early single-domain text-to-SQL datasets like GeoQuery [55], ATIS [9], and Restaurant [20] targeted specific information retrieval tasks, while more recent datasets such as WikiSQL. [58] and SPIDER [53] propose cross-domain dataset to require domain generalization. However, most cross-domain text-to-SQL. datasets still emphasize database schema rather than values, diverging from real-world scenarios. KaggleDBQA [24] addressed this by constructing 272 text-to-SQL pairs from eight databases on Kaggle, while other datasets like EHRSQL [25], SEDE [13], and MIMICSQL [46] collected diverse, large-value databases with more professional SQL queries. Despite these advancements, these datasets remain single-domain focused. Recent work has explored knowledge-inten intensvvvee text-to-SQL benchmarks [10, 57], aiding experts in knowledge grounding. BIRD is the first large-scale benchmark to incorporate these real-world features, emphasizing database values.
analysis through
8 Limitation and Future work
Despite the high quality of SQL annotation produced by double-blind annotation, the procedure is resource-intensive. Future research could explore hommat-computer interaction (HCI) based approach, incorporating advanced Al systems such as GPT-4 for taking parts of annotation duties, to maintain data quality while reducing human effort. In addition, SQLite was chosen as the primary SQL codebase for previous text-to-SQL benchmarks and this study since it's friendly to users. While presents difficulties in fetching Query Execution Plans (QEP) for precise efficiency computation and adapting to different SQL syntaxes. Future work will include PostgreSQL. and MySQL versions of BIRD to resolve these limitations and provide more robust research environment for both NLP and DB experts.
Conclusion
In this paper, we introduce BIRD, large-scale cross-domain, text-to-SQL benchmark with a particular focus on large database values. BIRD mitigates the gap between text-to-SQL research and real- world applications by exploring three additional challenges: 1) handling large and dirty database values, 2) external knowledge evidence, and 3) optimizing SQL execution efficiency. Our experimental results demonstrate that BIRD presents a more daunting challenge compared to existing benchmarks since even the most popular and powerful LLM, ChatGPT, falls significantly short of human performance. This leaves plenty of room for improvement and innovation in the text-to-SQL. tasks. Moreover, our thorough efficiency and error analyses provide valuable insights and directions for future research, paving the way for the development of more advanced and practical text-to-SQL. solutions in real- world scenarios.
10

---

Acknowledgement
We thank all constructive comments from anonymous reviewers. Reynold Cheng, Jinyang Li, Ge Qu and Nan Huo were supported by the Hong Kong Jockey Club Charities Trust (Project 260920140) and the University of Hong Kong (Project 104006830). Chenhao Ma was supported by NSFC under Grant 62302421, Basic and Applied Basic Research Fund in Guangdong Province under Grant 2023A151500 1280, Shenzhen Science and Technology Program ZDSYS2021 102111144500 Jinyang Li and Ge Qu were supported by HKU Presidential PhD Scholar Programme. Ge Qu was also funded by Hong Kong PhD Fellowship Scheme. This work was supported by Alibaba Group through Alibaba Research Intern Program.
References
[1] Peter Alsherg. Space and time savings through large data base compression and dynamic restructuring. Proceedings of the IEEE, 13:1114-1122, 1975.
[2] Anthropic. Introducing Claude. 2023. URL httpt:ttp///o /wwp///wwwww//////////// www.anthrroicc.com/ index/
introducing-claauee
[3] Ruichu Cai, Boyan Xu, Zhenjie Zhang, Xiaoyan Yang, Zijian Li, and Zhihao Liang. An encoder-decoder framework translating natural language to database queries. In Proceedings of the Twenty-Seventh International Join! Conference on Artificial Intelligence (IJCAI-18), page 3977-3983, 2018.
[4] Zefeng Cai, Xiangyu Li, Binyuan Hui, Min Yang, Bowen Li, Binhua Li, Zheng Cao, Weijie Li, Fei Huang. Luo Si, and Yongbin Li. STAR: SQL guided pre-training for context-dependent text-to-SQL parsing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 125-1127, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[5] Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. LGESQL: Line graph enhanced text-to-SQL SQL model with mixed local and non-local relations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 2541-2555 Online, August 2021. Association for Computational Linguistics.
[6] Shuaichen Chang, Jun Wang. Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng. and Bing Xiang. Dr.spider: diagnostic evaluation benchmark towards text-to-SQL. robustness. In The Eleventh International Conference on Learning Representationss 2023.
[7] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson. Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen Meier- Hells-Hel Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.022 22122044.231... 2022.
11

---

[9] Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Somicke-Smithe David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg, Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Technology: Proceedings of Workshop held at Plainshoro, New Jersey, March 8-11, 1994, 1994.
[10]
Longxu Dou, Yan Gao, Xuqi Liu, Mingyang Pan, Dingzirui Wang, Wanxiang Che, Dechen Zhan, Min-Yen Kan, and Jian-Guang Lou. Towards knowledge-intensive text-to-SQL semantic parsing with formulaic knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5240-5253, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[11] Yujian Gan, Xinyun Chen, Qiuping Huang. Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. Towards robustness of text-to-sql models against synonym substitution. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, 1 pages 2505-2515. Association for Computational Linguistics, 2021. doi: 10.18653/v1/200 laclllon....l...
[12] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian Lan-Guang Lou, Ting Liu, and Dongmei Zhang. Towards complex text-to-SLL in cross- dross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4534 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/119-11
[13] Moshe Hazoom, Vibhor Malik, and Ben Bogin, Text-to-SQL in the wild: A naturally-occurring dataset based stack exchange data. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021). pages 77-87, Online, August 2021. Association for Computational Linguistics. doi: 1/833//11/2211111ppppp//g 1.9.
[14] Joseph M. Hellerstein. Quantitative data cleaning for large databases. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, page 1197-1200, 2008.
[15] Jie Huang, Hanyin Shao, and Kevin Chen-Chhan Chang. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 2038-2047, Abu Dhabi, United Arab Emirates, December Association for Computational Linguistics.
2022.
[16] Binyuan Hui, Ruiying Geng, Qiyu Ren, Binhua Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, Pengfei Zhu, and Xiaodan Zhu. Dynamic hybrid relation exploration network for cross-domain context-dependent semantic parsing. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13116-13124. AAAI Press, 2021.
[17] Binyuan Hui, Xiang Shi, Ruiying Geng, Binhua Li, Yongbin Li, Jian Sun, and Xiaodan Zhu. Improving text-to-sql with schema dependency learning. In wwXiv:2003.04399 2021.
[18) Binyuan Hui, Ruiying Geng, Lihan Wang, Bowen Qin, Yanyang Li, Bowen Li, Jian Sun, and Yongbin Li. S?SQL: Injecting syntax to question-schenee scchoman--camaaaaaa interaction graph encoder for text-to-SQL parsers. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1254-1262, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18665/v1/022.ffl findings-acl.999
[19] Ihab F. Ilyas and Xu Chu. Trends in cleaning relational data: Consistency and deduplication. Foundations and Trends in Databases, 1393,, 2015.
[20] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung. Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653//1/111-18899
12

---

[21] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are shot reasoners. Advances in Neural Information Processing Systems, volume 35, pages 221199222211 2022.
[22] Jan Kossmann, Stefan Halfpap, Marcel Jankrift, and Rainer Schlosser. Magic mirror in my hand, which is the best in the land? an experimental evaluation of index selection algorithms. Proceedings of the VLDB Endowment, 13(12):2382-2395 2020.
[23] Premaa S. Kulkarni and Jagdish W. Bakal. Survey on data cleaning. In 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), page 2361-2366, 2014.
[24] Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. KaggleDBQA: Realistic evaluation of text-to-SQL parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2261-2273, Online, August 2021. Association for Computational Linguistics.
[25] Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Sco, Jong-Yeup Kim, and Edward Choi. Ehrsql: practical text-to-sql benchmark for electronic health records. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 15589-15600 Curran Associates, Inc., 2022.
[26] Dandan Li, Lu Han, and Yi Ding. Sql query optimization methods of relational database system. In 2010 Second International Conference on Computer Engineering and Applications, volume 1, pages 557-560 IEEE, 2010.
[27] Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph- aware layers for text-to-sqi parsing. ArXiv, abs/2301 .07507, 2023.
[28] Tanzim Mahmud, KM Azharul Hasan, Mahtab Ahmed, and Thwoi Hla Ching Chak. A rule based approach for nip based query processing. In 2015 2nd International Conference on Electrical Information and Communication Technologies (EICT), pages 78-82. IEEE, 2015.
[29] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey. ArXiv, abs/2302.07842,, 2023.
[30] Vamsi Krishna Myalapalli and ASN Chakravarthy. Revamping sqi queries for cost based optimization. In 2016 International Conference on Circuits, Controls, Communications and Computing (14C), pages 1 1-6. IEEE, 2016.
[31] Paulo H. Oliveira, Daniel dos Santos Kaster, Caetano Traina, and Ihab F. Ilyas. Batchwise probabilistic incremental data cleaning. ArXiv, 0///22011044334 2020.
[32] OpenAL. Gpt-4 technical report. ArXiv, abs/2303.087744 2023.
[33] Long Ouyang, Jeff Wu, Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.d00 022155, 2022.
[34] Hamid Pirahesh, Joseph M. Hellerstein, and Waqar Hasan. Extensible/rule based query rewrite optimization in starburst. In Proceedings of the ACM SIGMOD International Conference on Management of Data, pages 39-48, 1992.
[35] Mohammadreza Pourreza and Davood Rafiii. DIN-SQL: decomposed in-context learning of text-to-sql with self-correction. CoRR, abs/2304.11015 2023. doi: 10.48550/rrxiv224441110
13

---

[36] Jiexing Qi, Jingyao Tang, Ziwei He. Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang. Quanshi Zhang, and Zhouhan Lin. RASAT: Integrating relational structures into pretrained Seq2Seq model for SexI-to-SQI.. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 3215-3229, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[37] Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang. Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao. Jian Sun, Luo Si, Fei Huang, and Yongbin Li. A survey text-to-sql parsing: Concepts, methods, and future directions. In arXiv:2208. 13629, 2022.
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1677 2020.
[39] Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilittes of large language models. ArXiv, abs/2204 00499, 2022.
[40] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 9895-99011 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[41] Peter Shaw, Ming-We Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 922-938, Online, August 2021, Association for Computational Linguistics.
[42] Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsenseqa 2.0: Exposing the limits of ai through gamification. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks, volume 1. Curran, 2021.
[43]
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT- SQL: relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578. Online, July 2020. Association for Computational Linguistics.
[44] Lihan Wang, Bowen Qin, Binyuan Hui, Bowen Li, Min Yang. Bailin Wang. Binhua Li, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing. In Aidong Zhang and Huzefa Rangwala, editors, KDD 22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington. DC, USA, August 14- 18, 2022, pages 1889-1898. ACM, 2022. doi: 10.1 445/5534678.3539966
[45] Lijie Wang, Ao Zhang, Kun Wu, Ke Sun, Zhenghua Li, Hua Wu, Min Zhang, and Haifeng Wang. DuSQL: A large-scale and pragmatic Chinese text-to-SQL dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6923 6935, Online, November 2020. Association for Computational Linguistics.
[46] Ping Wang, Tian Shi, and Chandan K. Reddy. Text-to-sql generation for question answering on electronic medical records. Proceedings of The Web Conference 2020, 2020.
[47] Zhaoguo Wang, Zhou Zhou, Yicun Yang, Haoran Ding, Gansen Hu, Ding Ding, Chuzhe Tang, Haibo Chen, and Jinyang Li. Wetune: Automatic discovery and verification of query rewrite rules. In Proceedings of the 2022 International Conference on Management of Data, pages 94-107, 2022.
[48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, 11122001.1190, 2022.
14

---

[49] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang. Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong. Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 602-631, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[50] Xiaojun Xu, Chang Liu, and Dawn Song. Sqinet: Generating structured queries from natural language without reinforcement learning. ArXiv preprint, 2017.
[51] Navid Yaghmazadch, Yuepeng Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(0OPSLA):
1-26, 2017.
[52] Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. TypeSQL: Knowledge-based type-aware neural text-to-SQL generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), page 588-594, 2018
[53] Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale haben-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, page 3911 1 11211 2018
[54] Tao Yu, Rui Zhang, Alex Polozov, Christopher Meek, and Ahmed Hassan Awadallah. Score: Pre-training for context representation in conversational semantic parsing. 9th International Conference on Learning Representations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[55] John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Conference on Innovative Applications of Artificial Intelligence, pages 1050-1055, 1996.
[56) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. ArXiv, 05s/2205.01066 2022.
[57] Chen Zhao, Yu Su, Adam Pauls, and Emmanouil Antonios Platanios. Bridging the generalization gap in text-to-SQL parsing with schema expansion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5568-5578 Dublin, Ireland, May 2022. Association for Computational Linguistics.
[58] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2SQL: Generating structured queries from natural language using reinforcement learning. In CoRR (ass/109900003,, 2017.
[59] Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. Grounded adaptation for zero-shot executable semantic parsing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6869 6882. Association for Computational Linguistics, 2020. doi: 10. (000665////2222..oooo
[60] Rong Zhou. Research on key performance index prediction of distributed database based on machine learning algorithm. In Proceedings of the 2nd International Conference Cognitive Based Information Processing and Applications (CIPA 2022) Volume 2, pages 563-567. Springer, 2023.
[61] Xuanhe Zhou, Guoliang Li, Chengliang Chai, and Jianhua Feng. A learned query rewrite system using monte carlo tree search. Proceedings of the VLDB Endowment, 15(1 (5665,,,, 2021. doi: 10.14778134855503445555
15

---

[62] Xuanhe Zhou, Chengliang Chai, Guoliang Li, and Ji Sun. Database meets artificial intelligence: survey. IEEE Transactions on Knowledge and Data Engineering. 34(3):006611116 2022.
doi: 10.1 00.10999KDDEE22222229444411
16

---

A Datasheet for Datasets
We follow instructions provided by Datasheet for Datasets answer the important ques- tions considering this dataset.
A.1
Motivation
For what purpose was the dataset created? The advancement of Large Language Models (LLLM)) has raised concerns regarding whether state-of-the-ard LLMss such as ChatGPT and Codex, can replace human effort in real-world text-to-SQL. tasks involving large database values. That because their exceptional performance on previous academic tasks like SPIDER impresses researchers. However, we observe that current cross-domain text-to-SQL benchmarks only focus on the database schema, which lacks full attention to values, resulting in gap between academic and real-world applications. To address this issue, we introduce BIRD, the largest cross-domain text-to-SQL benchmark highlighting extensive and realistic databases for community development. Additionally, hope to observe the performance gap between LLMs and humans. Our experimental results indicate that, as now, LLMs are still unable replace human effort. As far know, BIRD is the first text-to-SQL benchmark to collect human performance.
Who created the dataset (e.g., which team, research group) and behalf of which entity (e-g., company, institution, organization)? Please refer to the author list for details. Our research team involves Star Lab The University of Hong Kong, Alibaba DAMO Academy Conversational Al (ConAl) Team, the Department of Computer Science at the University of Illinois Lrrana-Champaaig the Department of EECS at Massachusetts Institute Technology, the School of Data Science The Chinese University of Hong Kong (Shenzhen), and Database Group of Tsinghua University.
Who funded the creation of the dataset? This dataset is fully funded by the Alibaba DAMO Academy ConAl team. We spent 97,654 USD for presenting this data. The budget includes 10% for recruiting competent research interns, 80% for developing the benchmark, and 10% for refining and implementing the benchmark.
A.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? BIRD contains natural language questions, external knowledge evidence sentences, processed large databases, database description files (csv), and SQL queries.
How many instances are there in total (of each type, if appropriate)? BIRD contains 12,751 natural language questions, 12,751 external knowledge evidence sentences, 95 processed large databases, 95 folders of database description CSV files, and 12,751 ground truth SQL queries.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from larger set? In BIRD, divide it into three sets: training, development, and testing. Training and development sets are public while testing data set hidden for the fair evaluation of all text-to-SQU challengers. This could witness the real development of text-to-SQLs Ls in the LLM era.
Is there label target associated with each instance? In BIRD, we provide two labels for each question instance: SQLs (the target of input) and external knowledge evidence (expert annotated evidence for each expected SQL).
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? No.
Are there recommended data splits? Our data consists of 9,428 instances for the training set, 1,534 instances for the development set, and 1,789 instances for the concealed test set. The training
17

---

and development sets are derived from public databases, while the test set databases are curated and designed by our specialized team. We do this because some researchers express concerns that the remarkable performance of LLMs in text-to-SQI. tasks may not be attributed to an improvement in capabilities, but rather to the exposure of data and database values to the LLMs during the pre-training phase. To address these concerns, we opt to self-design new databases in testing using actual tabular data, thereby ensuring that LLMs do not preview the databases.
Are there any errors, sources of noise, or redundancies in the dataset? As stated in the main content, our double-blind annotation procedure is both expensive and rigorous, ensuring data quality. However, it is virtually impossible for any dataset, especially complex ones, to be entirely free of errors. Our team is committed enhancing the data even after this paper is accepted, thereby contributing to the text-to-SQL community. In addition, we encourage users to provide feedback and report errors on our data website, allowing us to rectify and enhance the dataset.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e-g., websites, tweets, other datasets)? Yes. all databases in training and development are collected under appropriate licenses. Please see Section 3.2 for more details
Does the dataset contain data that might be considered confidential (e.g,, data that is pro- tected by legal privilege by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.
Does the dataset identify any subpopulations (e.g,, by age, gender)? Some questions mention ages and genders, but they are just used detect the capability of models on text-to-SLss. No bias or other opinions are involved.
Is it possible to identify individuals (i.e., one indirectly (i.e., in combination with other data) from the dataset? No. All databases are collected from open-sourced platforms, and any sensitive data has already been processed before.
natural persons), either directly or
Does the dataset contain data that might be considered sensitive in any way (e-g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? No, this is QA-based text-to-SQL dataset, we don't require models to deliver any opinions on results. And also we don't it present any bias or opinions in the dataset.
A.3 Collection Process
How was the data associated with each instance acquired? Section 3 and Appendix B.2 introduce this in detail.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APls)? Section 3 and Appendix B.2 introduce this in detail. Our crowdworkers use Alibaba internal labeling software to annotate the data and examine the results.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities? No.
Who was involved in the data collection process (e-g, students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Four PhD students and two MS students are involved in the creation of database description files. Two independent teams of crowdworkers are recruited to annotate questions and SQLs. The question annotators are composed of 11 English native speakers and SQL. annotators are comprised of database engineers and DB students. The total consumption is 97,65 655 USD.
18

---

Over what timeframe was the data collected? From Sep. 2022 to Mar. 2023.
Were any ethical review processes conducted (e-g., by an institutional review board)? Yes, we take such issues very seriously. During the review process, we found that certain questions related to polities or inappropriate language. We have addressed these concerns by modifying the content and providing serious warning to the annotators responsible for such instances.
Did you collect the data from the individuals in question directly, or obtain via third parties or other sources (e.g., websites)? Section 3 and Appendix B.2 introduce this in detail.
Were the individuals in question notified about the data collection? Yes.
Did the individuals in question consent to the collection and use of their data? Sure, we recruited them and paid them satisfying salaries.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future for certain uses? No.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted? Yes, did very comprehensive analysis including error analysis, and efficiency analysis, the experiments of the paper and Appendix.
A.4 Prepooesiin/dteeinn/aalllng
Was any preprocesingcdeenngglbbiinng of the data done (e.g,, discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Yes, provide the token list for each question and SQLs from NLTK for users.
Was the "raw" data saved in addition to the proproeseedcceannc/hheeeed data (c.g., to support umanticipated future uses)? No. the software was used to preprocess/clean/laee the data available? Yes, https:/wwww.ltk..og
A.5 Uses
Has the dataset been used for any tasks already? No.
Is there repository that links to any or all papers or systems that use the dataset?
No.
What (other) tasks could the dataset be used for? Sure, our databases and analysis-style questions are most valuable, so they could be beneficial to DB-based code generation, data science analysis, etc.
Is there anything about the composition the dataset or the way it was collected and prepro- cessed/cleaneed/abeeeed that might impact future uses? No.
Are there tasks for which the dataset should not be used? No.
A.6 Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? No.
How will the dataset will be distributed (e.g., tarball on the website, API, GitHub)? All source codings and datasets could be found on our leaderboard website: https: //ts:://bi heae:://brrrdeeeeee rddbbench. gi github. 1c/. And we provide fast download links for the convenience of researchers who want to use our big data. Furthermore, the code repository can be found in https / / thub. com/ Al www.aaaeeaaacceeeeecc...aaaaaaaa DAMO Dww.DAAAOOO---teeeeeoeaa aateeee/main//biieeeeee hwt.m/annn/i//ee bird
19

---

When will the dataset be distributed? Now.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? Given the database size of BIRD is the largest until now, we are afraid that abusing ample database values may lead to inappropriate commercial use. Therefore, we claim that this dataset should be distributed under CC BY-NC 4.0.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
No.
A.7 Maintenance
Who will be supporting/losttn/maaaittnnnng the dataset? HKU STAR LAB and Alibaba DAMO Academy
How the ovner/curraoor/maaagr of the dataset be contacted (e.g., email address)? Contact bird. bench3n @ @emaalllcom com or the corresponding authors or co-first authors in the author list.
Is there an erratum?
No.
Will the dataset be updated (e-g,, to correct labeling errors, add new instances, delete instances)? Yes, will keep polishing and optimizing our data periodically.
the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., was the individuals in question were told that their data would be retained for a fixed period of time and then deleted)? No.
Will older versions of the dataset continue to be supportel/hsste/mminnanee??? No. The most updated version will be more reliable.
If others want to extend/augmmenthhiil on/contribute to the dataset, is there mechanism for them to do so? Yes, but they should contact the authors first.
20

---

B Appendix
B.1 Texi-to-SQL Difficulty
In order to help researchers deeply analyze model performance in various text-to-SQL. case levels, class all examples as simple (30%), moderate (60%), and challenging (10%). Previous work, such as SPIDER, computed difficulty mainly based on SQL complexity. However, we find that additional factors, such question comprehension, schema linking, and external knowledge reasoning, also influence model and human performance. Therefore, each SQL annotator is required to evaluate examples based on these factors, and experts conclude the ratings to divide examples into the three aforementioned difficulty levels. This approach offers more extensive difficulty analysis for text-to-SQL tasks. And the performance of ChatGPT on three different difficulty levels is shown in Table I. we take the approach of human scoring under established rules. A detailed crowdsoureing rule is employed to rate the difficulty when SQL annotators generate SQLs for each question. The process consists of evaluating four dimensions: The process consists of evaluating four dimensions:
1. Question Understanding: On a discrete scale from 1 to 3, annotators assess the ambiguity and difficulty of comprchending the question's intent, with being straightforward, being clear but requiring more thought, and 3 being extremely ambiguous.
2. Knowledge Reasoning: On discrete scale from 3, annotators rate the amount of external knowledge required to map the question to SQL., with indicating no knowledge is required, requiring evidence of external knowledge for generating SQLs that is easy to understand, and 3 requiring extensive knowledge and much more thoughts.
3. Data Complexity: need analyzing on discrete scale of 1-3, with I being simple schema and data, being complex schema and values understandable through database description files, and being highly complex and difficult comprehend values and schema even with description files.
Annotators rate the complexity of schema relations and data size that
4. SQL Complexity: Annotators rate the syntactic complexity of the target SQL query on a discrete scale of 1-3, with I being simple SQL without many keywords, 2 being more complicated than 1, and 3 being a highly complex SQL with many functions and
Each dimension is considered equally important for text-to- SQL annotations. SQLs are ranked based on these scores, and we present simple, moderate, and challenging difficulties at proportions of 30%, 60%, and 10%, respectively.
DEV SET
TEST SET
MODEL
simple
moderate
challenging
total
simple
moderate
challenging
total
(EX) ChatGPT
31.08
13.29
12.08
24.05
35.41
19.46
12.28
26.77
(EX) ChatGPT KG
45.44
26.14
19.01
37.22
49.21
31.89
20.70
39.30
(VES) ChatGPT
36,20
15.43
14.42
27.97
50.09
24.71
15.39
36.68
(VES) ChaiGPT KG
54.71
28.16
22.80
43.81
65.06
41.21
25.81
51.40
Table 4: The Execution Accuracy (EX) and Valid Efficiency Score (VES) are presented for both the ChatGPT model and its version with grounding (KG) for external knowledge evidence, taking into consideration development and testing datasets.
B.2 Annotation Entrance
Annotation Platform and Compensation. The data is collected from Alibaba-Appen' an internal version. Each Question annotator receives $0.6 reward for each validated question, while SQL annotators earn $1 per SQL contribution. We also invite text-to-SQL. experts and professors to join to check and annotate external knowledge evidence without compensation. There are - -1340 SQLs confirmed per week.
hntpps: huttpp:/aapea..cmm/oooo--- wwwp///ww////..//o/coo/ooo//ooee ccoodd
21

---

Text-to-SQL Experts. The three full till-time text-to-SQL experts in this project are: (1). A database research scientist who's published over 20 top DB conference papers (e.g. SIGMOD, VLDB). (2). A PhD student with research interests in text-to-SQI, who achieved state-of-the-ar results on text-to- SQL open challenges. (3). A DBA engineer with more than 10 years of experience in text-to-SQI. applications for both B2B and B2C businesses.
Question Annotation Entrance. We hire a group of native speakers of English with degrees above the bachelor's level and database-related knowledge to ask variety of natural language questions regarding the values of databases. To fulfill this objective, we have adopted the following procedure: (1). ER diagrams and database description files are documented to assist the annotators in understanding the databases; (2). we present the annotators with three databases from different domains and require them to generate 10 questions for each database; (3). these questions are then assessed by 3 text-to-SQL experts applying predefined rules. Those questions earning at least two votes are marked as valid. Only annotators capable of generating no less than 8 valid questions per database are preserved. As result, 11 native speakers contribute questions to BIRD.
SQL Annotation Entrance. With the purpose of enhancing the quality of our SQL queries, assemble a team of skilled data engineers and database students. The team undergoes rigorous testing through the text-to-SQL evaluation process, which assesses their capability of generating SQL queries for a variety of questions facing different domains of databases. Each annotator is asked to answer 10 questions, and only those who score least 9 out of 10 will be qualified annotate SQL queries for BIRD.
B.3 Question Distribution
Figure contains the detailed question types and their examples.
B.4 Experiment Details
FT-based Models. T5 is a strong and versatile pre-trained language model (PLM) for text- to-text generation that has achieved state-of-the-art performance in variety of semantic parsing tasks, including text-to-SQI. We concatenate the question with serialized database schema as input [40, 49, 41J. And SQL can be fetched in an end-to-end fashion by easily fine-tuning. While seq2AST- based methods [43, 5] are also effective in text-to-SQL, actually their grammar rules utilized during decoding are constrained on specific datasets [25]. We implement our codes mainly based on the hugging-face transformers library 9, We set the max input length as 1024, the generation max length as 512, and the batch size as 32. We also adopt Adafactor as our primary optimizer with a linear decayed learning rate of 5e-5. All experiments are conducted on one NVIDIA Tesla A 100 80GB, which is available for most research centers. We set the random seed as for all runs of FT-based models since 1 is an optimal seed proven by previous SOTA models [27, 49].
ICL-based Models. Codex (code-davinci-e ular and powerful large-scale pre-trained language models (LLMs) for code generation driven by ICL. They can produce multiple types of codes, including SQL, from human instructions without additional training. We employ programming-basee based prompts, as described in [39], to collect results by calling the API. Also, we choose the Azure OpenAl API to align the codes with other variants of LLMs. Given that models not allowed access to unseen databases and ground-truth SQLs in the evaluation set, a zero-shot generation strategy is the most appropriate. Moreover, to investigate the impact of multi-step reasoning of L.LMs on BIRD, we implement the Of-ho-Of- (COT) technique [48] by easily adding the prompt sentence "Let s chink step by step. before the generation of SQLs [21]. However,
and ChatGPT 5--turb) are pop-
find out the output of ChatGPT is too uncertain with many unexpected explanations, thus we provide a I-shot pseudo example for ChatGPT to learn the procedure of thinking and output format. The detailed prompt design is shown in Figure 9, In order to minimize the randomness of results, we set the temperature as O to ensure reproduction.
Knowledge Fusion. In the baseline implementation, we naively concatenate the knowledge evi- dence sentences with questions and database schemas, but we can observe a significant improvement
hnnps//hugginngfacc.o
22

---

Question Type
Sub Type
Question 'QQ
Percentage
Fundamental
Matcc-bussdd
How many gas stations in CZE has Premium gus?
83,9 %
Type
SELECT COURZ (CasBtation) D) FRON casetation MEBRE Countay
[CEB) 2010 Segnant
'"Preen umn
Ranking
What are the ritles of the top posts with the highest popularity?
20.3 %
SELECT
posta
VievCount DBSC
LIMIP
Comparison
How many color cards with horders have been ranked higher than 12000 on EDHRec?
16.7%
SELBCT (id)
WHERE wdlazweaakk barderColoz 'borderlesa"
Counting
How many of the members' hometowns are from Maryland state?
30.4 %/4
SELECT COUW2 (12 member id) ESCOR sip code AB '1 TNEER J0IN nember AS 12 CN 21.2ip_ posppodd 12.21p 'Naryland"
Aggregation
the average of superheroes from Marvel Comics?
15.7%
SELECT AvG (ma, beight, cn) FROM supeshero publ
spubllisherr id
72.1d WHERS 12.gubliches pane 'Horvel contos"
Reasoning
Domain Knowldgge
Name the ID and age of patienl with two or more Inhoratory examinarions which show hematoclit level exceeded the normal range.
23.6 %
Type
SELBC2 21 2I.IE, STRETIHE (1O97, COICIENS FIMESTAMP) STREZINE ('EY', 21.3iethdayl
71 TNNER J0TN Saboratary AS Facient
22. 1B WHEаE 71.10
SELECE 2D FROM Laboratory WHERE BCT 52 GROUE BY 1D HAVING
Numeric Computation
Among the posts with score of over 20, what is the percentage being owned an
24.5 %
SE18C7
CAST (BOK (ILF (22 Age 65, 1, 01) AB REAL EROM AS INNER
100
WHERE 11.8:ore
20
Synonym
many clients opened were women (female)
Jesenik
7.2%
SELECO COUNDIT) client, id) FROM client district AB 21.sisttiot__ 12 12.distzict, id WHERE 21, gender "p" Allo 12.22 'Jesenik"
Value
Among the weekly issuance accounts, how many have
70.1 %
FROM AS account, 12, a2.oout ld SHEEE 122 Erequency 'POPLATEE TYDNE AND 21. Pllannunn 200000
SE18C7 COUNC
INBER accoint
Figure 8: Questions in the BIRD contain two main categories. The Fundamental Type of questions comparable other text-to-SQL benchmarks. The Reasoning Type of questions requires external knowledge grounding to answer.
by this easy method. A more complicated and effective strategy of knowledge grounding for ChatGPT and T5 would an important future topic. The knowledge evidence sentences are concluded the external knowledge provided by annotators described in Section 3.3.
23

---

ICL Prompt Input
esample
Prompt
singer
CREATE custemers
Esternal Knowledge: August of 2012 means Data contains 200205 the yearmenth d20c of the datesheser Price per unit of product Price Ammunt
singer 18
TEXT not null
Custame11 INTEGER UNIQUE not nul
ormmry key,
primary key.
nation
TEXT not null.
Segment TEXT null,
drame TEXT nul,
1nsme
TIXT rull,
Currency TEXT null
Instruction
noll
understadng
INTEGER
Know edge, answer following questions the tables provided above.
waar INTEOER
CREATE gasstations
TEXT TEXT null
salary REAL nall
GasSuationDD INTEGER UNIQUE not
Question Prompt
phone number INTEGERR
sul,
primary
For the people who paid more 29,00 per unit of product No. Give their consumpl on statiss in the August 2012.
lee REAL mull.
ChainDD INTEGER Countyy TEXT null
nil
I
Segment TEXT nul
birth, Using valld SOUns and understading External knowledge, anower the tollowing questions for the tables provided above.
Kroodaddger
1-shot COT Prompt
The final SOL is: Let's think stop by stea.
The fimal SQL in Let's think step by step.
CREATE TABLE yearmonth
ChotGPT COT
bnowdedge,
konwledgg,, No which price per unt of product fheer people who paid per unit of product is No 3, should out the consumption statiss of the people in August of 2012.
singers by yoar"
Date
to
we should find the singers step netion
Consumption REAL
Amount; than
CDUNTT)) to courn how many singens Finally the SQU in: SELECT COUNT!") FROM singer WHERE your Srrttyycar>> 27/4/p>
primary key (Custamertf, Date).
foriign bay (Caviomeril)) (D) telerennes customers (Castmmril)
SQL Output
Finally the SQL
SELECT yearmonth..onsumgeenee Consumption FROM transactions, 1k JOIN yearmonth transactions, Cu.ttosomrrlD yearmonth yearmonth.CustomeriD CustomerID
AND transactions, Date yearmonth.Date WHERE transactions Ik.ProductID 5 AND transactions Price / transactions_ Amount > 29.00 AND yearmenth.Date '201208;;
Figure 9: The detailed prompt design for implementation of ChatGPT + KG + COT.
B,5
Efficiency Analysis Details
Two strategies for performing text-to-efficient-SO0 are presented in Figure 10. Examples show that both two-stage optimization and embodied databases can help semantic parsings generate more efficient SQLs.
B.6
Error Analysis Details
Figure ll presents a detailed analysis of errors made by ChatGPT.
B.7 Evaluation Details
During double- blinle annotation in BIRD, we encountered numerous ambiguous issues that led mismatches, predominantly due unclear user intents. The most serious ambiguity is the use of "DISTINCT". Some annotators believe it should present only unique values, such as names, and cities, while others argue that it should be used only when questions explicitly mention "di ffefeettt" or "di st inct i ive". Therefore, we use HashSet rather than List to compare final results since HashSet disregards row order and automatically filters repetitive rows reduce this am- biguity. However, this may result in false positives for questions utilizing "ORDER BY." We identify three "ORDER BY" usage scenarios in BIRD: 1) Rank-based questions (e.g., "Show me the top 5 students according to their math scores "): The order is less important as long as the results contain the correct students. 2) Superlative questions: (e.g.. "List the lengest river in the USA"): The answer typically contains only one item (or tied results), so the impact is minimal. 3) Questions requiring specific order (e.g., "Show me the top five students based on their math scores in descending order"): This scenario explicitly requires correct ordering and may lead false positives. However, such in- stances uncommon, accounting for less than 1% of BIRD.
24

---

Query Rewriting
Ex1.1 Question:
Last of who locsted Vienna, Austria obsained the badgo?
ChatGPT SQL:
GR1XKT Age FROM users WHERE Tocation 'Vienne, Austriea
(SELECT Dsarid FROM
badges)
Optimized SQL (limo-asvigg parcentage: 99.92%)
SEL3C7 G.age FROM users AS 9 INKER J0I8 badges VVennea Aus Lil'
ON u.Id b. b.UscIId KHERE u.Location
Away:
By applying a JOHN operation usstead o subquery with IN can suprove efflciency, as the database may euecuse the JOIN and filering processes concuriently y in just one operttion without the nood to store the intermediate reuills to filter primary quary.
Ex1.2 Question:
How of the members' hometowns from Mary state?
GRLUXCT COONE
nember JOIN xip_code gode menberr.dig xip, pode..ipp pode "Hay yland!
s.p wode Le
Optimlzed SQL: saving percentage:
SELECT COONS (ousbber member id) FRGN membez INNER JOCN kip code mumber..xip
55 code, 2-1 1.p cote WHRRR xpp wwweoodettttt atate 'Maryl land!
Away:
Utilizing COUNT functio on NOT-NULL column, as apposed COUNT("), can increase time efficiency. This rewritten SQL aubles the datibroe count NOT-NULL -UL...vuuus within singl colute,, niller than compute all nows including those with colume is selected this NOT-NULL column
NULL valoes. Unallyy the
Ex1.3 Quesiiin::
Who the owner of the account with the lanest loan amount?
ChaiGeT SQL:
SELECT c-clienc id FROM client e TINER CorN di sp CH exclient, fd d.client TINER 3OTK Town ON d.eccouns Id Lvecesunt_ id ORDDER BY L.monnn DESC LINIS
Optimized SQL: saving pencemage:: 62.39%)
gRLEECT evel lens id cliient e TEMER corN disp GM eveliient d.el
I8NER J0EN Town: 1 oa d.eccountt id sucuant, id Wuzze 1.stuuul SELECT
NAX (amount)
FROM lown)
Take Away:
In an unindesed envirenment, employing the MAX function can potentislly yirld flster results since avoids the need for sorting. which could run against larg table.
Adding Indexes Database
Question:
How many accounts are there the distnet of "Pisako"?
CharGPT
SELBCC COUND(")
distriot a.district d.district
4.A2
Added Indexes: (time-saving 87.27%)
account district,
(district_
UNIQUE INDEX distsius, distsipi, ieet sindex distsiot (diutrict id)
Take Away:
Adding indesss into
significantly speed of scuaning engine guickly rows match specific
eubble
Figure 10: Two possible solutions and explanations to improve efficiency are presented. The first batch of examples shows how optimize SQL efficiency by rewriting SQL based on rules. The last example is to show that adding indexes to databases can also improve SQL efficiency without rewriting them.
B.8 VES Details
Regarding E, in our experiment, we consider time as the main metric to represent efficiency, where E E (c, 30s). Here, is small positive constant to prevent floating-pol point overflow. The single E is not stable due to machine status. Lower E refers to faster execution speed, which is more efficient.
Concerning R, it represents normalized efficiency ratio between human-annotated SQL queries and predicted SQL queries to reduce the influence of machine status. The stability of this metric ensured by rumning this computation 100 times for each example, filtering outliers, and subsequently
25

---

Wrong Schere Linking (41,6%
Mbandersttndngg Komneogg bvstenee (174%)
Quarte::
Oswike:
oes of senom
Howe
1991
temnn bav Sedee Cargeetio feal
hanc doncan shhde Rnceds wiich poinal
fe Labk lypert fow wlock
Indeee::
Evilace
WUVDESSUNgggat COUNTspent wask
Tox: Vocc de yur moas outt thead
Gread Trans
Groand
achocks X3 1880 2008 Frpe 12
oe "1222" cecace
Gread Inis:
ceaD0s Suanaar, 12, Cheetec Pasding
TAL Mareet p0-122
budget Mast catepory "Vieot Alb
Faetie cast peaeert auglenttaing
CheatPF
ChitGPT sout
Checait NUE.
ChaGGe SQU:
1-cecCode 2-0000oda VESSE
rESSAST baciet racm
maar NaLPa coiing "rors" 2012
eclooca AnD court "e" Ade P.Pandirohyp
Jagua 5-17) 20
Abanddrntzadgng Datiatase Content (41l3%)
Syntax Erree (3.7%)
Uemtke:
Querdie:
gydd, and in wil
hul??
fire aub comnaiig 2013
Erieat
grou d'u Hlukjcccin padl
Fishres
modtag eegereet edm Meringg held 303
Y3Allane das 2033
Eridasec
Grosel Tratk
Greand Trets
Greeal
26 19399 artt acteatls an
cade 1o0 and Liwcoorda As
SELECT COCSC lanent, 145 EROM evens
CONCEET (ConcomeeDD = 312 208 Courtoract
22 Co.dda E2.c0oooda
neees names bo
33587166 1p", 0032800 "2322"
BLArggEGnee MALL CARRA
T1 T1LANqACSNeCS Aogacccesd
ChacPT SQU
20TH rchoole satarores
setwola c000ode MaEREDERO strpa
Bep MEEE eicmate CE ('ooze",
ChaGPe sgu
ChaiGT SQUL
Ioy" van (nea Jewert) (atk))
Chersiy SQU
221304) CSBLECT Conecaptiion FRC
KIDE Conmumption 2006
Juyit ateee LIMIT
UODER CkrgSerteess
231306)
Figure 11: 4 major types of error cases are presented. Some cases are shortcuts for better presentation.
computing the average. Considering the rapid advancement of technology, it is impractical to anticipate the fastest SQL performance. Currently, the range for the efficiency ratio, R, defined as Re E (0, +oo). E(x) (efficiency of predicted SQL) is much lower than EYY (efficiency of ground truth SQL according to EX), then the relative efficiency score R will be increased. In short, higher R refers to higher efficiency.
When measuring VES, we run 100 times for each SQL in the same CPU and evaluate average results after dropping the outliers. The STD of VES on dev set and test set after 10 trials are 0.043 and 0.025 respectively. We detect outliers in the following procedures:
1. Compute the mean and standard deviation of the dataset.
2. Then calculate the lower threshold as mean 3 standard deviation and the upper threshold as mean + 3 x standard deviation.
3. Statistically, approximately 99.7% of the data points fall within 3 standard deviations of the mean.
B.9 Human Performance Collection
The procedure of collecting human performance is still rigorous. During the annotation, all data is divided into 10 batches for better management and error tracks by experts. The first 8 batches of data are the final training data and dev data for public use, and the remaining 2 batches of data are used for testing. We consider the annotation of the first 8 batches of data learning process for SQL annotators since their erroneous SQLs could be fixed by experts and learn how to generate good-quality SQLs for this task. Then their first scores set from the final two batches can be viewed as human performance since we don't 't interrupt and assist them during the examination and all errors are preserved. After testing, we proceed with the following double-blind SQL. annotation procedures as Section 3.4 to correct SQLs for these data by a discussion with experts. And SQLs after the second round of double-blind annotation are collected as ground truth.
examination, conducted by testing
B.10 Distribution of Open-source Databases
The databases in BIRD are all in accordance with one of following licenses:
26

---

Public Domain Public Domain Mark
public domain license refers to legal designation that allows intellectual property, such creative works or inventions, to be freely used, shared, and built upon by anyone without restrictions. When work is in the public domain, no longer protected by copyright, patent, or trademark laws.
Creative Commons Attribution 4.0 International This license is one of the open Creative Commons licenses and allows users to share and adapt the dataset so long they give credit to the creator.
CC-BY
CC-BY-SA
Creative Commons Srribution-Shar...e 4.0 International
This license is one of the open Creative Commons licenses and allows users share and adapt the dataset so long they give credit to the creator and distribute any additions, transformations, or changes to the dataset under this license.
GPL General Public License
The GPL was created by the Free Software Foundation (FSF) and also known as the GNU GPL, as it is used by the GNU Project. And it allows users use, study, share, and modify the software under certain terms and conditions.
CPOL Code Project Open License
It is a software license that is often used for articles, tutorials, and sample code shared on The Code Project website. The CPOL is intended to more permissive license, allowing developers use, modify, and distribute the software without many of the restrictions imposed by other licenses like the GPL.
CCO Creative Commons Zero
is a public domain dedication tool created by Creative Commons. It allows creators to waive all their copyright and related rights in work, effectively placing it in the public domain. This means that anyone can freely use, share, modify, and build upon the work without seeking permission or providing attribution to the original creator.
B.11
SQL Function Taxonomy
SQL functions in BIRD mentioned Table span across multiple categories including:
Window Functions, i.e., OVER ()
Date Functions, i.e., JULIANDAY ()
Conversion Functions, i.e., CAST ()
Math Functions, i.c., ROUND ()
String Functions, i.e., SUBSTR ()
B.12 Keyword Statistic
We have conducted comprehensive analysis of the keywords employed in the BIRD dataset and visualize the results in the form of a nice-looking word cloud, which can be found in Figure 12. We further classify keywords into following categories:
Main Body Keywords SELECT FROM WHERE AND OR NOT IN EXISTS IS NULL IIF CASE CASE WHEN.
Join Keywords INNER JOIN LEFT JOIN ON AS.
Clause Keywords BETWEEN LIKE LIMIT ORDER BY ASC DESC GROUP HAVING UNION ALL EXCEPT PARTITION BY.
27

---

NOT AND
DESC AND
NOT
STRFTIME
ABS
FROM
pre-trained language is JOIN
PARTOOG
DISTINCT
ARD
IN
ES
ON BY
WHESE
NONE
ALL
HE
MAR
AS
LEFT
PRTTO NOW
DESC
INNER
ROUND
AS
CAST
LIMIT
MEN
BOVING WEST TR
BET BETWWEN
OR
ASC
CONTO
UNION
FROM MIN
WHERE
BETWEEN
IS
LIKE
BOLADAA
B-Y
LEHOTH
LEFT
OF
GROUP
THEN
AVG
SELECT
ON
IN
FEO
OR
LIMIT
UIAL
NUUL
SUM
CAST
CASE
SUBBTR
IS
ORDER
HAVING
MAX OR
CASE
SELECT SEAR
LIKE COUNT
DRDER
Figure 12: Keyword cloud presentation for SQLs in BIRD.
Aggregation Keywords
AVG COUNT MAX MIN ROUND SUM.
Scalar Keywords ABS LENGTH STRETIME JULIADAY . NOW CAST SUBSTR INSTR.
Comparison Keywords
.
Computing Keywords
B.13 Study about TeX Models
The fundamental principle of cross-domain text-to-SQL. parser involves the construction of an encoder to learn representations of questions and schemas, followed by decoder to generate SQLs [37]. For example, IRNET [12] designs an encoder consisting of attention-based Bi-LSTM for learning question and schema representations, and a decoder to predict SQLs based on the encoded intermediate representations. RATSQL [43], SDSQL [17], LGESQL [5], and S [18], Proton [44] enhance the representation learning of natural language questions and database schema via relational graph neural network. R2SQL [16], SCORE [54], and STAR [4] enhance contextual learning for conversational text-to-SQL. tasks. Later, sqquence-to-sqquee
models (PLMs) such as T5 [38] become popular in text-to-SQI tasks due to their portability and capability of generation across different datasets. These models achieve impressive results by fine- tuning with minimal effort. Furthermore, RASAT [36] enhances T5's structural information encoding via schema alignment into the encoder, while Graphix [27] equips T5 with multi-hop reasoning to achieve state-of-the oheet-rttttrtttt results on complicated cross-domain text-to-SQL tasks. In recent years, LLMs such as ChatGPT [33], Palm [8], OPT [56], have attracted considerable attention due to their powerful zero-shot reasoning and domain generalization capabilities. ChatGPT can perform exceptionally well on semantic parsing tasks, including text-to-SQL. tasks, with minimal input data. In fact, in the BIRD project, ChatGPT even performs more impressively than initially expected.
Study about SQL Efficiency Efficient execution of SQL queries on big databases has been significant topic in both academia and industries. Many techniques are proposed to improve SQL query efficiency, by index selection [22], SQL optimization [26, 61]. etc. SQL optimization is common method for enhancing the efficiency of SQL queries. Several SQL optimization algorithms [28, 30, 47], such rule-based optimization and cost-based optimization, are proven effective. Rule-based optimization employs a set of principles to transform the SQL query into a form that can be executed more efficiently. On the other hand, cost-based optimization estimates the execution cost of various query plans and selects the one with the lowest cost by analyzing the statistic distribution of database values. Similar the NLP community, there are also recent works utilizing artificial intelligence for query optimization such as [61]. Index prediction is another important technique for improving SQL execution efficiency. Researchers propose many algorithms of index prediction [60] based on various optimization criteria, such as minimizing SQL execution time, and maximizing index utilization. In this work, we provide VES measure the efficiency of text-to-SQL generators to encourage them to generate accurate and fast SQLs for users.
28


Reviews:
[{"review_id": "gEE67NIZe5", "reviewer": "Reviewer_Kfee", "summary": "", "questions": "", "limitations": "Even data analysis platforms, which may eventually or potentially have complex schemas and a large scale of values compared to previous benchmarks such as SPIDER, often undergo post-processing from their original data lakes. This is typically done to align with specific goals, such as dataset competition (e.g., Kaggle) or machine learning research.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- With the emergence of Large Language Models (LLMs), the text-to-SQL community has recognized a need for more challenging benchmarks. Although several recent works have acknowledged this need and proposed challenging benchmark datasets, no general-purpose, large-scale cross-domain benchmark is currently available for the community. To fulfill this need, the authors have proposed a new dataset, aiming to reflect the demands of real-world, large-scale applications.\n- The authors have shifted the focus of text-to-SQL tasks to text-to-efficient-SQL, introducing an evaluation metric known as the Valid Efficiency Score (VES). This adds an important dimension of efficiency to text-to-SQL evaluation, essential for practical and large-scale applications.\n- Although some details may be missing, the research has been conducted with a rigorous annotation process and detailed experimental analysis. This indicates a high level of quality and careful effort.\n- This work includes an interesting approach of comparing model performance with human performance (during the 1st round of the annotation process), providing a valuable benchmark and context for evaluating progress in the field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "d58iduyieZ", "reviewer": "Reviewer_QBSP", "summary": "", "questions": "", "limitations": "* The questions are classified into fundamental type and reasoning type, would be nice to see benchmarking results detailed on each category, preferably both macro- and micro-categories. \n* In section 6.3, it is not clear what is the purpose of mentioning Two-stage optimization, and how it is connected to BIRD. In addition, more explanation and results should be presented to elaborate the significance and efficacy of “Chat w/ Database”\n* Some  potential improvements can be done for reader-friendliness, to name a few: a) Figures for illustrative purposes, e.g. Figure 1 and 2 are difficult to read due to small text size. b) Are results in table 2 and 3 percentage? Would be nice to note down. c) Line 195, typo tex-to-SQL. d) Line 194-196 seems like the two scenarios are mixed up in terms of which one requires self-knowledge grounding. e) Line 208 mentioned a non-existing table 4.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The proposed benchmark scheme takes execution efficiency into consideration which is an important addition to the current evaluation scheme.\n* Curated dataset is featured as large-scale, multi-domain coverage and various question/query categories, resembling closer to real-world scenario compared to currently available alternatives.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "GRs1Tj7FDr", "reviewer": "Reviewer_YmxV", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It is clear that a lot of annotation effort and quality control have been put into the benchmark construction. Compared to previous large-scale academic text-to-SQL benchmarks, BIRD introduces several challenges that are closely related to real-world scenarios. I believe this is a highly valuable benchmark for the community.\n\n2. The evaluation of SQL efficiency is new in the text-to-SQL domain. This evaluation further demonstrates that existing text-to-SQL models are still far from optimal regarding generating efficient SQL queries.\n\n3. The authors curated the test data by the specialized team, so that the test set is less likely to be exposed to LLM training.\n\n4. There is an extensive evaluation of different models and approaches for text-to-SQL generation. While on Spider, the performance of fine-tuned T5 models is not much worse than prompting with GPT models, it is interesting to see that GPT models perform much better on BIRD than finetuned models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9k8YEHPWYS", "reviewer": "Reviewer_3iDu", "summary": "", "questions": "", "limitations": "Comments on limitations, e.g. different query engines, are included in supplemental material, but not in the main paper.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Realistic benchmark for text-to-sql with more complex queries and databases that cover a variety of domains, which is a much needed contribution.\n- BIRD introduces background knowledge e.g. schema information that is shown to improve text-to-sql methods. BIRD also introduces an efficiency metric for evaluating the efficiency of generated SQLs, which is an important novel aspect to evaluate methods on.\n- The experiments and analysis support the motivation of the benchmark, e.g. efficiency metric and external knowledge, and compare the content of the benchmark and model performances on the most commonly used existing benchmark, Spider.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VfKoednqUC", "reviewer": "Reviewer_fFFH", "summary": "", "questions": "", "limitations": "- In section 3.4 it is stated that \"if executed result set is \"NULL\", experts will make *slight* changes [...] until the associated SQLs can provide a valid result set\", then further on \"if the evidence is insufficient to generate SQL or contains errors, experts will be in charge of correcting them\". This is a critical lack of detail on the kind of corrections that were required and the amount of effort needed to fix the LLM-generated SQL in order to get them to work. If the expert-corrected queries were considered correct for the EX and VES scores (as the result of the query was correct), then they should still be penalized in some way due to the fact that human intervention was required to make them viable. There should be some figure on how often this pattern occurs, and partially correct queries should also be kept track of and tallied when evaluating the performance. \n\n- As querying LLMs incurs in a non-negligible cost (as is also pointed out by the authors, e.g., in section 6.4), the query cost/cost per token should also be accounted for when selecting the most suitable LLM, as it would be a relevant metric to rank models by.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The submission introduces BIRD, an extensive benchmarking dataset to evaluate text-to-SQL tasks which includes hard datasets and requires reasoning to include external knowledge on the DB to produce an optimal result. \n\n- The Valid Efficiency Score (VES) metric is introduced as a means of evaluating not only the correctness of the query result provided by the LLM, but also whether or not the query is efficient according to some metric (e.g. the execution time). \n\n- By emphasising the need to reason and understand external information, the benchmark can model a more realistic scenario for how a human would approach the problem of creating a SQL query on a given Database. This makes the benchmark more realistic and highlights how LLMs need to be able to understand external information to work well. \n\n- The experimental section provides good insight in the shortcomings of one of the methods under evaluation (ChatGPT), suggesting potential avenues of improvement.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HbJj2HRSzz", "reviewer": "Reviewer_7VcG", "summary": "", "questions": "", "limitations": "- As incorporating external knowledge evidence is one the core contributions, is there an ablation study that illustrates how each of four categories (Domain Knowledge, Numeric Computing, Synonym, Value Illustration) affect the performance improvement in models?\n- How is external knowledge consumed in T5 models during training?\n- Is there a fixed rule for annotating SQL difficulty ratings?", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- A new large-scale collection of realistic text-to-SQL dataset\n- Incorporating external knowledge paired with the database\n- Incorporating SQL efficiency for the first time in text-to-SQL evaluation\n- Clearly written paper\n- Data generation process is easy to follow", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work proposes BIRD, a new benchmark dataset for text-to-SQL parsing. Given that the previous benchmarks such as Spider and WikiSQL consist of relatively simple databases (simple schemas, small rows), and the rapidly increasing text-to-SQL performance of recent models such as ChatGPT, the community is in need of a more challenging and realistic dataset. BIRD is certainly a timely contribution, consisting of complex databases from diverse domains, incorporating the need for external knowledge, and the consideration for execution efficiency. This will no doubt generate interest from many researchers in the field of text-to-SQL parsing.
Decision: Accept (Spotlight)