SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinyang Li $^{1, \clubsuit}$ , Binyuan Hui $^{2, \clubsuit}$ , Ge Qu $^{1, \clubsuit}$ , Jiaxi Yang $^{2}$ , Binhua Li $^{2}$ , Bowen Li $^{6}$ , Bailin Wang $^{5}$ , Bowen Qin $^{2}$ , Ruiying Geng $^{2}$ , Nan Huo $^{1}$ , Xuanhe Zhou $^{3}$ , Chenhao Ma $^{6}$ , Guoliang Li $^{3}$ , Kevin C.C. Chang $^{7}$ , Fei Huang $^{2}$ , Reynold Cheng $^{1}$ , Yongbin Li $^{2}$
<sup>1</sup> The University of Hong Kong <sup>2</sup> DAMO Academy, Alibaba Group
3 Tsinghua University 4 Shanghai AI Laboratory 5 MIT CSAIL
<sup>6</sup> The Chinese University of Hong Kong, Shenzhen
<sup>7</sup> University of Illinois at Urbana-Champaign
{j10725,quge}@connect.hku.hk,ckcheng@cs.hku.hk
binyuan.hby@alibaba-inc.com
Text-to-SQL parsing, which aims at converting natural language questions into executable SQLs, has gained increasing attention in recent years. In particular, GPT-4 and Claude-2 have shown impressive results in this task. However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database values leaving the gap between academic study and real-world applications. To mitigate this gap, we present BIRD, a BIG bench for laRge-scale Database grounded in text-to-SQL tasks, containing 12,751 text-to-SQL pairs and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the new challenges of dirty and noisy database values, external knowledge grounding between NL questions and database values, and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL models must feature database value comprehension in addition to semantic parsing. The experimental results demonstrate the significance of database values in generating accurate text-to-SQLs for big databases. Furthermore, even the most effective text-to-SQL models, i.e. GPT-4, only achieve  $54.89\%$  in execution accuracy, which is still far from the human result of  $92.96\%$ , proving that challenges still stand. We also provide an efficiency analysis to offer insights into generating text-to-efficient-SQLs that are beneficial to industries. We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research. The leaderboard and source code are available: https://bird-bench.github.io/.
Text-to-SQL parsing [55, 50, 51, 3, 52, 37], which focuses on converting natural language questions into SQL queries, has attracted significant research interests from both academia and industry. This attention stems from its potential to empower non-expert data analysts in automatically extracting desired information from ubiquitous relational databases using natural language. Recent advances in neural models, including those based on large language models (LLMs), have led to an impressive performance on existing benchmarks such as SPIDER [53] and WikiSQL [58]. For instance, the execution accuracy of the top-performing model in the SPIDER leaderboard has increased from  $53.5\%$
$^{\ddagger}$ Work done during an intern at Alibaba DAMO Academy.
Figure 1: Examples of challenges in our BIRD benchmark. 1) databases contain values of noisy data types [14, 23, 19, 31]. In the left example, the average salary could be fetched by processing the data type from string (TEXT in SQLite) to float (REAL in SQLite) after deleting the special tokens, "US$" and "", ". 2) external knowledge and reasoning are required. In the middle example, models must handle that only "OWNER" accounts are eligible for loans. 3) query execution efficiency needs to be considered. In the right example, the adoption of more efficient SQL queries leads to significant gains in speed, which is of great value in industries.
[59] to  $85.3\%$  [35] over the past three years. The latest SOTA parser [35] in SPIDER benefits from the powerful understanding and coding capabilities of the large language model (LLM), and such excellent performance leads us to ask a question: Can LLM already serve as a database interface?
The answer is no, as previous benchmarks focus on database schema with few rows of database values leaving the gap between academic study and the real world. As shown in Figure 1, first, we discovered that current state-of-the-art models still struggle to generalize to more realistic situations characterized by large database sizes and noisy values. Second, the growth in database sizes often results in much context compression, making it challenging to reveal the entire context [1]. Thus it requires external knowledge reasoning for a comprehensive understanding. Third, existing benchmarks do not account for SQL execution efficiency, which holds significant practical importance in real-life applications, notably in the case of large databases. Motivated by these observations, we aim to develop a new text-to-SQL benchmark that better represents real-life scenarios and narrows the gap between experimental and practical settings.
In this work, we propose BIRD, a BIg Bench for LaRge-Scale Database Grounded in Text-to-SQLs for real-world applications. BIRD contains complex 12,751 examples of querying information over 95 big databases with a total size of 33.4 GB spanning 37 professional domains. For training and development, we collected and modified 80 open-source relational databases from real analysis platforms (Kaggle, Relation.vit). To further avoid data leakage, we curated 15 additional relational databases for a hidden test set. Given these databases, we rely on crowdsourcing to collect natural language questions and the corresponding SQLs. Additionally, we propose a new evaluation metric Valid Efficiency Score (VES) to evaluate the efficiency of generated SQLs. To the best of our knowledge, BIRD is the first text-to-SQL benchmark to incorporate efficiency, promoting more efficient query methods within the context of massive and noisy database values.
We evaluate the performance of state-of-the-art text-to-SQL parsers using two popular methodologies: fine-tuning (FT) with T5 [38], and in-context learning (ICL) with advanced large language models (LLMs) such as ChatGPT [33] (gpt-3.5-turbo), Claude-2 [2] (claude-2.0), GPT-4 [32] (gpt-4-32k). Our experimental results demonstrate that the current models struggle to generalize well on BIRD. Specifically, even the GPT-4 only achieves  $54.89\%$  in execution accuracy. In comparison, the performance still lags far behind the human performance of  $92.96\%$ , proving that challenges still stand. Moreover, we perform a comprehensive analysis to provide insight and direction. We encourage further research by the NLP and DB communities to jointly address the more realistic settings presented in this benchmark.
2 Task Formulation & Annotations
Text-to-SQL refers to the process of converting a natural language question  $\mathcal{Q}$  into a SQL query  $\mathbf{Y}$  capable of retrieving relevant data from a database. The database can be represented as  $\mathcal{D} = \langle \mathcal{C},\mathcal{T}\rangle$
Figure 2: An Overview of the BIRD Annotation Workflow in (a). This figure depicts a four-step procedure. (1) The workflow begins with specialists assembling and producing databases and description files. (2) Experts then teach and evaluate crowdsourcing people, keeping only those who pass the evaluation. (3) Question annotators create a corpus of questions using databases and their corresponding description files. (4) SQL annotators produce SQL files, equipped with databases, descriptions, and questions. (b) and (c) also depict the Double-blind annotation procedure and an example of database descriptions.
where  $\mathcal{C}$  and  $\mathcal{T}$  are columns and tables respectively. When dealing with complex database values, such as BIRD, it is crucial to incorporate external knowledge evidence, denoted as  $\mathcal{K}$ , to improve the models' understanding of database values. Finally, the text-to-SQL could be formulated as:
$$
\mathbf {Y} = f (\mathcal {Q}, \mathcal {D}, \mathcal {K} \mid \boldsymbol {\theta}), \tag {1}
$$
where the function  $f(\cdot \mid \theta)$  can represent a model or neural network with the parameter  $\theta$ .
3 Dataset Construction
3.1 Annotation Entrance
To deliver a high-quality benchmark, we administer thorough exams to all applicants and only hire those who pass these rigorous tests. Further information is available in the Appendix B.2.
3.2 Database Source
It is difficult to collect databases with complex schemas and sufficient value due to privacy protection. Earlier works [45, 53] choose to self-design database schemas and value production. Nonetheless, the value distribution and schemas may differ from real-world scenarios in this way. In our work, we obtain and process databases from three different sources to enrich real-world attributes.  $32\%$  of our databases are sourced from Kaggle*, a platform renowned for holding data science competitions with difficult, noisy values and schemas. Another  $48\%$  come from CTU Prague Relational Learning Repository†, an open platform for machine learning research with multi-relational data. The remaining  $20\%$  are built by acquiring open tables, synthesizing and standardizing schemas, and generating database constraints. All of these databases contain real and large value distributions and are easily accessible with the appropriate licenses. Finally, we present 95 databases consisting of 69, 11, and 15 databases for training, development, and testing respectively. Our databases cover 37 professional domains, including blockchain, sports, health care, politics, etc. We anticipate that it will be a significant resource for researchers to explore domain generalization in semantic parsing tasks with large database values.
\*https://www.kaggle.com/
$^{\dagger}$ https://relational.fit(cvut.cz/
3.3 Question Annotation
Database Description File. The Database Description File is a crucial resource designed to aid annotators in comprehending database values, thereby allowing them to ask insightful questions. It offers two primary pieces of information regarding the database. (1) Full schema names: database table and column names are frequently abbreviated, which are difficult to understand. (2) Value description: this aspect is particularly useful when phrases or tokens in a question do not directly match values in the database.
External Knowledge Evidence. In our study of professional data analysis, we find that external knowledge evidence is required to map the natural language instructions into counterpart database values. Therefore, we collect and classify such evidence into four categories: (1) Numeric Reasoning Knowledge: this category refers to the mathematical computation required for certain SQL operations. In our benchmark, we present 8 basic math operations, including 4 complex operations as [7]: MINUS, ADDITION, DIVISION, MULTIPLY. BIRD also contains compositional operations over basic ones, such as percentages, formulas, etc. (2) Domain Knowledge: this category consists of domain-specific knowledge that is utilized to generate SQL operations [10, 57]. For instance, a business analyst in the banking business requires knowledge of financial indicators such as return on investment and net income in order to generate effective SQL queries. (3) Synonym Knowledge: this category includes words or expressions that have the same or similar meanings regardless of how they are phrased differently [11]. (4) Value Illustration: this category refers to detailed descriptions of database values, including value types, value categories, and the mapping combinations of columns and values that correspond to entities, for example: "center" can be represented by "pos = C" in the database professional_basketball.
3.4 SQL Annotation
Double-Blind Annotation. As shown in Figure 2 (b), we employ a double-blind approach [42] for SQL annotation. This approach involves two independent SQL annotators who generate SQLs for the same question without discussion. The annotated SQLs are executed in databases, and those yielding identical results are gathered. Otherwise, the SQLs are checked with experts until a consensus is reached. Double-blind procedures can dramatically reduce the SQL annotation error rate, as there is a small probability for two skillful annotators to generate the same incorrect results when databases have large values. The more semantic-equivalent and efficient SQL selected by experts for each question is picked as ground truth SQL in BIRD, and the external knowledge evidence sentences are recorded for each SQL if utilized.
Examination. Experts evaluate each text-to-SQL pair to ensure the highest quality of data. The evaluation process includes two dimensions: SQL validity, and text-knowledge-SQL alignment. Firstly, the SQL validity will be confirmed that each SQL is executable and can return a valid result from the database. The "valid result" refers to the set of results that is not "NULL". If the executed result set is "NULL", experts will make slight changes to the conditions of the questions until the associated SQLs can provide a valid result set. Secondly, text-knowledge-SQL alignment is involved to ensure that each SQL can be generated with the given texts and knowledge evidence. If the evidence is insufficient to generate the SQL or contains errors, experts will be in charge of correcting them.
4 Data Statistics
Overall Statistics Table 1 presents an overview comparison between BIRD and other cross-domain text-to-SQL benchmarks. As the statistics demonstrate, BIRD is a large-scale cross-domain benchmark, covering complex SQL functions, knowledge reasoning, and efficiency evaluation.
Question Statistics Database values bring more challenges in text-to-SQLs. In order to underscore this, we classify questions into two macro-categories: Fundamental Type and Reasoning Type, and each contains 4-5 micro-categories in detail. The Fundamental Type of questions refers to those that can be answered without database value comprehension. It contains Match-based (83.9%), Ranking (20.3%), Comparison (16.7%), Counting (30.4%), Aggregation (15.7%). The
Table 1: An overview comparison between BIRD and other cross-domain text-to-SQL benchmarks. In SQL, Function pertains to the SQL functions (Appendix B.11). Knowledge refers to whether or not this dataset necessitates external knowledge reasoning from the model. Efficiency refers to whether or not this dataset takes into consideration execution efficiency.
Dataset	# Example	# DB	# Table/DB	# Row/DB	Function	Knowledge	Efficiency
WikiSQL [58]	80,654	26,521	1	17	X	X	X
SPIDER [53]	10,181	200	5.1	2K	X	X	X
KaggleDBQA [24]	272	8	2.3	280K	X	✓	X
BIRD	12,751	95	7.3	549K	✓	✓	✓
a) Database domain distribution w/ size
b) Database value type distribution
Figure 3: This is a comprehensive database distribution in the BIRD. a) shows the domain and size distribution of each database. And b) shows the data type distribution of databases.
Reasoning Type entails questions that demand the external knowledge grounding on values, which is exclusive to BIRD. To be specific, the questions about Domain Knowledge (23.6%), Numeric Computing (24.5%), Synonym (7.2%), Value Illustration (70.1%), are involved in BIRD. There are ample examples in Appendix B.3. In addition, we observe that 70.1% of the questions need value illustrations. This indicates that more real-world questions in text-to-SQL applications demand a thorough understanding of database values, which is consistent with our motivation for creating the BIRD benchmark.
Database Statistics In BIRD, we investigate the distribution of database domains, database size, and value types. Figure 3 (a) presents a detailed distribution of domains and their counterpart databases in a sunburst diagram for both training and development sets. The area of each semi-circle corresponds to the number of text-to-SQL pairs in this database. Figure 3 (a) also shows the size distributions of databases. The darker color means a larger size of databases, and vice versa. For example, the database Donor is the largest database with 4.5 GB in this dataset. Furthermore, we observe from Figure 3 (b) that a considerable proportion of BIRD's data comprises date-related values. Considering that real-world applications often rely on time-sensitive data [25], the prevalence of such questions highlights the practical purposes.
SQL Statistics We provide the complexity and diversity of SQLs in BIRD. As illustrated in Figure 4, we present a comprehensive distribution analysis of SQLs across four dimensions. No. Tokens / SQL and No. JOINS / SQL demonstrate the intricacy of the SQLs in BIRD. No. of Keywords and No. n-grams / SQL (n=3) serve as the support for the diverse patterns of SQLs since we decouple the question and SQL annotation procedures to make the situation more realistic [6].
5 Evaluation Metrics
In contexts of practical data analysis, text-to-SQL models are prioritized for delivering expected results accurately and efficiently. Thus we provide two metrics in BIRD, execution accuracy (EX)
Figure 4: A comparative statistical analysis of SQL queries in the BIRD dataset and other cross-domain text-to-SQL benchmarks.
and valid efficiency score (VES) to evaluate text-to-SQL parsers confronted with large real-world database values.
Execution Accuracy (EX) EX is defined as the proportion of examples in the evaluation set for which the executed results of both the predicted and ground-truth SQLs are identical, relative to the overall number of SQLs [37]. Considering the result set as  $V_{n}$  executed by the  $n^{th}$  ground-truth SQL  $Y_{n}$ , and the result set  $\hat{V}_n$  executed by the predicted SQL  $\hat{Y}_n$ , EX can be computed by:
$$
\mathrm {E X} = \frac {\sum_ {n = 1} ^ {N} \mathbb {1} \left(V _ {n} , \hat {V} _ {n}\right)}{N}, \tag {2}
$$
where  $\mathbb{1}(\cdot)$  is an indicator function, which can be represented as:
$$
\mathbb {1} (V, \hat {V}) = \left\{ \begin{array}{l l} 1, & V = \hat {V} \\ 0, & V \neq \hat {V} \end{array} \right. \tag {3}
$$
Valid Efficiency Score (VES) VES is designed to measure the efficiency of valid SQLs generated by models. It is worth noting that the term "valid SQLs" refers to predicted SQL queries whose result sets align with those of the ground-truth SQLs. Any SQL queries that fail to fetch the correct values will be declared invalid since they are totally useless if they cannot fulfill the user requests, regardless of their efficiency. In this case, the VES metric considers both the efficiency and accuracy of execution results, providing a comprehensive evaluation of a model's performance. Formally, the VES can be expressed as:
$$
\mathrm {V E S} = \frac {\sum_ {n = 1} ^ {N} \mathbb {1} \left(V _ {n} , \hat {V} _ {n}\right) \cdot \mathbf {R} \left(Y _ {n} , \hat {Y} _ {n}\right)}{N}, \quad \mathbf {R} \left(Y _ {n}, \hat {Y} _ {n}\right) = \sqrt {\frac {\mathbf {E} \left(Y _ {n}\right)}{\mathbf {E} \left(\hat {Y} _ {n}\right)}} \tag {4}
$$
where  $\mathbf{R}(\cdot)$  denotes the relative execution efficiency of predicted SQL in comparison to ground-truth SQL, allowing for machine status-related uncertainty.  $\mathbf{E}(\cdot)$  is a function to measure the absolute execution efficiency for each SQL in a given environment. Furthermore, we incorporate the square root function to minimize random instances that are abnormally faster or slower than the ground-truth SQLs. Here, efficiency can refer to running time, throughput, memory cost, or merged metrics. In BIRD, we consider the running time mainly at this time. Appendix B.8 provides a detailed description of the VES.
6.1 Baseline Models
We present the performance of two types of baseline models in BIRD. The first type of model is based on fine-tuning (FT) techniques, which outputs SQL by tuning all parameters of language models to learn the annotated train set. On the other hand, the second type of model based on
$^{\ddagger}$ In BIRD evaluation, we run 100 times for each SQL in the same CPU and evaluate average results after dropping the outliers.
Figure 5: A bar chart provides a clear visualization of the performance of advanced models on BIRD.
Table 2: The Execution Accuracy (EX) of advanced text-to-SQL models in BIRD. The human performance is also provided.
Models	Development Data	Testing Data
w/o knowledge	w/ knowledge	w/o knowledge	w/ knowledge
FT-based
T5-Base	6.32	11.54 (+5.22)	7.06	12.89 (+5.83)
T5-Large	9.71	19.75 (+10.04)	10.38	20.94 (+10.56)
T5-3B	10.37	23.34 (+12.97)	11.17	24.05 (+12.88)
ICL-based
Palm-2	18.77	27.38 (+8.61)	24.71	33.04 (+8.33)
Codex	25.42	34.35 (+8.93)	24.86	36.47 (+11.61)
ChatGPT	24.05	37.22 (+13.17)	26.77	39.30 (+12.53)
ChatGPT + COT	25.88	36.64 (+10.76)	28.95	40.08 (+11.24)
Claude-2	28.29	42.70 (+14.41)	34.60	49.02 (+14.42)
GPT-4	30.90	46.35 (+15.45)	34.88	54.89 (+20.01)
GPT-4 + DIN-SQL	-	50.72	-	55.90
Human Performance	-	-	72.37	92.96 (+20.59)
in-context learning (ICL), can generate results without additional training. In FT models, we select T5 family [38] as the main baseline models. For ICL-based models, we provide zero-shot results of Codex (code-davinci-002), ChatGPT (gpt-3.5-turbo), GPT-4 (gpt-4-32k), Claude-2 (claude-2.0), Palm-2 (text-bison-001). Additionally, we also implement a state-of-the-art (SOTA) model of SPIDER, DIN-SQL [35], to evaluate the challenges proposed by the BIRD dataset. Table 2, Table 3 and Figure 5 present the overall results of advanced language models on BIRD.
6.2 Execution Accuracy Analysis
Table 2 and Figure 5 presents stratified performances of various models in BIRD. GPT-4 surpasses all baseline language models. Claude-2 closely follows, demonstrating outstanding abilities in semantic parsing and knowledge reasoning. Further, the incorporation of a dedicated reasoning prompt by [35], enables DIN-SQL + GPT-4 to achieve a new state-of-the-art result on BIRD. It contains value sampling, few-shot demonstrations, and self-correction. Despite considerable advancements in Language Model Learning (LLMs) and prompt intelligence, the performance of these models lags obviously behind human capabilities. Not only does this gap highlight the complex nature of BIRD, but it also presents opportunities for uncovering more capable models or advanced reasoning prompt methods applicable to real-world text-to-SQL scenarios.
Table 3: The Valid Efficiency Score (VES) of advanced text-to-SQL models in BIRD. The human performance is also presented.
Models	Development Data	Testing Data
w/o knowledge	w/ knowledge	w/o knowledge	w/ knowledge
FT-based
T5-Base	7.78	12.90 (+5.12)	8.97	14.71 (+5.74)
T5-Large	9.90	22.74 (+12.84)	12.25	25.00 (+12.75)
T5-3B	13.62	25.57 (+11.95)	15.17	27.80 (+12.63)
ICL-based
Palm-2	20.82	28.64 (+7.82)	31.32	38.41 (+7.09)
Codex	33.37	43.41 (+10.04)	35.40	41.60 (+6.20)
ChatGPT	27.97	43.81 (+15.84)	36.68	51.40 (+14.72)
ChatGPT + COT	32.33	42.30 (+9.97)	49.69	56.56 (+6.87)
Claude-2	32.75	45.28 (+12.53)	39.32	55.77 (+16.45)
GPT-4	34.60	49.77 (+15.17)	40.20	60.77 (+20.57)
GPT-4 + DIN-SQL	-	58.79	-	59.44
Human Performance	-	-	70.36	90.27 (+19.91)
6.3 Baseline Performance on Spider
Figure 6: The EX results of the same baseline models on the SPIDER and BIRD dev set.
SPIDER [53] is the most prevalent and complex cross-domain text-to-SQL benchmark. It mainly focuses on evaluating schema-relevant semantic parsing capabilities. To demonstrate the increasing difficulty of the BIRD dataset due to its complex database schema and values, we visualize the execution accuracy of the same baseline models on both the BIRD and SPIDER datasets. To ensure a fair evaluation, all models are furnished with knowledge about values, and the same programming prompt is implemented for Language Models (LMs) across the two datasets. Figure 6 shows the concentration on database values makes BIRD become the most challenging text-to-SQL benchmark. This disparity in the performance of each model demonstrates the need for further research and development of models capable of handling complicated database schema and values.
6.4 Efficiency Analysis
According to Table 3, we can observe that models with higher EX can more possibly achieve higher VES. This can be explained by the prerequisite that text-to-SQL models must accurately predict results in order to attain a higher VES, which fulfills the practical purpose.