RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Complexity Factors Analysis
To gain insights into the SQL complexity within Archer, Figure 4 illustrates the relationship between the EX score and various factors, including question length, SQL length, number of value slots, number of tables mentioned in SQL, and SQL nested level. The performance demonstrates a decreasing trend as the question becomes longer, the SQL length increases, the number of value slots rises, the number of tables mentioned in the SQL grows, or the SQL nested level escalates.
As shown in Table 1 , Archer exhibits considerably higher complexity across these factors when compared to other publicly available text-to-SQL datasets.
Bad Case Analysis
We randomly selected 50 executable but incorrect examples generated by GPT-4 + DIN-SQL and identified the following common error types:
Incorrect Logic
: GPT-4 sometimes struggles with hypothetical questions that involve complex logic. For instance, when asked "If all cars produced by Daimler Benz company are 4-cylinders, which 4-cylinder car needs the most fuel to drive 300 miles?", the model might generate SQL queries like WHERE T1.Cylinders = 4 AND T4.Maker = ’Daimler Benz’ . However, the correct query should be WHERE T1.Cylinders = 4 OR T4.Maker = ‘Daimler Benz’ as there could be other 4-cylinder cars aside from Mercedes-Benz. This reveals a limitation in comprehending the hypothetical nature of the question.
Incorrect Knowledge
: GPT-4 may make commonsense errors when generating the SQL, such like unit conversions. For example, if a question requests fuel consumption in liters per hundred kilometers, but the database only contains fuel efficiency data in miles per gallon, the accurate conversion formula is liters_per_hundred_kilometers = 235.2145 / MPG . However, GPT-4 employs an incorrect formula like (100 * 3.78541) / MPG .
Incorrect Schema Understanding
: GPT-4 sometimes struggles to correctly link query entities to the corresponding database columns. For example, when asked about the "average single cylinder displacement of an 8-cylinder car", GPT-4 might generate a query like SELECT avg(Edispl) FROM cars_data WHERE Cylinders = 8 . However, in this case, the query should calculate the average single cylinder displacement, like SELECT AVG(1.0 * Edispl / Cylinders) AS avg_displ FROM cars_data WHERE Cylinders = 8 . This error highlights the need for the model to understand database column names, especially when they involve abbreviations commonly used in real-world databases. (Note that in the Spider dataset, annotators tend to use exact column names in their queries, e.g., What is the average edispl for all Volvos?)
Other Detail Errors
: For example, GPT-4 may also exhibit minor errors such as forgetting to multiply 1.0 for float calculations.
Related Work
The earliest text-to-SQL datasets, including ATIS Dahl et al. ( 1994 ); Iyer et al. ( 2017 ) , GeoQuery Zelle and Mooney ( 1996 ); Iyer et al. ( 2017 ) , Scholar Iyer et al. ( 2017 ) , Academic Li and Jagadish ( 2014 ) , IMDB Yaghmazadeh et al. ( 2017 ) , Yelp Yaghmazadeh et al. ( 2017 ) , Advising Finegan-Dollak et al. ( 2018 ) and Restaurants Giordani and Moschitti ( 2012 ); Tang and Mooney ( 2000 ); Popescu et al. ( 2003 ) , were limited to a single database.
Consequently, models trained on these datasets struggled to generalize to unseen databases as they were tested on the same database used for training.
To address such limitations, Zhong et al. ( 2017 ) introduced WikiSQL in which the databases in the test set were not present in the training set. However, the SQL queries in WikiSQL were generated automatically using simplified assumptions, which may not fully capture the complexity of real-world queries.

For a comprehensive cross-domain text-to-SQL dataset, Yu et al. ( 2018 ) presented Spider dataset, which is currently the most widely used text-to-SQL datasets.
However, Spider excludes questions that require external knowledge, like commonsense reasoning and mathematical calculations, which are often essential for real-world applications.

Then, Wang et al. ( 2020 ) proposed DuSQL, a Chinese cross-domain text-to-SQL dataset that includes math-related questions. However, DuSQL’s queries and questions are relatively simple due to automatic generation and grammar restrictions. Dou et al. ( 2022 ) extended DuSQL with external knowledge in their KnowSQL dataset.
Unfortunately, KnowSQL is not publicly unavailable.

In real-life scenarios, databases can be dirtier with abbreviated and obscure naming of tables, columns, and data values. To address this, Lee et al. ( 2021 ) proposed KaggleDBQA with realistic databases. Li et al. ( 2023c ) proposed BIRD benchmark for the text-to-SQL task on big and dirty databases with a total size of 33.4 GB.

In contrast to these existing text-to-SQL datasets, Archer focuses specifically on questions involving complex reasoning and offers both English and Chinese questions to query English databases across various domains. Notably, all questions and SQL queries in Archer are manually annotated by humans and thoroughly reviewed by professionals, ensuring high-quality annotations for training and evaluation purposes.
Conclusion
In this paper, we present Archer, a complex bilingual text-to-SQL dataset with three distinct reasoning types: arithmetic, commonsense, and hypothetical reasoning. Experimental results on Archer, obtained from both LLMs and fine-tuned models, suggest plenty of space for improvement.
Acknowledgement
This work is supported by Huawei’s Dean’s Funding (C-00006589) and the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1).
Limitations
The evaluation metric used in Archer is execution accuracy. This metric may be perceived as an upper-bound performance measure, as SQL queries producing the same execution results on a single database may still possess different semantic meanings. To overcome this limitation, we plan to release a test suite in the future that evaluates SQL queries on multiple databases, allowing for a more comprehensive assessment of semantic accuracy.
Ethics Statement
As mentioned in the submission, we select our databases from Spider Yu et al. ( 2018 ) , which is public for academic use and does not contain sensitive information.
The construction of our dataset involved the active involvement of human participants. We recruited and provided training to five annotators who possessed backgrounds in databases. These annotators were assigned the tasks of generating questions based on the databases, writing SQL queries, and paraphrasing the questions. Importantly, no sensitive personal information was involved throughout this process.
Our human annotation study underwent evaluation by the departmental ethics panel, which deemed it exempt from ethical approval. This exemption was based on the fact that all participants were employees of the University of Edinburgh and were therefore protected by employment law. Furthermore, participants received compensation at the standard hourly rate designated for tutors and demonstrators at the university.
To promote academic usage, we intend to freely release the dataset online.
Figure 1: Archer examples with three reasoning types: arithmetic, commonsense, and hypothetical reasoning. (See more examples in Appendix D ); Archer examples with three reasoning types: arithmetic, commonsense, and hypothetical reasoning. (See more examples in Appendix D )
Figure 2: The annotation process of our Archer.; The annotation process of our Archer.
Figure 3: GPT-3.5 + CT-3 execution accuracy comparison across and within different reasoning types. A refers to arithmetic. H refers to hypothetic. C refers to commonsense.; GPT-3.5 + CT-3 execution accuracy comparison across and within different reasoning types. A refers to arithmetic. H refers to hypothetic. C refers to commonsense.
Figure 4: GPT-3.5 + CT-3 execution accuracy performance w.r.t different complexity level. The abbreviations used are as follows: QL for the average question length (1: [0,15)], 2: [15,20), 3:[30,45), 4: [45,)), SQLL for the average SQL length (1: [0,50)], 2: [50,100), 3:[100,150), 4: [150,)), VS for the average number of value slots per question (1: [0,3)], 2: [3,6), 3:[6,9), 4: [9,)), TM for the average number of tables mentioned in each SQL (1: [0,2)], 2: [2,3), 3:[3,5), 4: [5,)), NL for the average nested level per SQL (1: [0,1)], 2: [1,2), 3:[2,3), 4: [3,)).; GPT-3.5 + CT-3 execution accuracy performance w.r.t different complexity level. The abbreviations used are as follows: QL for the average question length (1: [0,15)], 2: [15,20), 3:[30,45), 4: [45,)), SQLL for the average SQL length (1: [0,50)], 2: [50,100), 3:[100,150), 4: [150,)), VS for the average number of value slots per question (1: [0,3)], 2: [3,6), 3:[6,9), 4: [9,)), TM for the average number of tables mentioned in each SQL (1: [0,2)], 2: [2,3), 3:[3,5), 4: [5,)), NL for the average nested level per SQL (1: [0,1)], 2: [1,2), 3:[2,3), 4: [3,)).
Figure 5: The example of API Doc prompt, CT-3 prompt, and CT-3+COT prompt.; The example of API Doc prompt, CT-3 prompt, and CT-3+COT prompt.
Figure 6: The example of Archer data requiring Arithmetic Reasoning.; The example of Archer data requiring Arithmetic Reasoning.
Figure 7: The example of Archer data requiring Commonsense Reasoning.; The example of Archer data requiring Commonsense Reasoning.
Figure 8: The example of Archer data requiring Hypothetic Reasoning.; The example of Archer data requiring Hypothetic Reasoning.
Comparison of public text-to-SQL datasets. The abbreviations used are as follows: #Q for the number of unique questions, #SQL for the number of unique SQLs, #DB for the number of databases, #Dom for the number of domains, T/DB for the number of tables per database, C/DB for the number of columns per database, QL for the average question length, SQLL for the average SQL length, VS for the average number of value slots per question, TM for the average number of tables mentioned in each SQL, NL for the average nested level per SQL, GB and OB for the average number of GROUP BY and ORDER BY clauses per SQL respectively. A, H, C, and Lang represent arithmetic, hypothetical, commonsense, and language, respectively. The cross mark, - denote absence and presence respectively. The statistics for BIRD, CSpider, and Spider is based on training and dev sets as their test sets are unavailable. Language is represented as en for English databases and questions, zh for Chinese databases and questions, and zh for English databases and Chinese questions.
Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.
Performance with respect to different reasoning types.
Performance with respect to different arithmetic operations on data with arithmetic reasoning only.
Performance for questions needed commonsense reasoning with and without explicit knowledge input.
Performance comparison for hypothetical questions and corresponding factual questions.