RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
The text-to-SQL task is an important NLP task, which maps input questions to meaningful and executable SQL queries,
enabling users to interact with databases in a more intuitive and user-friendly manner.
State-of-the-art methods Pourreza and Rafiei ( 2024 ); Li et al. ( 2023a , b ); Scholak et al. ( 2021 ) relying on large language models have achieved execution accuracy above 75% on the Spider dataset Yu et al. ( 2018 ) , which encompasses complex SQL grammar and cross-domain settings. Recently, Pourreza and Rafiei ( 2024 ) achieved remarkable results with an impressive 85.3% execution accuracy on the Spider dataset, leveraging the enhanced capabilities of GPT-4 .

However, previous text-to-SQL datasets Yu et al. ( 2018 ); Finegan-Dollak et al. ( 2018 ); Yaghmazadeh et al. ( 2017 ); Iyer et al. ( 2017 ); Zhong et al. ( 2017 ); Li and Jagadish ( 2014 ); Giordani and Moschitti ( 2012 ); Popescu et al. ( 2003 ); Tang and Mooney ( 2000 ); Dahl et al. ( 1994 ) , have limitations that prevent them from capturing complex reasoning effectively.
For example, Spider Yu et al. ( 2018 ) purposely excludes questions that would require external knowledge, like common-sense inferences or mathematical calculations. This exclusion limits Spider’s ability to properly test how well models can handle real-world scenarios, which often require a deeper level of reasoning capabilities.

In this paper, we present Archer,
an innovative dataset designed to incorporate three distinct types of reasoning: arithmetic, commonsense, and hypothetical reasoning. By including such varied reasoning skills, Archer seeks to challenge and expand the capabilities of text-to-SQL models, equipping them to manage more intricate and nuanced queries. Figure 1 showcases data examples from Archer that demonstrate these three reasoning abilities.

To evaluate the challenge posed by Archer, we conducted experiments with both large language models (LLMs) and fine-tuned models. However, all models demonstrated inferior performance when dealing with Archer. Even the model that achieved a high place on the Spider leaderboard managed only 6.73% execution accuracy on Archer test sets. These findings highlight substantial potential for improvement, indicating that Archer indeed provides a significant challenge to current models.
Reasoning Types
In this section, we present the three different types of reasoning in Archer: arithmetic, commonsense, and hypothetical reasoning.
Arithmetic reasoning
Arithmetic reasoning pertains to the act of resolving mathematical problems through logical and analytical thought processes. According to an analysis of SQL queries from practical applications like the Baidu search engine and customer service and data analysis robots by Wang et al. ( 2020 ) , mathematical calculations account for a significant portion across SQL applications. However, previous high-quality datasets contain very few questions that involve calculations, and such questions are typically auto-generated with simple grammar. In contrast, all question-SQL pairs included in Archer necessitate arithmetic reasoning and are manually annotated to ensure high quality.
Commonsense reasoning
Commonsense reasoning refers to the capacity to make logical deductions based on implicit knowledge and a broad understanding of how things function in the world. Archer includes questions that necessitate models to comprehend the database, infer missing details, and generate logical inferences to create accurate SQL queries.
As illustrated in Figure 1 , for the question "Which 4-cylinder car needs the most fuel to drive 300 miles? List how many gallons it needs, and its make and model." , the database does not provide an explicit schema about the fuel used to travel 300 miles for each car. It only provides each car’s fuel consumption in MPG. Solving this question requires commonsense knowledge, specifically the understanding of "Fuel used is calculated by dividing distance driven by fuel consumption" to derive the correct SQL.
Hypothetical reasoning
Hypothetical reasoning takes the complexity a step further, requiring models to have counterfactual thinking ability, which is the ability to imagine and reason over unseen cases based on the seen facts and counterfactual assumptions. Archer includes questions that involve hypothetical situations, requiring the model to understand and reason about conditional relationships. As illustrated in Figure 1 , consider the hypothetical question "If all cars produced by the Daimler Benz company have 4-cylinders, then in all 4-cylinder cars, which one needs the most fuel to drive 300 miles? Please list how many gallons it needs, along with its make and model." .
In this question, the underlying assumption contradicts the factual information stored in the database. The model must comprehend this assumption and convert it into the SQL condition d.fullname = "Daimler Benz" or a.cylinders = "4" .
Corpus Construction
As illustrated in Figure 2 , we create Archer in the following six steps, spending around 300 hours of human labor in total: § 3.1 Database Collection, § 3.2 Question Annotation, § 3.3 SQL Annotation, § 3.4 SQL review, § 3.5 Question Review and Paraphrase, § 3.6 Final Review and Process.
Database Collection
In a noteworthy research study conducted by Yu et al. ( 2018 ) , a total of 200 high-quality databases across various domains were meticulously collected and created, requiring approximately 150 man-hours. Out of these, 166 databases were made publicly available.

Since not all Spider databases support the proposed reasoning types,
we carefully selected 20 databases across 20 domains from the Spider 166 databases based
two criteria. Firstly, we applied a
script to keep only databases with a minimum of 3 tables and 20 columns within each database, as well as a minimum of 6 columns with time or numeric data types.
Secondly, we manually checked the filtered databases. These two steps ensure that each selected database contains sufficient information to support complex reasoning.
Question Annotation
Two bilingual (English and Chinese) Ph.D. students with SQL experience were assigned the task of generating questions based on 20 databases. The annotators were required to propose 25-30 questions for each database, ensuring that the questions met the following four requirements:
) Arithmetic Reasoning:
Each question should incorporate arithmetic reasoning. The annotators were expected to include a minimum of five questions for each arithmetic reasoning type (addition, subtraction, multiplication, division).
) Hypothetical Reasoning:
At least five questions should involve hypothetical reasoning. For each question using hypothetical reasoning, the annotators were also required to propose a corresponding factual question.
) Commonsense Reasoning:
The annotators were encouraged to propose questions that involve commonsense reasoning. However, the number of questions with commonsense reasoning was not strictly limited. This flexibility acknowledged that not all databases support commonsense reasoning, and not all arithmetic calculations necessitate it.
) Complex SQL Grammar:
The annotators were encouraged to propose questions that require the utilization of complex SQL grammar, such as GROUP BY, ORDER BY, and JOIN.
The annotators were asked to write each question in both English and Chinese. Besides, they were instructed to indicate the reasoning types involved (arithmetic: addition, subtraction, multiplication, division; hypothetical; commonsense), and provide the relevant knowledge or formulation if the question incorporated commonsense reasoning.
SQL Annotation
In order to mitigate cognitive bias, we employed a diverse set of annotators for the tasks of generating questions and writing SQL queries. Two Ph.D. students, who possess strong SQL skills, were specifically chosen to translate the natural language questions into SQL queries. Their responsibilities encompassed the following:
) Clarity Ensuring:
The annotators reviewed both English and Chinese questions to identify any ambiguity and restructure them accordingly.
) SQL Writing:
The annotators were instructed to use consistent SQL patterns when multiple equivalent queries are applicable for similar questions.
) Verification and Correction:
The annotators were also responsible for reviewing the annotations pertaining to reasoning types and the common knowledge necessary to solve each question.
SQL Review
To ensure the correctness of the annotated SQL for each question, we employed a professional SQL expert to review all the SQL queries and rectify any incorrect ones. Subsequently, the original SQL annotators were responsible for verifying the SQL queries corrected by the expert. In cases where there are differences of opinion between the expert and the annotators regarding the corrected queries, they were required to engage in a discussion and reach a consensus to finalize the SQL annotation.
Question Review and Paraphrase
We employed two native English speakers and two native Chinese speakers to review and paraphrase English and Chinese questions, respectively. Initially, their task was to assess the naturalness and grammatical accuracy of the questions. Subsequently, the annotators were requested to provide a paraphrased version of each question in order to enhance the dataset’s robustness.
Final Review and Processing
In the final stage of our process, we assigned the task of reviewing the English and Chinese questions, SQL, and annotations relating to reasoning types and commonsense knowledge to our most seasoned annotator. Once this comprehensive review was completed, we ran a script to ensure that all SQL queries are executable.
Dataset Statistics and Comparison
In Table 1 , we present a summary of the statistics for Archer as well as other publicly available text-to-SQL datasets. We conducted a comparative analysis of Archer and other datasets based on four key perspectives: scale, complexity, reasoning distribution, and language.
Scale
Archer consists of 1,042 Chinese questions, 1,042 English questions, and 521 corresponding SQL queries, covering a wide range of 20 distinct databases spanning 20 domains. Each database in Archer, on average, consists of 7.55 tables and 45.25 columns.
Archer stands out for its inclusion of multiple domains and a higher average number of tables and columns.

It is worth noting that WikiSQL Zhong et al. ( 2017 ) and DuSQL Wang et al. ( 2020 ) are exceptionally large databases generated automatically. Inspired by them, Archer has the potential to serve as a valuable resource for summarizing SQL templates and training SQL-to-text generators to create large-scale datasets in line with our reasoning setting. In this project, we do not utilize Archer for automatic question-SQL pairs generation. This possibility is a potential future direction.
Complexity
Archer distinguishes itself by its considerably higher level of complexity compared to existing text-to-SQL datasets. Several factors contribute to this complexity:

Firstly, the average question length in Archer is significantly longer than that in other datasets. This poses a challenge to models because longer inputs increase the likelihood of misunderstandings or misinterpretations of specific question details.

Secondly, the average SQL length in Archer stands at 79.71, which is significantly longer than that of other datasets except for ATIS, which contains only one table. Longer SQL statements increase the likelihood of generating incorrect code.

Thirdly, value prediction, which is crucial in SQL generation, is often undervalued in current research. Interestingly, Pourreza and Rafiei ( 2024 ) achieved an execution accuracy of 85.3% on the Spider dataset without utilizing database content. This is primarily because Spider SQL queries typically contain an average of only 0.93 value slots, with most values explicitly quoted in the question. In contrast, Archer emphasizes the importance of values, with an average of 6.21 value slots per SQL. Furthermore, Archer questions do not explicitly quote exact values; instead, they naturally mention value information, mirroring real-world scenarios.

Fourthly, SQL queries in Archer refer to an average of 2.17 tables, suggesting that a substantial number of the questions require the use of information from multiple tables to derive SQLs.

Fifthly, the level of SQL statement nesting in Archer is higher than that in other datasets, indicating a greater degree of reasoning complexity required to answer Archer questions, which often necessitates the use of multiple subqueries.

Finally, Archer exhibits a high usage rate of complex SQL grammar features such as GROUP BY and ORDER BY in each SQL, surpassing the frequency of usage seen in nearly all other datasets.
Reasoning Distribution
All questions in Archer require arithmetic reasoning. This means that mathematical calculations and operations are essential in understanding and answering these questions effectively.
Additionally, 44.0% of the questions involve hypothetical reasoning, where the model needs to reason about hypothetical scenarios to derive the correct SQL. Furthermore, 51.4% of the questions require commonsense reasoning, where the model needs to utilize general knowledge and commonsense understanding to produce the correct SQL.

It is worth noting that the majority of previous text-to-SQL datasets do not incorporate arithmetic and commonsense reasoning. Moreover, none of the previous datasets contain questions that involve hypothetical reasoning. Therefore, the inclusion of these types of reasoning tasks in Archer sets it apart from previous datasets and presents new challenges for models in the field of text-to-SQL understanding and generation.
Language
Unlike most previous text-to-SQL datasets that focus solely on English, Archer provides both English and Chinese questions. This bilingual feature of Archer enhances the evaluation and training capabilities of text-to-SQL models, catering to the needs of users in both English and Chinese languages, while forming a solid base for potential support of more languages for Archer, which is left as a future work.
Baseline Models
We benchmark the performance of two types of presentative text-to-SQL models on Archer: LLMs and finetuned Models.
LLMs
LLMs have shown strong performance on commonly used text-to-SQL benchmarks, such as Spider.
To analyze the difficulty of the whole Archer, we provide the zero-shot results of GPT-3.5 ( gpt-3.5-turbo ) with different prompt settings: API Doc , CT-3 , CT-3+COT . API Doc follows the style of the Text-to-SQL example provided by OpenAI, which includes the schema information in a comment style. CT-3 , introduced by Rajkumar et al. ( 2022 ) , includes the CREATE TABLE commands for each table and the results of executing a SELECT * FROM T LIMIT 3 query on each table. Compared with API Doc, CT-3 provides more information like declarations of column types and foreign keys, and a small amount (3) of content examples. CT-3+COT implement the Chain-Of-Thought (COT) technique on top of the CT-3 prompt by appending the prompt sentence "Let’s think step by step." before the SQL generation. Following the work of Li et al. ( 2023c ) , we provide a 1-shot pseudo example for LLMs to learn the procedure of thinking and output format.
Furthermore, we evaluate the performance of GPT-4+DIN-SQL Pourreza and Rafiei ( 2024 ) on Archer. As a highly-ranked solution on the Spider leaderboard at the time of writing, it consists of four modules: (1) schema linking, (2) query classification and decomposition, (3) SQL generation, and (4) self-correction. The initial three modules exploit the in-context learning ability of GPT-4 with ten shots, while the self-correction is conducted by GPT-4 in a zero-shot setting. Note that we do not evaluate GPT-4+DIN-SQL on Archer Chinese questions because it is designed for English datasets.
More details on the prompts can be found in Appendix A .
Fine-tuned Models
T5-based fine-tuned models have shown promising results on the Spider leaderboard. It is, however, worth mentioning that many top-tier models on the leaderboard are customized specifically for the limited SQL grammars present in the Spider dataset. Given that our dataset contains more complex grammatical structures compared to Spider, these specialized models may not be suitable for our needs. As a result, we select vanilla T5 models as our baselines instead of the aforementioned variants. We evaluate English questions using T5-base , T5-large , T5-3B , and evaluate Chinese questions using mT5-base , mT5-large , mT5-xl .
We concatenate the natural question Q and database schema into a sequence as input in a format as below:

x=[q_{1},...,q_{|Q|}|t_{1}\mathbin{:}c_{1}^{t_{1}},...,c_{|t_{1}|}^{t_{1}}|...|t_{|\mathcal{T}|}\mathbin{:}c_{1}^{t_{|\mathcal{T}|}},...,c_{|t_{|\mathcal{T}|}|}^{t_{|\mathcal{T}|}}] (1)

where q_{i} is the i^{th} question token, t_{j} is the j^{th} table, and c_{k}^{t_{j}} is the k^{th} in the j^{th} table.
Following the works of Li et al. ( 2023a ); Lin et al. ( 2020 ) , we extract the potential database cell values and append them to their corresponding columns.
Evaluation Metrics
We employ two evaluation metrics: VAlid SQL (VA) and EXecution accuracy (EX).
VA is the proportion of the predicted SQL statements that can be executed successfully, no matter with correct or incorrect answers.
EX is the proportion of the predicted SQL statements where the execution results match those of the gold SQL statements. We computed EX of each instance use a new evaluation script as shown in Algorithm 1 , which mitigates the false-negative issue in present publicly available evaluation scripts caused by permutations of columns and rows.
Data Split
Among the 20 databases, we split 16, 2, and 2 databases as training, dev, and test sets, respectively.
The databases for Archer training set are collected from the Spider training set, and the databases for Archer dev set and test set are collected from the Spider dev set. We strive to introduce as few new SQL keywords as possible during SQL annotation to facilitate the integration of our dataset with the Spider and CSpider datasets.
We also report the performance of T5 finetuned on the augmented training set which consists of Archer training set and Spider/CSpider training set.
For LLM baselines, we assess the zero-shot performance of GPT-3.5 on the full Archer to evaluate the dataset’s overall difficulty. As for GPT-4+DINSQL, due to its high cost and extended response time, we only test it on Archer test sets.
Hyper-Parameters
For GPT-3.5 baselines, we set stop sequence to [‘- -’, ‘;’, ‘#’] and the temperature to 0. In the case of GPT-4+DIN-SQL, we adhere to the default setting as outlined in Pourreza and Rafiei ( 2024 ) . For T5 baselines, we employ the Adafactor optimizer with a learning rate of 5e-5. For T5-base/mT5-base and T5-large/mT5-large, we adopt a batch size of 6 and a gradient descent step of 5. For T5-3b and mT5-xl, we use a batch size of 2 and a gradient descent step of 16.
To adjust the learning rate, we utilize linear warm-up with a warm-up rate of 0.1, followed by cosine decay.
During inference, we set the beam size to 8. We set the maximum epoch to 128, having checkpoints every 10 epochs as well as the last epoch. We then select the optimal checkpoints based on their EX performance on the development set.
Overall Evaluation
We summarize the performance of LLMs and finetuned models in Table 2 . The low performance of these models on Archer suggests that Archer presents a significant challenge. This underscores the considerable potential for future improvement in this domain.
LLM
GPT-4+DIN-SQL obtain EX score of 6.73% on Archer test set, while it is able to achieve 85.3% test-suite execution performance on Spider test set Pourreza and Rafiei ( 2024 ) .
To evaluate the overall difficulty of Archer, we test the zero-shot performance of GPT-3.5 with API Doc, CT-3, CT-3+COT prompts on the full Archer data. Among the three kinds of prompts, CT-3 achieves the highest EX scores on both English data (EX: 13.34%) and Chinese data (EX: 12.86%). As expected, CT-3 performed slightly better than API Doc, likely due to its inclusion of more useful information, such as declarations of column types and foreign keys. However, the addition of COT in CT-3+COT did not outperform CT-3 on the complete Archer. On the other hand, for the Test set only, CT-3+COT slightly outperform CT-3.
From Table 2 , we observe a significant decrease in VA when using COT, suggesting that COT suffers from having more syntax errors in the generated SQL. Although CT-3+COT achieved a higher EX score than CT-3 and API Doc specifically for questions involving arithmetic and commonsense reasoning, it performed less effectively on questions that require hypothetical reasoning (cf. Table 3 in Appendix C ).
Finetuned Models
From Table 2 . we observe that T5 from scale base to 3B (XL) trained on Archer training set achieve 0.00% EX scores. This outcome could be attributed to the small-scale nature of Archer combined with its high complexity.
However, when Archer training set was augmented with the Spider/CSpider training set, the VA scores of T5 models exhibited a substantial improvement. Specifically, the T5-3B model trained on the augmented training set achieved an EX score of 4.81% on the English test (matching the performance of GPT-3.5+CT-3+COT) set and 1.92% on the Chinese test set (matching the performance of GPT-3.5+CT-3).

These results suggest that Archer has the potential to advance the development of text-to-SQL systems with complex reasoning.
Different Reasoning Analysis
To gain a comprehensive understanding of the difficulty levels within the complete Archer across various reasoning types, we conducted a thorough analysis using the GPT-3.5 model with the CT-3 prompt, which demonstrated the highest performance on the full dataset. Additional results for GPT-3.5 with alternative prompts can be found in Appendix C .
Overall Comparison
Figure 3 -(1) shows the performance on questions with different kinds of reasoning. The results reveal that questions solely based on arithmetic reasoning exhibit significantly higher performance compared to those involving additional forms of reasoning. Specifically, hypothetical reasoning presents a greater challenge than commonsense reasoning. Moreover, questions that require the integration of all three reasoning types exhibit the poorest performance.
Arithmetic Reasoning
The performance on questions that exclusively require arithmetic reasoning across various arithmetic operations is presented in Figure 3 -(2). The findings indicate that subtraction and division pose greater difficulty compared to addition and multiplication.
Commonsense Reasoning
On commonsense reasoning, in Figure 3 -(3), we compare the performance of GPT-3.5+CT-3 on such questions under two settings. The first setting involves directly inputting the question itself, while the second setting involves inputting the concatenation of the knowledge and the question. The results reveal that explicitly stating the knowledge within the question can aid in generating correct SQL queries.
This
suggests that leveraging external knowledge bases could be beneficial in solving similar questions. However, incorporating external knowledge into text-to-SQL tasks presents significant challenges in general. Firstly, models need to compare information from natural language questions with the relational database to determine if external knowledge is required. Secondly, models need to extract the most relevant knowledge from external knowledge bases. Last but not least, the process of integrating this knowledge into the text-to-SQL generation process remains largely unexplored.
Hypothetical reasoning
On hypothetical reasoning, in Figure 3 -(4), we compare the performance of these questions and observe a significant performance gap. The EX performance on factual questions exceeds 17%, whereas the performance on hypothetical questions falls below 7%, confirming the difficulties involved.