Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

Authors: Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, Tao Yu
Primary area: datasets and benchmarks
Keywords: llm benchmark, data science and engineering, code generation, text-to-sql, llm agent

Abstract:
Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics.
We introduce Spider 2.0, an evaluation framework comprising $632$ real-world text-to-SQL workflow problems derived from enterprise-level database use cases. 
The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake.
We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-level codebases. 
This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding $100$ lines, which goes far beyond traditional text-to-SQL challenges.
Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 21.3\% of the tasks, compared with 91.2\% on Spider 1.0 and 73.0\% on BIRD.
Our results on Spider 2.0 show that while language models have demonstrated remarkable performance in code generation --- especially in prior text-to-SQL benchmarks --- they require significant improvement in order to achieve adequate performance for real-world enterprise usage.
Progress on Spider 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-world enterprise settings.
Our code, baseline models, and data are available at [spider2-sql.github.io](spider2-sql.github.io) .

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730156196643

thread_type:
official_review

<thread_object>
creator: reviewer_c1as

<content>
summary:
The paper presents Spider 2.0, a new benchmark for Text-to-SQL systems that introduces more complex and realistic SQL tasks compared to benchmarks like Spider 1.0 and BIRD. Spider 2.0 tasks require models to leverage project-level information, such as codebases and documentation, for generating accurate SQL queries. It also includes more SQL dialects such as DuckDB, Snowflake, and BigQuery, and utilizes advanced SQL functions such as ST_DISTANCE and CORR. 

The benchmark demonstrates its difficulty by showing that even state-of-the-art language models achieve a maximum accuracy of only 15.12%. This result highlights significant gaps in current Text-to-SQL approaches that perform well on enterprise-level SQL generation tasks. Spider 2.0 can potentially advance Text-to-SQL systems, bringing them closer to being practical for enterprise applications.

strengths:
1. Spider 2.0 is the first Text-to-SQL benchmark to incorporate complex contexts — such as data pipeline code and SQL dialect documentation — into the SQL generation process, reflecting an essential aspect of real-world SQL writing in a data engineer’s workflow. This marks a significant advancement over BIRD, which only includes simple evidence (1-2 sentences) as external knowledge.
2. SQL queries in Spider 2.0 are derived from actual projects and tutorials, enhancing the benchmark's realism and quality. Each database instance is selected to ensure sufficient complexity, with a minimum of 200 columns per database.
3. The evaluation in this paper is highly insightful, offering readers a clear perspective on the current limitations of LLM-based text-to-SQL approaches.

weaknesses:
The goal of this paper is to simulate real-world enterprise text-to-SQL workflows. While the authors have made substantial progress toward this aim, several important aspects of enterprise workflow characteristics remain unaddressed.

1. Context Complexity: The paper has included codebase and SQL docs as the context. However, in the enterprise setting, business context is also crucial. For example, different companies have different ways to compute daily active users or user retention rates. There can be a page-long documentation to explain how a metric should be calculated or even a page-long formula that defines a metric directly. This kind of context is missing from the benchmark. 
2. Schema Complexity: The paper says it selects databases where the number of columns is larger than 200. However, that does not guarantee that the schema is complex enough. For example, these 200 columns can be stored in one table. It is not clear how complicated the table relationship is. How many joins are involved in the SQL?
3. Question Complexity: The questions are generated by the authors based on the SQLs crawled from forums or tutorials, which might not reflect the nature of real-world BI questions, which often ask for trends or patterns. Including real data engineers to audit the benchmark or participate in the benchmark generation process can make it more convincing.

Despite these limitations, I recognize the authors' efforts in taking an important first step toward an enterprise setting.

questions:
1. How many tables per database in Spider 2.0? The paper only reports the number of columns per database. 
2. The paper's abstract states that it can "generate multiple SQL queries with diverse operations, often exceeding 100 lines." However, the queries average 144 tokens, which seems unlikely to translate into over 100 lines. Am I misunderstanding something here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707815938

thread_type:
official_review

<thread_object>
creator: reviewer_bdrh

<content>
summary:
This paper introduces Spider 2.0, a benchmark aimed at assessing the capabilities of large language models (LLMs) in handling complex text-to-SQL tasks within enterprise environments. Unlike previous datasets, Spider 2.0 involves real-world databases containing thousands of columns and diverse SQL dialects, with tasks that require interaction with database metadata, project-level codebases, and external documentation. The benchmark emphasizes challenges such as processing extensive schemas, using advanced SQL functions, and generating multi-step SQL queries.

Key findings highlight that current LLMs, even state-of-the-art models like GPT-4, struggle with Spider 2.0, achieving a success rate of only 15.1%. This reveals significant gaps in their ability to handle industrial-grade SQL workflows, which involve complex data transformations and analytical tasks. Spider 2.0 aims to push the development of more capable models for practical enterprise applications by providing a realistic and challenging benchmark.

strengths:
S1. It is important to have Spider 2.0 for real enterprise Text-to-SQL. Spider 1.0 and BIRD focuses on multi-table tasks, while WikiSQL and GeoQuery only focus on single table tasks. These datasets are popular, but they still represent a very small part of the whole data analytics/science workflow. The emerge of Spider 2.0 tries to fill this gap.

S2. The collection of Spider 2.0 complies the data manipulation workflow in real enterprise scenarios. Detailed types of queries and error analysis are shown in detail, which indicates the future direction of improvement.

S3. Spider 2.0 is a valuable dataset for the future agent of data analytics, which will be an important future trend of the data science domain. This paper also provides Spider-Agent, which can be a new baseline of data analytics agent.

weaknesses:
W1. There should be an opportunity to have an end-to-end benchmark for enterprise data analytics/ data science workflow with both SQL query and Python. Also, it would be better to also consider queries on Hadoop/Cassandra, which are also widely-used storages.

W2. In this paper, schema linking and table mapping results are shown. It would be better to open-source these two related datasets to extend the gap of LLM-based Text-to-SQL schema linking.

W3. It would be better to show the number of tokens of Codebase+DB Info+Documents, which should be an important parameter for picking up proper LLMs to construct agents.

questions:
1. W3

2. May I ask where can I find the proper json file for Spider 2.0 and Spider 2.0 Light in your repo?

3. Please figure out the difference between your work and MLE-Bench made by OpenAI
</content>
</thread_object>

</thread>
<thread>
creation date:
1730793278682

thread_type:
official_review

<thread_object>
creator: reviewer_xhzp

<content>
summary:
The paper introduces a new benchmark, Spider 2.0, which consists of 595 real-world text-to-SQL workflow tasks over ~1000 columns. The  text-to-SQL tasks are significant more challenging compared to the commonly used Spider and Bird benchmarks. The authors evaluate multiple LLMs using Spider 2.0 and present a set of interesting findings.

strengths:
S1. The new benchmark dataset is publicly available and critical for the community and enterprises that develop LLM-based text-to-SQL solutions.

S2. The dataset creation is solid and clearly presented in the paper. The SQL complexity including functions and dialects is closer to the real-life enterprise setting.

S3. The analysis provides some interesting findings, especially the error analysis of SQL generation.

weaknesses:
W1. The agent aspect of the benchmark should be emphasized more as it largely separates Spider 2 from the other benchmarks. It is reasonable to use execution results as the metric for evaluation. However, the other important aspect is the ideal multi-turn process for each task. For example, what is the typical number of turns and the sequence to complete the task.

W2. The difficult level in Sec. 3.1 is acceptable but not comprehensive. The SQL complexity reflects one aspect of the task difficulty. The other aspects include the data complexity (e.g., whether the task requires access to nested schema, any question/data ambiguity, etc.). If these are not as straightforward as the number of tokens, associating some tags to each task would help one to understand the specific challenges associated with the tasks.

W3. It seems that SQL is the only "code" allowed to execute the task. However, certain tasks such as data transformation might be easier to be executed through script languages (e.g., Python). IMO, a more realistic setting should allow LLMs to decide the best way to execute a (sub-)task. This will enable the LLMs, such as GPT-4o, to better leverage their strength in code generation.

questions:
Q1. Is CodeS-15B fine-tuned on Spider 2, or the fine-tuned model from the repo was directly used for evaluations?

Q2. Among dialect function usage errors, what is the ratio between simple syntax errors (i.e., LLMs not knowing the specific syntax) and completely incorrect chosen function?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731356176601

thread_type:
official_review

<thread_object>
creator: reviewer_mvyz

<content>
summary:
The paper introduces Spider 2.0, a new NL2SQL benchmark designed to address the challenges of real-world enterprise text-to-SQL workflows. Unlike previous benchmarks, Spider 2.0 focuses on 595 complex workflow problems derived from real-world enterprise use cases involving databases often exceeding 1,000 columns and stored across diverse systems like BigQuery and Snowflake. These problems require models to handle intricate SQL workflows, understand long contexts, explore database metadata, manage dialect variations, and interact with project-level codebases. The tasks frequently involve generating multi-step, nested SQL queries exceeding 100 lines, encompassing diverse operations from data transformation to analytics. Spider 2.0 thus pushes beyond traditional text-to-SQL challenges, demanding higher levels of reasoning and adaptability from language models.

The evaluations reveal significant gaps in existing language model capabilities, with the introduced code agent framework solving only 15.1% of Spider 2.0 tasks, compared to 91.2% on Spider 1.0 and 73.0% on BIRD. This highlights the difficulty of handling real-world enterprise scenarios, such as schema linking in large databases, navigating diverse SQL dialects, planning complex query sequences, and effectively leveraging external documentation. The insights from Spider 2.0 underscore the need for substantial advancements in autonomous code agents to meet enterprise requirements.

strengths:
S1) Originality: Spide 12.0 significantly extends traditional text-to-SQL benchmarks by focusing on real-world enterprise scenarios.  It incorporates a broader and more complex range of challenges, such as handling extremely large databases, diverse SQL dialects, and intricate workflows. This originality lies in redefining the scope of text-to-SQL tasks to better align with practical, enterprise-level applications.

S2) Quality: the benchmark created and curated 595 real-world text-to-SQL workflow problems derived from enterprise use cases, ensuring relevance and rigor. The inclusion of diverse database systems like BigQuery and Snowflake, along with tasks involving database metadata, codebases, and nested SQL queries, reflects meticulous data collection and a comprehensive representation of real-world complexity.

S3) Clarity: the paper is very well organized and carefully written. The content is very rich and deep. By providing detailed analysis and explicit examples of schema linking, dialect handling, and multi-step query generation, it offers clear insights into the difficulties faced by language models in enterprise scenarios.

S4) Reproducibility: the paper provides source code.

weaknesses:
I don't find any major weakness

questions:
No questions
</content>
</thread_object>

</thread>
<thread>
creation date:
1732168926045

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
# To all reviewers

We would like to express our sincere gratitude to all the reviewers for their thoughtful and constructive feedback on our submission. We genuinely appreciate the time and effort invested in helping us refine our work. 
We are also delighted that our benchmark is considered “*originality, quality, clarity, reproducibility*” (reviewer mvyZ), “*The dataset creation is solid*” and “*interesting findings, especially the error analysis*” (reviewer XhZP), “*Spider 2.0 is a valuable dataset for the future agent of data analytics*” (reviewer Bdrh) and “*The evaluation in this paper is highly insightful*” (reviewer c1AS).

We believe Spider 2.0 will make a highly impactful contribution. Its significance extends beyond the **text-to-SQL** domain: It builds upon the legacy of Spider 1.0 and BIRD, becoming the new most challenging text-to-SQL benchmark, while also serving as a general-purpose benchmark for **code generation and agent-based tasks**. Much like SWE-Bench and MLE-Bench, we envision Spider 2.0 as a foundational resource for advancing code intelligence research.

Our detailed responses address individual queries and suggestions from each reviewer, covering the following types of issues:

1. **More statistical information**: We have provided explanations for this information: difficulty level (Reviewer XhZP), context length (Reviewer Bdrh), number of tables (Reviewer c1AS) and SQL length (Reviewer c1AS).
2. **Complexity of task**: We addressed reviewer c1AS's concerns about complexity from three perspectives: Context, Schema, and Question.
3. **Evaluation metrics for agent task**: We appreciate Reviewer XhZP's suggestion and have provided an explanation of our existing analysis results on the agent evaluation metric.
4. **Can other programming languages be used**: Both Reviewer Bdrh and Reviewer XhZP raised concerns about whether Python can be used to solve the tasks in Spider 2.0. We have provided an explanation regarding this.
5. **Open-source more information**: As Reviewer Bdrh suggested, we will include more information, such as schema linking and table mapping details, when releasing Spider 2.0.


Once again, we thank all reviewers for their invaluable suggestions and insights, which have significantly contributed to the improvement of our paper. Please feel free to reach out if further clarification is required to assist in the final assessment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1740543728704

thread_type:
public_comment

<thread_object>
creator: ~harshraj_bhoite1

<content>
title:
Evaluating Spider 2.0: A Critical Review of Enterprise-Level Text-to-SQL Benchmarking

comment:
Summary:
The paper presents Spider 2.0, a benchmark designed to evaluate the performance of language models in real-world enterprise text-to-SQL workflows. Unlike prior benchmarks, which often use simplified databases, Spider 2.0 includes diverse database systems (e.g., BigQuery, Snowflake, ClickHouse) and real-world queries that involve thousands of columns, complex SQL dialects, and project-level dependencies. The authors highlight that existing LLMs struggle significantly with these challenges, achieving only 15.1% success, compared to 91.2% on Spider 1.0. This finding underscores the need for more robust models and techniques to bridge the gap between research-focused text-to-SQL tasks and enterprise applications.

Soundness: 4 (Methodology is rigorous, but evaluation could be more flexible)
Presentation: 3 (Well-structured but could be more concise in places)
Contribution: 4 (Important benchmark for advancing text-to-SQL research)

Strengths:

Enterprise Relevance: Unlike prior text-to-SQL datasets, Spider 2.0 closely mirrors real-world enterprise scenarios, making it a highly valuable benchmark.

Comprehensive Benchmark: Covers multiple database types, SQL dialects, and data transformation tasks, making it far more realistic and challenging than previous datasets.

Dual Task Settings: Provides both Spider 2.0 (interactive agent setting) and Spider 2.0-lite (standard text-to-SQL conversion), allowing for a flexible evaluation framework.

Thorough Evaluation: The paper benchmarks multiple state-of-the-art models, highlighting clear performance gaps between research-focused and real-world SQL workflows.

Weaknesses:

Rigid Evaluation Approach: The success criteria assume a single correct SQL query, whereas enterprise SQL tasks often allow multiple valid solutions. Using execution-based evaluations would provide a more flexible performance measure.

Limited Human Evaluation Details: The study mentions manual annotation, but inter-annotator agreement, annotator expertise, and verification steps are not well-documented.

Scalability Concerns: The dataset features extremely large schemas (thousands of columns), but the paper does not explore schema filtering mechanisms that could help LLMs handle them more effectively.

Lack of Adaptive Strategies: While LLMs perform poorly, the study does not experiment with fine-tuning approaches or hybrid retrieval-augmented methods, which might offer significant improvements.

Questions:

How do you ensure fairness in evaluating models across different SQL dialects? Are there cases where models perform better on specific dialects due to training data biases?

What are the primary failure points in model-generated SQL? Are issues related to incorrect schema linking, JOIN errors, or SQL syntax mistakes?

Would this benchmark be extensible to SQL tasks that integrate with external BI tools or ETL workflows? This could make Spider 2.0 even more representative of enterprise SQL use cases.

Flag For Ethics Review: No ethics review needed.
Rating: 8 (Strong contribution, recommended for acceptance with minor revisions)
Confidence: 4 (Confident in the assessment, but some additional details could strengthen the conclusions)
Code Of Conduct: Yes
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Spider 2.0, a new NL2SQL benchmark designed to address the challenges of real-world enterprise text-to-SQL workflows. Spider 2.0 focuses on 595 complex workflow problems derived from real-world enterprise use cases involving databases often exceeding 1,000 columns and stored across diverse systems like BigQuery and Snowflake. 

Overall, this paper is well structured and clearly written. The Spider 2.0 dataset is the first Text-to-SQL benchmark to incorporate complex contexts. The new dataset is publicly availbale and critical for the community and enterprises that develop LLM-based text-to-SQL solutions. Moreover, Spider 2.0 is also a valuable dataset for the future agent of data analytics. The benchmark creation is solid and clearly presented. The experimental analysis also provides some interesting findings.  All reviewers agree the acceptance of this paper.