Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
Authors: Fangyu Lei
Jixuan Chen
Yuxiao Ye
Ruisheng Cao
Dongchan Shin
Hongjin SU
ZHAOQING SUO
Hongcheng Gao
Wenjing Hu
Pengcheng Yin
Victor Zhong
Caiming Xiong
Ruoxi Sun
Qian Liu
Sida Wang
Tao Yu
Venue: ICLR 2025 Oral (2025)
Keywords: LLM Benchmark
Data Science and Engineering
Code Generation
Text-to-SQL
LLM Agent
Primary area: datasets and benchmarks
TL;DR: A benchmark for enterprise-level Text-to-SQL involving complex databases, challenging tasks, and real-world scenarios.

Abstract: Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics.
We introduce Spider 2.0, an evaluation framework comprising $632$ real-world text-to-SQL workflow problems derived from enterprise-level database use cases. 
The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake.
We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-level codebases. 
This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding $100$ lines, which goes far beyond traditional text-to-SQL challenges.
Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 21.3\% of the tasks, compared with 91.2\% on Spider 1.0 and 73.0\% on BIRD.
Our results on Spider 2.0 show that while language models have demonstrated remarkable performance in code generation --- especially in prior text-to-SQL benchmarks --- they require significant improvement in order to achieve adequate performance for real-world enterprise usage.
Progress on Spider 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-world enterprise settings.
Our code, baseline models, and data are available at [spider2-sql.github.io](spider2-sql.github.io) .

Published as a conference paper at ICLR 2025
SPIDER 2.0: EVALUATING LANGUAGE MODELS ON WARLD ENTERPRISE TEXT-TO-SQL WORK-
FLOWS
Fangyu Lei Jixuan Chen Yuxiao Ye* Ruisheng Cao
Dongchan Shin
Hongjin Su Zhaoqing Suo Hongcheng Gao Wenjing Hu Pengeheng Yin
Victor Zhong* Caiming Xiong Ruoxi Sun Qian Liu
Sida I. Wang Tao Yu University of Hong Kong Salesforce Research Sea Al Lab
Google Deepmind
Google Cloud Al Research * *University of Waterloo
ABSTRACT
Real-world enterprise text-to-SQL workflows often involve complex cloud or lo- cal data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics. We introduce Spi- der 2.0, an evaluation framework comprising 632 real-world text-to-SQL. work- flow problems derived from enterprise-level datahase cases. The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such BigQuery and Snowflake. We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation. and even project-level codebases. This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding 100 lines, which goes far beyond traditional text-to-SQL chal- lenges. Our evaluations indicate that based on ol ol-preview, work successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD. Our results Spider 2.0 show that while language mod- els have demonstrated remarkable performance in code generation - especially in prior text-to-SQL benchmarks they require significant improvement in order to achieve adequate performance for real-world enterprise usage. Progress on Spi- der 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-worke enterprise settings. Our code, baseline models, and data are available sp der2 sql. 1 github.
code agent frame-
INTRODUCTION
Automated code generation can serve as crucial bridge between humans and data, assisting dividuals in achieving difficult or monotonous tasks using complex data. A significant portion of existing data is stored relational databases, where SQL serves as an essential interface that fa-
cilitates human interaction with these data. In this context, semantic parsing text-to-SQU tQL (Dahl cet al.. 1994; Zelle & Mooney] [996] Zettlemoyer & Collins) 2005) Li & Jagadish| 2014 Zhong important technology that assists data analysts performing routine queries, orchestrating data workflows, and accomplishing advanced business intelligence, thereby significantly reducing repetitive human labor and alleviating the burden programmers. Large language models (LLMs) have demonstrated excellent capabilities in generating code (Chen) al. 202T] Austin al.) 202T) partieularly in transforming natural language questions into SQL queries. Notably, methods based on GPT-4 achieved execution accuracy of 91 .2% and 73.0% on the classic benchmarks Spider 1.0 YYu al. 2018) and BIRD (Li et al. 2024b), respectively.
et al.
2017 Yu et al., (018)
Although LLMs excel on these datasets, they often use non-industrial databases with few tables and columns, featuring simplistic SQL. and questions that fall short of real-world complexity and overlook diverse SQL. dialects. By contrast, real-world data are stored across a diverse array of
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Q:I need a daily report on key sales acliviies--coerinng events held, leads generated, and the status of opportunities.
covering tasks compleced.
Language Models
Text-to-SQL Workflow Environment Database
SQL
Documents
SELECT (1 dbe.date_tranct cay" cloos, date'] 13
Diverse
Database Meradata
CASE
System
External Knowladge
NOT MN Closed HEN "Lost"
is PHEM
SQL Dialect Docs
NoT
Apps
Real
Query Interface
SQLa/ Pythos
IN ('updlire" 'Gost usn') THEN "rapsllne"" eramus
FaON "mportunntyy)
Codebase
dbt
[+ 100 linss cmiteed
Complex
Table
acooumtidd
data/
macros!
models!
schema
Fure fredlask
15ST 10t8 event de.ditt
salastorce pwent-arttton artivityy Sare
Scherna
coltreer ovntted)
projectyml
leods.nd onj
ON da.data day apporminttieg
www.ttedddcaaeeeeeoooo dats
Table 485:
suhoforce.dd
...
+2 fines
ON ss. da..ate,aayy asgartunitten, wwwpeeeee_tttcoeeeeeooooeoo close, datt
Figure I: Spider 2.0 aims to evaluate LLMs on real-world enterprise-level text-to-SQL workflows. Solving each task requires understanding database metadata, consulting SQL dialect documentation, handling complex workflows, and performing intricate reasoning to generate diverse SQL queries.
database systems, each with its own unique SQL dialects, introducing wide range of SQL syntax and functions. Additionally, these enterprise-level application databases are characterized by large- scale schemas with thousands of columns and complex nested structures. Moreover, real-world text-to-SQL workflows require the utilization of project codebases, external knowledge, and various contexts to construct intricate SQL queries across multiple steps, complete various operations, and build comprehensive data engineering pipeline. This includes data wrangling to clean and organize the data for analysis, data transformation to restructure and enhance the data, and conducting data analytics to extract insights that inform decision making and drive strategic initiatives. All these complexities underscore the pressing need for a more realistic enterprise-level benchmark.
We present Spider 2.0, a benchmark that reflects world data workflows to facilitate the develop- ment of text-to-SQU models in enterprise applications, encompassing 632 real-world complex data wrangling, transformation, and analysis tasks. As illustrated in Fig. [ the databases in Spider 2.0 are sourced from industrial applications (e.g. Google Analytics and Salesforce) and feature massive schema items (an average of 812 columns) with unique structures (e.g., nested columns in Fig. [I] multiple schema in Fig. 12p. along with terabyte-scale data volumes. They encompass a variety of database systems, including local databases (c.g.. SQLite and DuckDB) and cloud data warchouses (e.g., BigQuery and Snowflake). Complicated SQL dialects for these databases are curated from technical tutorials, community forums, and open-source projects. On average, each ground-truth SQL query contains 144 tokens and includes advanced functions (e.g., ST ST_DISTANCE( (21, 222) measures the shortest distance between two points), exhibiting level of complexity notably sur- passing previous benchmarks. All tasks are based on project codebases along with documents and database interface to simulate real-world text-to-SQL writing scenarios.
Unlike previous datasets, Spider 2.0's agentic task setting does not rely on pre-prepared inputs (question and database schema) or expected outputs (predicted SQL). Instead, it incorporates a real project codebase and a database interface. This complexity extends beyond merely predicting an SQL query; it involves navigating the project and dynamically interacting with complex databases through SQL queries and command-line scripts (in Python Shell). The task objective is to per- form intricate data transformations within the database or to extract analytical insights from the data. This task setting closely mirrors real-world enterprise SQL workflows, requiring the model to refer to the codebase and documentation, generate multiple SQL queries, and dynamically interact with the environment to complete complex tasks and derive the final result. To simplify performance comparisons with previous text-to-SQL methods and benchmarks, and to support faster development and evaluation, we also introduce Spider 2.0-lite and Spider 2.0-snow, self-contained datasets with preprocessed database schema and documentation, the former is hosted on BigQuery, Snowflake, and SQLite, while the latter is entirely hosted on Snowflake. This setting omits the codebase and restricts output to SQL only, thus eliminating the need to predict final answers or transform the database. While they are sourced from the same raw data as Spider 2.0, these settings are not easier than Spider 2.0 because text-to-SQL setting have access to less information (c.g., execution feedback). We present Spider 2.0-lite and Spider 2.0-snow as direct tatt-iiputtt-----ouuee lenges that easier to work with using current advanced text-to-SQI. parsers, and Spider 2.0 as
chal-
2

---

Published as a conference paper at ICLR 2025
real-world data workflow challenge that involves interacting with diverse sources to perform data transformation and analyses.
Our evaluation Spider 2.0 indicates significant room for improvement in deploying LLMs within real-world enterprise text-to-SQL. workflows. The best ol ol-previww based code agent framework achieves a performance of only 21.3%, underscoring the significant deficiency in L.MMs' capability to serve as proficient SQL experts (Tab. (2) As for Spider 2.0-lite setting, even the most advanced text-to-SQL parser could successfully address only 5.7% of the questions, stark contrast to the execution accuracy of 91.2% on Spider 1.0 and 73.0% on BIRD, thereby highlighting the substantial challenges ($3.2). Our detailed analysis further identifies major obstacles enterprise text-to- text-to-SQL, SQL, including accurately linking schemas from extremely large databases, correctly handling different SQL dialects, planning sequences of nested SQL. queries to perform complex transformation and analytical tasks, and effectively leveraging external documentations and understanding project-level codebase. ($4.1 and $4.2). These challenges in Spider 2.0 represent crucial steps toward creat- ing benchmark that closely aligns with real-world scenarios. With Spider 2.0, we aim to enable the development of a new generation of intelligent autonomous agents capable of data engineering workflows in real-worlle enterprise settings.
BENCHMARK CONSTRUCTION
In this section, we introduce the task definition, general annotation pipeline, and dataset statistics for Spider 2.0, Spider 2.0-snow and Spider 2.0-lite. For concrete examples, refer to App B
2.1 TASK DEFINITION
Fig. 2 illustrates the task definition of both code agent setting and traditional text-to-SQL setting.
Question
Spider 2.0-lite/ Spider 2.0-snow
Code agent task. Spider 2.0 is defined as a comprehen- sive code agent task. Given a question Q, a database in- terface I. and a codebase C (with project context, config- uration, and documentation, illustrated in Fig. the task to iteratively modify the code (SQL/Python) C based on observations O₂ execute(C, Q) until the final result obtained. In other words, we
Text-to-SQL Parser
Self-contained
SQL Query
Question
Spider 2.0
Stlaa/
A (tex//tableddatabase) use the final observation Oş as an agent's answer to the question, i.c., A Ok.
Databsase
LM Agent
SQLe/Python Escc
Intermedion
Tex--to-SQL. task. In contrast Spider 2.0, Spider 2.0- snow and Spider 2.0-lite are formulated as self-contained tasks. Given database schema D, a natural language ques- tion Q, and auxiliary documentation & as inputs, the text- to-SQL parser f(-) is required to output the correspond- ing SQL query S = f(Q, D f), where 0 is the param- eters of the parser. Spider 2.0-lite's database is hosted on diverse databases like Spider 2.0, while Spider 2.0-snow is entirely hosted on Snowflake, with greater focus on text-to-SQL generation.
Final
Roalt D
Figure 2: We offer two settings: tradi- tional pox-inputttooSSLL-ounn Spider 2.0-lite/snow, and agentic Spider 2.0.
2.2 ANNOTATION PIPELINE
Eight authors majoring in computer science, all highly proficient in SQL, carry out the data annota- tion process. The annotation pipelines consist of the following six steps:
1) Database and SQL collection. We collect various databases from cloud data warchouses, includ- ing BigQuery public data, Snowflake Marketplace data, and other platforms, to ensure that they meet specific criteria: each database must contain more than 200 columns or have a nested schema struc- ture. After filtering, we select 74 BigQuery, 54 Snowflake, 30 SQLite, 40 DuckDB, 10 PostgreSQL, and 5 ClickHouse databases. From the corresponding tutorials and forums, we gather 1, 021 com- plex SQL queries, as well as 157 data transformation projects sourced from Fivetran and DBT (see A.2.B.2.. To meet our criteria, the SQL queries must contain more than 50 tokens (tokenized by whitespace; for reference, the average token count of BIRD (Li al. 2024b) is 30.9). Furthermore, queries must originate from real projects or tutorials, not from synthetic examples or corner cases. Ultimately, retain 547 high-quality SQL queries and 78 DBT projects.
3

---

Published as a conference paper at ICLR 2025
2) SQL rewrite to prevent data leakage. To avoid contamination and ensure the credibility of Spider 2.0's evaluation, annotators are required to rewrite each SQL and verify that they are bug- free. The rewrites are performed at two levels of increasing complexity: the surface and semantic levels, as detailed in Tab. [ 84.2% of the examples underwent surface-level rewriting, while 42% experienced semantic-level rewriting. Annotators must ensure that the rewritten SQL executes suc- cessfully, completes in an acceptable time, and returns non-empty results. 85.98% of these SQL queries utilize advanced functions in various dialects (App B.7.1). while 10.76% require additional DBT tools, posing challenges due to the need to integrate the project context.
Table 1:
The rewrite categories are follows: "Surface" rewrites adjust the parameters and the answer format, while "Semantic" rewrites expand the question's 's meaning. Each table reference in Example column represents the details of rewrite examples for the corresponding type.
Rewrite Categories
Example
Answer format
Tab.
13
replace the one channel with the channel ranking by sessions.
Surface
Condition parameters
Tab.
more complex filter condition: citibike is faster than a taxi.
Advanced calculation
Tab.
calculate originality score based on selected publications.
Advanced requirements
Tab.
16
change page view order to page conversion rate.
Semantic
Merge related SQLs
Tab.
77
merge geography related and weather-related queries.
SQL codebase files
App
change SQL and YML files in the original project.
3) Codebase and context setup. For each complex SQL query in Spider 2.0-lite and Spider 2.0- snow, we collect the external reference documents necessary to complete the task. Since the tasks span multiple database types, we gather documentation on SQL dialects and external functions, as shown in Tab. [8 Additionally, for Spider 2.0, preserve the original codebase of the SQL-related project. For Spider 2.0, besides collecting reference documents, annotators also gather resources such codebases, datahase interfaces to establish the context for each task (Fig. D-- Since some complex data transformation intentions may not be fully captured by a natural language question, annotators provide additional context, including data model descriptions (App(B.2) or predefined answer files (App B.5). to maintain clarity while addressing potential ambiguities.
4) Natural language task instructions annotation. Annotators are required to write questions based on the SQLs and context gathered in Step 3, crafting two versions for different settings. The instructions are designed to balance both naturalness and unambiguity. Due to the differences be- tween Spider 2.0 and Spider 2.0-lite/snow, code agent tasks demonstrate greater naturalness in its questions because provides contexts and predefined files to guide the answers, while text-to-SQI. tasks prioritize unambiguity, ensuring clearer and more straightforward specifications (see App[B.6] for differences). Annotators manually write the instructions, making them natural by avoiding blunt descriptions, removing ambiguity in the expected results, and ensuring that all SQL conditions are clearly mentioned. Also, the DBT-project tasks (see Fig. T and ABp(B.2). which are realistic data transformation coding scenarios, are exclusively used in Spider 2.0. Annotators craft task instruc- tions based on the provided context. After the initial annotation, they verify the semantic equivalence between the SQL queries and instructions, paraphrasing for elarity with the help of LLMss
5) Execution-based focused evaluation. In this step, annotators are required to obtain results from the databases programmatically and write evaluation scripts (details in App The evalua- tion scripts can process the results in the form of strings, tables, and database files. is important to note that in table-based evaluations, predicted results may include numerous columns, which might not exactly match the gold standard answers. This discrepancy often arises because some ques- tions do not specify the columns that should be returned. To mitigate this, the evaluation scripts are specifically focused on the essential components of the answers, ignoring non-essential columns and emphasizing the core elements outlined in the instructions. This method facilitates targeted assess- ments of key columns for each task, thus significantly reducing the occurrence of false negatives. For Spider 2.0-lite and Spider 2.0-snow, these settings require that the output must be SQL, so the evaluation will compare the execution results of the SQLs using the table-based assessment method.
6) Quality control. To ensure the quality of our benchmark, each instruction, the gold SQL query, and evaluation script are reviewed by least three annotators. We require the annotators to peatedly review steps 3). 4), and 5) to ensure the correctness, naturalness, and unambiguity of the annotations. Consequently, 45% of the examples have errors identified by the first validators. After
4

---

Published as a conference paper at ICLR 2025
discussions and corrections, following the second round of iteration with the second validators, only 5% of the examples contain errors. Then we correct all errors and refine all annotations, and ulti- mately, all examples deemed fully annotated. Additionally, of our automatic evaluation by providing a set of false results to determine if they would be correctly classified false, along with various correctly formatted results verify their classification true.
perform a "red team" assessment
2.3 DATASET STATISTICS
We present a detailed statistical analysis of the features of Spider 2.0, Spider 2.0-snow and Spider 2.0-lite, comparing them with multiple previous datasets in Tab. 2 our datasets demonstrate strong complexity and realism in aspects such as databases, SQLs, and task scenarios.
Table 2: Statistical comparison among Spider 2.0, Spider 2.0-snow and Spider 2.0-lite, and other text-to-SQL benchmarks. Tok. and Func. refer tokens and functions, respectively. * denotes the statistics from dev set due to the inaccessibility of test set. For more statistics, refer to A.p
# Test Examples
Tees DB
# Col /DB
# ToT.. /SQL
# Func. /SQL
External Knowledge
SQL Dialect
Project Level
Dataset
WikiSQL Zhong et al. 2017
15,878
5,230
6.3
12.2
0.0
x
Spider 1.0 Yu ct al. 2018
2,147
40
27.1
18.5
0.0*
x
x
x
KaggleDBQA (Lee al
2021
272
8
23.4
13.8
0.0
V
x
X
SEDE Hazoom et al.
2021
857
212.0
46.9
1.4
X
x
X
BIRD al. 2024b
1,789
15
54.2
30.9
0.4*
x
X
Spider 2.0-lite
158
803.6
144.5
6.5
Spider 2.0-snow
547
152
812.1
161.8
6.8
Spider 2.0
632
213
743.5
148.3
7.1
BigQuery
Diverse database systems and SQL dialects. As shown in
Snowflake
Fig. 3 and Tab. 3 our benchmarks feature diverse array of database systems, including cloud data warehouses like Big-
34.7%
Clickhouse 2.3%
Query and Snowflake, locally hosted databases such as Post- gres and ClickHouse, and lightweight systems like SQLite and DuckDB. This diversity distinguishes our benchmarks from previous work by encompassing various SQL dialects. No- tably, 85.98% of the examples require the use of specialized functions from these dialects, with an average of 7.1 special functions utilized in each ground-truth SQL..
4 4.7% Postgres
14.1%
SQLite
DuckDB
Figure 3: Data distribution on dif- ferent database systems.
Table 3: Statistics of Spider 2.0 task features.
Real and complex database schema. As shown in Tab. 2 the databases in Spider 2.0 are equipped with large-scale schemas comprising extensive tables and columns, effectively mirror- ing real-world enterprise environments. As shown in Tab. these databases are characterized by complex schema struc- tures (e.g., multiple and nested schemas, partitioned tables; see
Statistics
Number (% Total)
Total Levels (ttokens)
632 (120())))
-Fasy (Atokens <
160 (25.32%)
Fig. and Fig. 12), and dynamic tables that are updated daily.
Medium (80 #tokens < 160)
279 (44.15%)
Hard (#okkens 160)
Additionally, the data encompasses a broad spectrum of com-
With Bigquery
214 (33.866)
plex types (Fig. 16). extensive volumes, and diverse scopes
With Snowflakd flake
198 (31.33%)
(Fig. 15 rendering it more diverse than previous datasets.
With SQLite
135 (21.36%)
With DuckDB
(10,76%)
With Postgres
Challenging tasks across the data engineering pipeline. The examples in our benchmarks are collected from real tutorials and forums, covering wide range of issues encountered in
With Clickhouse
(1.11%)
With Project level (DBT)
78 (12.34%)
Documentation
(12.97%)
With Functions
474 (75.00%))
data pipelines, including data wrangling, data transformation,
With Partition Tables
54 (8.55%9%%
and data analysis (see App B. for examples). The difficulty
With Multiple Schemas
140 (22. 15%%)
of these questions significantly exceeds that of previous SQL- related benchmarks, as the SQL queries in Spider 2.0 contain
With Nested Schemas
117 (18.51%)
With StringNNumbrr Answer
162 (25, 63%)
With Table Answer
392 (62.03%%
significantly more columns, tokens, and functions per query
With Database
(12.3334%
than those prior work (see Tab. 2 and Fig. 18 for examples).
Real projects scenarios with codebases and documents. As demonstrated in Tab, 2 and tasks in
both datasets require access to documentation, like external knowledge B.4] and SQL dialect BB7). necessitating deep understanding of these resources. Compared to other prior works,
5

---

Published as a conference paper at ICLR 2025
for each task in Spider 2.0, we provide a codebase context to simulate real workflow (App(B.5) More notably, some tasks introduce innovations such as project-level data transformation work- flows built on DBT (App/B.2), widely used tool for managing data transformations and analytics engineering. Successfully addressing these tasks requires navigating complex project codehases and databases, comprehending documentation, processing intricate contexts, and generating diverse queries through multi-step execution and reasoning.
3
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Evaluation metrics. For Spider 2.0, we use the Success Rate (SR) metric, which measures the proportion of task instances successfully completed. For Spider 2.0-lite and Spider 2.0-snow, the output for each task must be an SQL, we use the widely used metrie Execution Accuracy (EX) et al. 2018 Li et al., 2024b). We employ the execution-based focused evaluation (App(A) to deter- mine the success of each result for Spider 2.0 and assess the accuracy of SQL. execution results for Spider 2.0-lite. The evaluation scripts are designed to accept output in the form of strings, tables, or database. For each example, an evaluation script is run for each example, producing score of either or 1. It is worth noting that in table-based evaluations, predicted results may contain numerous columns, leading results that are not exactly the same the gold answer. This occurs because, for some examples, questions do not explicitly specify which columns to return. The evaluation scripts specifically focused on the essential components of the answers, disregarding irrelevant columns and concentrating on the core elements specified in the instructions.
Difficulty level. We tokenize the gold SQL queries based whitespace and classify their difficulty according to the number of tokens: < 80 tokens as Easy, 80 - 159 as Medium, and > 160 as Hard]
LLMs. We experiment with state-of-the-art LLMs, including open-source representatives such as DeepsecKCoder-V2., (Zhu et al.) 2024). Qwen2.5-72B-Instnnn [Team 2024) and Llama-3.1-405B (Meta A1, 2024), and closed-soures ones including Gemini-Pro-Is (Reid et al. 2024), Claude- 3.5-Sonnet (Anthropic 2024) and GPT (OpenAl 2023) families (GPT-4o, GPT-4, ol ol-preview and o3-mini). Follow (Yang al., 2024a Chen et al., 2024), we use a temperature of 0.0 and truncate from the beginning of the input if still exceeding the max tokens limit required by the models.
Code agent frameworks. We utilize several state-of-the-an frameworks, which have demonstrated excellent performance on other benchmarks. These include Reflexion (Shinn et al. (023), CodeR (Cen et al. 2024), AutoEval (Pan et al., (2024). Inspired by React (Yao et al. 2022) and Intercode Yang et al. 2023). we develop agent framework called Spider-Agent, which is primarily focused on database-
coding tasks and projects. The framework allows for multi-turn interactions with the database via command-line interfaces until the final answer is obtained. The implementation details of Spider-Agent are shown in App C.T
Text-to-SQL methods. We also evaluate several state-of-the -rrt--rr- art and widely recognized text-to-SQU methods, including approaches based on prompting LLMs such as DIN-SQL (Pourreza & Rañiei 2024). DAIL-SQL (Gao al. 2024) and CHESS (Taleii et al. 2024). alongside SFT CodeS (Li et al. 2024a), which fine-tuned open-source models on extensive text-to-SQL corpora. DAIL-SQL and CHESS achieve the best performance among all accessible methods on the Spider 1.0 and BIRD benchmark, respectively. During implementation, we optimize the prompt organizations across all methods better align with tasks, incorporating sampled cell values, external knowledge, and SQL dialect specifications (see Fig. 21)
3.2 EVALUATION RESULTS
Existing LLMS are still far from being expert on real-worid text-to-SQL workflow tasks. As shown in Tab and Tablo we used Spider-Agent and its variants to conduct tests on Spider 2.0, Spi- der 2.0-Lite, and Spider 2.0-Snow The ol-preview and 03-mini achieve the highest performance, with a peak success rate of 23.77% on Spider 2.0-snow and 23,40% on Spider 2.0-lite, indicat- ing significant potential for further improvement. surpasses both GPT-4o and Claude-3.5-Sonnet across the Easy, Medium, and Hard cases, highlighting its superior reasoning capabilities. The open- source LLM DeepSeek-V V3 showed performance of 8.78%, still has significant room for improve- ment. The results shown in Tabl combined with the DBT project examples, also exhibit similar
While there are various ways to measure difficulty, SQL. length here the most common and significant metric for experimental reference.

---

Published as a conference paper at ICLR 2025
Execution Accuracy (EX) of different models using Spider-Agent on Spider 2.0-lite and Spider 2.0-snow, grouped by difficulty level.
Table 4:
Spider 2.0-Lite
Spider 2.0-Snow
Model
Easy
Medium
Hard
Overall
Easy
Medium
Hard
Overall
ol ol-preview
33.59%
23.58%
15.03%
23.22%
39.84%
21.14%
15.61%
23.77 %
03-mini 32.03%
26.02%
13.87%
23.40%
31.25%
18.29%
11.56%
19.20%
Claude-3.5-Somnet
26.56%
15.85%
6.94%
15.54%
25.00%
16.26%
7.51%
15.54%
GPT-4o 22.66%
13.41%
5.78%
13.16%
24.22%
11.38%
6.94%
12.98%
DeepSeck-V3 V3
19.53%
6.50%
4.05%
8.78%
20.31%
6.1%
4.05%
8.78%
Qwen2,5-Coder 13.89%
4.17%
3.38%
5.30%
11.72%
4.47%
2.31%
5,48%
Table 5: Execution Accuracy (EX) for baseline methods on three text-to-SQL datasets: Spider 1.0, BIRD, Spider 2.0-lite and Spider 2.0-snow.
EX
Method
Spider 2.0-lite
Spider 1.0
BIRD
Spider 2.0-snow
Easy
Medium
Overall
DIN-SQL GPT-4o
85.3%
55.9%
0.00%
5.79%
0.43%
0.00%
1.46%
DAIL-SQL + GPT-4o
86.6%
57.4%
2.20%
13.20 %
5.58 %
1.24 %
5.68%
CHESS + GPT-4o
87.2%
66.7%
1.28%
9.92%
3.00%
1.24%
3.84%
SFT CodeS-15B
85.4%
59.3%
0.00%
1.65%
0.86%
0.00%
0.73%
trend. Tab. iillustrates that Spider 2.0-lite and Spider 2.0-snow present significant challenges for traditional text-to-SQL methods. The highest performing method, DAIL-SQL GPT-4o, achieves an EX of only 5.68%, which is markedly lower compared to its score of 86.6% on Spider 1.0 and 57.4% on BIRD datasets. With efficiently filtering the minimal sufficient schema, CHESS GPT-4o is able to tackle more instances than DIN-SQL. Despite being extensively fine-tuned. SFT CodeS- 15B is far from solving Spider 2.0-lite, with EX score of only 0.73%, which further reveals the significant complexity gap between Spider 2.0-lite and the current text-to-SQL. corpus. For Spider 2.0-snow, even the best method achieves only 2.20% EX, highlighting the increased challenge due to SQL dialect differences.
Existing code agent frameworks struggle with solving database-related coding tasks. Tab and Tab. 6 show that the current agent frameworks are still unable to effectively address the tasks. The chal- lenge that they must not only explore the code- base and documentation, but also navigate complex databases and generate SQL queries that are far more intricate than typical code. This demands high level code grounding capability. Spider-Agent provides crucial baseline for Spider 2.0. facilitating the evaluation of various I.LMs, underscoring the potential for significant advancements and inspiring methodology enhancements for future research. We also observe that the model must be proficient de- bugging from SQL execution feedback and explor- ing the schemas of different types of databases (e.g.. Snowflake), which poses a significant challenge to code agent's s capabilities. There still signifi- cant room for improvement in Spider-Agent when it comes to enterprise-leve SQL tasks, in order to fully unleash LLMs' text-to-SQL capabilities.
Table 6: Success rate (SR) of different frame- works and models on Spider 2.0. The costs under different settings are shown in Tab 21 Spider 2.0 consists of Spider 2.0-Lite along with DBT-project tasks.
Framework
Model
SR (f)
AutoEval
GPT-4o
5.70%
Reflexion
GPT-4o
7.28%
CodeR
GPT-4o
7.91%
ol-Preview
21.36%
Claude-3.5-Somme
14.87%
GPT-4o
12.34%
GPT-4
9.86%
Spider-Agent
Qwen2.5-72B
6.17%
DeepSeck-V22 V2.5
5.22%
Gemini-Pro Pro-i-
2.53%
Llama- 1iie.3.144055 1-00BB
2.21%

---

Published as a conference paper at ICLR 2025
4 ANALYSIS
4.1 ANALYSIS OF DIFFERENT TASK TYPES
LLM-agent frameworks struggle interpreting databases with nested schema.
As shown in Tab. 7 the model often performs poorly when han- dling columns with nested types. Nested columns are common scenario industrial-grade databases (see Fig. [T], where data is stored in array, dict formats within a single column. This poses significant challenges for LLMs understanding the schema. As shown in Fig. 29 L.MSs encounter schema linking errors due an incomplete understanding of the information contained within nested fields. Most databases with nested types face the issue that models find it difficult to fully grasp the function of each nested column's internal information, while humans comprehend the schema through multi-step reasoning and iterative understanding.
Table Model performance on databases with nested columns in non-dbt projects.
Task Subset
%
(1)
w/ Nosted Columa w/o Nested Columns
18.51%
10.34%
08.04%
27.58%
The performance drops when external documents are required.
From Tab. g we observe that when tasks involve external docu- ments, the model performs poorly, correctly answering only 11 ex- amples out of full dataset that accounts for just 1 11.54%. Through error analysis, we find that the model is not incapable of ground- ing complex documents information. These models typically have the correct problem-solvine database, but fails at the most crucial step: grounding the com- plex requirements from the documents into SQLs. As the document shown in Fig. 13 the gold SQL shown in Tab. 16 The failure case shows that the model cannot combine complex document with schema information and convert it into SQL. query (Fig. 28
Table 8: Performance of the model on external document tasks in non-dbt projects.
Task Subset
of Total
strategies and effectively explore the
w/ External Doc
12.97%
11.54%
w/o External Doc
73.58
26.64%
LLM-agent frameworks struggle to address project-level tasks. As shown in Tab. g the LM agent's performance DBT-based project tasks poor, solving only 12.82% of tasks with just 10 examples correct. This underscores the challenges in there tasks, which can be attributed to: (1) Data transformation projects often require multiple SQL. queries to complete various models, neces- sitating a comprehensive understanding of the project. (2) These tasks involve complex context usage, demanding strong repository exploratory capabilities, (3) Data is stored in databases, requiring the agent to transform data while exploring existing data, alongside SQL coding. Fig. 26 illustrates the action process of op-revee successfully solving task defined in [B2 while Fig. 27 is a failure case due to the failure to explore the information in the "mr..md file to solve a monthly recurring revenue classification.
Table 9: DBT Project.
Performance on
Task Subset
SR (t)
w/ DBT Project 12.34%
12.82%
w/o DBT Project
87.65%
23.22%
4.2 ERROR ANALYSIS OF SQL GENERATION
We conduct detailed analysis of the errors encountered by both code agent frameworks on randomly sampled 300 examples, as il- lustrated in Fig. 4 Representative errors along with their statisties and causal analysis are as follows.
Whong bable 10.%%
ppess soor 4%
secerrte ereing
Werrig coloven 18.8%
Erroneous data analysis (35.5%). Compare to the previous benchmarks, Spider 2.0 and Spider 2.0-lite exhibit significantly complex data analysis demands that challenge the models' capa- bilities: 1) Dialect function usage (10.3%). This includes pro- cessing temporal (e.g., DATE TRUNC) or geographic data (e.g.. ST ST_DISTANCE). These functions require a nuanced understanding, which the models often fail to exhibit. 2) Advanced data calcula- tion (7.5%). Model struggle with tasks like grouping samples analyze trends within groups (using NTILE), or applying formu- las for statistical values (e.g.. CORR for Pearson correlation coeffi- cients; STDDEV for standard deviation). 3) Intricate query planning (17.7%). Gold SQLs typically involve multiple nested queries, in-
27
Conditon error
11, 1..5%
dess A aoree Saa Soa
Erronwoos
Joln etrol 8.3%
aceely
Peecon Jed pessorentt calouderoo
28.5%
iecumee plarnin
Figure 4: Statistics of errors. For detailed descriptions and examples of each error cate- gory. see App C.3
8

---

Published as a conference paper at ICLR 2025
termediate result processing through common table expressions (CTEs), merging results from various sub-queries via set operations. However, models often inadequately handle these complexi- ties. Refer to Fig. 5 for case studies erroneous data processing.
Wrong schema linking (27.6%). This category includes errors with wrong tables and columns. For column linking errors (16.6%), the average number of columns per database in Spider 2.0-lite far exceeds those in other benchmarks (over 755 compared to approximately 54 in BIRD), making ac- curate column linking extremely challenging. Regarding table linking (10.1%) 16), although examples from BigQuery support advanced syntax features like (TABLE_SUFF F IX) and wildcard expressions, the models show limited flexibility in leveraging these features, even in few-shot setting.
JOIN errors (8.3%). While foreign keys represent known schema relationships essential for valid SQL JOIN operations, databases in BigQuery often lack explicit foreign key. This omission forces models to infer potential keys based on column names and descriptions, leading to errors.
Table 10: EX for baseline methods on Spider 2.0-lite under oracle setting. To seek the highest possible performance, we also employ the latest ol-preview as the base LLM.
Table 11: EX for DAIL-SQL. Spider 2.0-lite under few-shot setting with manually selected demonstrations.
Method
EX (1)
Method
EX (1)
w. w.Oracle Fune w/o Orucle Func
0-shot
1-shot
3-shot
DAIL-SQL GPT-4o
5.85%
5.68%
+
5.68%
6.40%
6.76%
DAIL-SQL oo--preeee ol-prrview
9.51%
12.60%
Quantion:
Quatiin
For pet trpe with dors dordire more and the wordd vinvts and brooon to vimmes, tripuhanation anttrratttttttttiiiiii lato Biggarilles aeenge
Cas you grap carilly the veek they fiedt need the deeting five July 2 201, end ihenty tee grves fivt has thwhiifaant intettion male wheh fourh woeh Tiic greap shood ramed Manday dus ther rook Peee providette
Gold SQT:
Gold Rowdl::
SQL:
SELOCT
Syentiiyy
thdrr fiist sesston start
Appropete into veatly
tetal
10T1 dets
celoclate veat sonce atort
TORAATES (D.
SELECT DEST2RCT
space (M:19]
'eveet yoan''s
AG
con wes nate stree
(rays stare start as weess
owe
comsne duratien admates)
047t100er dirst teach Eonestanpt (RY) x
Cuest the rember intalied esers each veek
COUSTI33 trine,
15D nos user
SELECT ved. conart, webks strce start,
moh c SELECT
with evert dets bettl 2010 on 463 NC 7,
FED wath. mittissttetionn
an darortin, In peuaeeeee,
firct coccioe stare dores
IRCUP
rase chiroguraet pelpe
SHITT nina ssen ponann 14,
Inif und
so aecenip ceronsuoey
calralate store the start cuch
1997 19th eeterit
As past
SELECT DESTINC
JA0 for
GOLP in daratiin ptloutt sartile
DATE wwt.t (..)
Usss Inttial meao Lonoe start
ceos quartile
for. www.://ttttttttt.oomm www.saaa..ttttttttruu.
Predkted SQUI
Predeted Readh
Pridkted SQL::
SRjrCT
(nDream) 35 everapr hare
Tittieeer etrst trech Comestand), WEB (HOBAY)
tria, secoidd, fane
1II AS actiie wtwpoueeeoooorroreeeeee H4M wasen putllc wew.tttt....ttt..........
500e COSTTECT
Prodiced Resalt:
(Dnzee an tedp, pecuudo
raze rasr rrips
serees in an 1994 siveeryy
112100
retejieer, firer "ourn (tw.draaaa) ner
rene By srek rekert
coou qurrtile
ceosk quartile
LEGC 13411
(a)
(b)
Figure 5: Case study of two representative incorrect SQL predictions due to erroneous data analysis. (a): An example of incorrect data calculation, where quantiles were incorrectly divided based on the number of trips, rather than on the trip duration as required. (b): An example of incorrect planning, where the predicted SQL incorrectly sorted data by the number of users, rather than by the required retention ratio. The prerequisite for achieving this is to properly plan a sequence of CTES. Additional examples of error cases across all categories are available in Fig. anl Fig. 23
4.3 ANALYSIS OF DIFFERENT EXPERIMENTAL SETTINGS
Providing oracle functions leads to a slight performance improvement. Considering that Spider 2.0 and Spider 2.0-lite involve SQL dialects from various database systems, we provide syntax and

---

Published as a conference paper at ICLR 2025
function documentation for each system to prevent the methods from suffering due to lack of syn- tax knowledge. For each example, we manually include the relevant function documentation that may be required, eliminating the need for retrieval method and ensuring that the necessary syntax knowledge is readily accessible. As shown in Tab. TO providing oracle SQL. function documen- tation results in only slight improvement in model performance. This suggests that, to certain extent, models are capable of selecting appropriate functions and understanding their basic usage and syntax. However, the critical challenge lies in accurately utilizing these functions to reflect user intentions, as illustrated in Fig. a).
Few-shot prompting has little impact on performance. Spider 2.0-lite not divided into train and dev sets, we manually select representative examples from the same SQL dialect as the SQL to be predicted, with distinct characteristics (encompassing multiple CTE or nested queries, or requiring intricate data processing) to serve as few-shot examples. Unexpectedly, few-shot in-context learning shows only marginal improvements in performance (see Tab. [T). This may be due to the gap between the simplistic text-to-SQL pre-training data used with LLMs and the complexity of the few-
shot examples. Additionally, extensive schema prompts may hinder the model's ability to effectively assimilate information in the few-shot examples.
5 RELATED WORK
Code generation and text-to-SQL benchmark. As model capabilities advance, code generation benchmarks have become more complex and generalized. Many benchmarks (e.g.. SQL-Spider Yuu
al.
2018). Bash-NL2Bash (Lin al. 2018). Python-HHumnnEval (Chen et al. (02T) treat code
generation as seq2seq tasks. Many previous works (Lai et al. 2023 Yin et al. 2023 Huang et al.
2024
Chan al. [2024] Jing et al. (024) define code generation tasks for data science. MLAgent-
Bench (Huang et al. (023) and Intercode (Yang et al., 2024b) focus on interactive environments, coding tasks. Spider2-V (Cao al. 2024) proposes data science and engincering benchmark in multimodal setting. Many previ- ous datasets (Zhong et al. 2017 Lee et al. 2021 Hazoom et al. 2021 Wang et al. 2020 Li et al.)
while Senc (Jmenee et al. 2023) emphasizes repository-
2024b) have made significant contributions to the advancement of text-to-SQL tasks. However, isting text-to-SQL benchmarks primarily target lightweight local databases, much smaller in schema scale and data volume than cluster- hosted industrial databases, and fail to capture the agentic nature of SQL programming using various dialects in real scenarios. Spider 2.0 bridges the gap between research and enterprise-level industrial text-to-SQI. workflows.
Code agent framework and text-to-SQI. methods. The intersection of generative code models and interactive problem-solving has spurred significant advancements in both agent-based frame- works and text-to-SQL methodologies. Recent efforts aim to enhance the reasoning capabilities of language models, as evidenced by surge in agent methods designed for code generation tasks ( (Yao) et al. 2022 |022| Zhang et al. [022] Chen et al. 2023] Wang et al. 2023bb Shinn et al. [224 Zhang et al., 2024 Xia et 2022). Several works have designed special actions to standardize agent operations (Wagg et al. 2024| Yang et al. (024a). For methods specifically designed for text-to-SQI,, sev-
eral tine-tuning methods (Li et al. 2024a) and L.MM-prompting methods (Dong et al.) 2023 Wang cet al., [203a;; Zhang al.) 2023] Talaci al.) 2024; Pourreza & Rafici, 2024, Gao al. 2024)
have achieved strong performance on previous benchmarks. We propose Spider-Agent, a code agent framework specifically designed for database-relate tasks, showcasing strong performance in this domain. For Spider 2.0-lite, we also adapt several text-to-SQL methods to suit our benchmark.
CONCLUSION
We propose Spider 2.0, benchmark for real-world enterprise-level text-to-SQL workflow tasks. encompasses diverse database systems with various SQL dialects, large and complex database schemas, and challenging tasks across the data engineering pipeline, all set within real project scenarios including codebases and documentation. Despite being the most advanced LLMs (ol- preview), they still perform poorly on Spider 2.0, achieving success rate of only 21.3%, which underscores its status as a highly challenging benchmark. Spider 2.0 presents a novel challenge for text-to-SQL research, providing direction towards more realistic and intelligent solutions.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The authors of this paper were supported by the ECS (27212023) from RGC of Hong Kong. We thank Snowflake for their generous support in hosting the Spider 2.0 Challenge. We also thank Tianbao Xie, Yiheng Xu, Fan Zhou, Yuting Lan, Per Jacobsson, Yiming Huang, Canwen Xu, Zhewei Yao and Binyuan Hui for their helpful feedback on this work.
REFERENCES
Anthropic.
The claude model family: Opus. sonnet, haiku. cdn. http::/cc......oooooooooooooooaa 2024.
https://www- Card Caadde Claude...pff
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.0775 2021.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, et al. Spider2-v: How far are mul- timodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10956. 2024.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung. Dane Sherbur, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang. Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong. Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.0133 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1xXiv:2107.0337. 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schaerli, and Denny Zhou. Teaching large language models to self-debug. In The 6 Annual Meeting Of The Association For Computational Linguistics, 2023.
Deborah A Dahl, Madeleine Bates, Michael K Brown, William M Fisher, Kate Hunicke-Smith David S Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. Expanding the scope of the atis task: The atis-3 corpus. In Human Language Technology: Proceedings of a Workshop held at Plainsboro. New Jersey, March 8-11, 1994, 1994.
Xuemei Dong, Chao Zhang, Yuhang Ge. Yuren Mao, Yunjun Gao, Jinshu Lin, Dongfang Lou, et al. C3: Zero-shot text-to-sql with chatgpt. arXiv preprint arXiv:230..07336, 2023.
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. Proceedings of the VLDB Endowment, 17(5): 1145, 2024.
Moshe Hazoom, Vibhor Malik, and Ben Bogin. Text-to-sql the wild: A naturally-occurring dataset based on stack exchange data. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pp. 77-87, 2021.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking large language models as ai research agents. arXiv preprint arXiv:2310.03302. 2023.
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang. Xiao Liu, Jun Zhao, et al. Da-code: Agent data science code generation benchmark for large language models. arXiv preprint arXiv:2410.07331, 2024.

---

Published as a conference paper at ICLR 2025
Carlos E Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2023.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents to becoming data science experts?, 2024. URL https: / rxiv. .rg. 240.. 077033
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18345. PMLR, 2023.
Lee, Oleksandr Polozov, and Matthew Richardson. Kaggledbqa: Realistic evaluation of text-to-sql parsers. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 2261-2273, 2021.
Fei Li and HV Jagadish. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1), 2014.
Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data. 2(3):1-28, 2024a.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang. Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024b.
Xi Victoria Lin, Chenglong Wang. Luke Zettlemoyer, and Michael D Ernst. NI2bash: corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Meta AL Introducing meta Llama 3: The most capable openly available LLM to date, April 2024. URL http:/ttt///..eeeee com/ foom/bloge blog/ meta- 1lama-33311 3// Accessed: 2024-04-18.
OpenAl. Gpt-4 technical report. arxiv 2303 2303.08774. View in Article, 2:13, 2023.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
Mohammadreza Pourreza and Davood Rafici. Din-sql: Decomposed in-context learning of text-to- sqi with self-correction. Advances in Neural Information Processing Systems, 36, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflex- ion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Shayan Talaci, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. Chess: Contextual harnessing for efficient sqi synthesis. arXiv preprint arXiv:2405. /6755, 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024, URL ht / qwennm. github. wt/blo/////////ooo qwe.2.
12

---

Published as a conference paper at ICLR 2025
Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian- Wen Zhang. Zhao Yan, and Zhoujun Li. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arXiv:2312.11242, 2023a.
Qian-Wen
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. slan-and-solve prompting: Improving zero-shot Sfain-of-thoughe reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 3609-2634, 2023b.
Ping Wang, Tian Shi, and Chandan K Reddy. Text-to-sal generation for question answering electronic medical records. In Proceedings of The Web Conference 2020, pp. 350-361, 2020.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng. Heng Ji, and Graham Neubig, OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, 2024. URL htttttt:://aa pttpps://arxiv..o wtwxii..rg//bb//4000
16741
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying Ilm-based software engineering agents. arXiv preprint
2024.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing
and benchmarking interactive coding with execution feedback. CoRR, abs/2306. 21t222206..18899 14898, 2023, doi: 10.48550/arxiv.23366 14898. BRLhhtps: wtg/:/o..ooo/.../ http:/10.488888 48550/arXiv 2306. 14898
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent computer interfaces enable software engineering language models, 2024a.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua How- land, Paige Bailcy, Michele Catasta, Henryk Michalewski, et al. Natural language to code gen- eration in interactive data science notebooks. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 126-173, 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao. Shanelle Roman, al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 1991-3921. 2018
John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pp. 1050- 1055, 1996.
Luke S Zettlemoyer and Michael Collins. Learning to map sentences to logical form: structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Confer- ence on Uncertainty in Artificial Intelligence, pp. 658-666, 2005.
Hanchong Zhang, Ruisheng Cao, Lu Chen, Hongshen Xu, and Kai Yu. Act-sql: In-context learning for text-to-sql with ceoommttccall-ggneeaaee thain-of-thought. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3501-3532, 2023.
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. Planning with large language models for code generation. In The Eleventh International Confer- ence on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2025
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1592-1604, 2024.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li,
Huazuo Gao, Shirong Ma, et al. Deepseek-coder Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 1 1931, 2024.
14

---

Published as a conference paper at ICLR 2025
A SPIDER 2.0 EVALUATION SCRIPTS
In this section, we present the detailed definition and discussion of evaluation metrics for Spider 2.0- and Spider 2.0.
Spider 2.0-lite. The setting of Spider 2.0-lite resembles that of a traditional text-to-SQL task in which text-to-SQL. parsers are required to generate SQL. queries. Therefore, Execution Aceu- (EX)(EX) is used the primary evaluation metric. Slightly different from existing works, employ an execution-basel focused evaluation, which measures whether all columns in the gold value are present in the output of the predicted SQL query. This is defined as follows:
En 1(0", N
EX
(1)
where ((,,0) 1, if us E 6, E
(0, v; $ Bu; E
where o; represents the i-th column of data frame v, v" and i" denote execution results of the gold SQL and predicted SQL for the n-th instance in the evaluation set, respectively. Empirically, this evaluation method significantly reduces the false negative rate without increasing the number of false positives. Given that the ground-truth values result from extensive data wrangling, transformation, and analysis, it is difficult for models to manipulate or exploit the system.
Spider 2.0. We the Success rate (SR), which measures the proportion of task instances suc- cessfully resolved. Humtn-written evaluation scripts are used to determine whether an example is resolved. For each example, we provide string-based, table-based, and database-based evaluation functions, depending on the type of answer output, as shown in Tab. 12
Examples. Maintaining naturalness and unambiguity is often a confficting challenge. To ad- dress this, we provide an example to illustrate the important parameters "condition_cod ccols" and "ig- nore order". Achieving balance between these two aspects quite challenging, which is why we incorporate this mechanism into our evaluation scripts.
Given a data frame with a set of column vectors {er], each representing the cell values for the i-th column, prediction á is considered equivalent with v if and only if for any V; E v, 2/ E é. Therefore, at such times, we only check whether a specific column appears in Intuitively, if all columns in the reference table appear in the result table, the result is considered correct.
For example, as illustrated in Fig. 6 the question does not explicitly specify which columns are required in our response. Consider the following question: "The company management has re- quested a detailed report on the year-to-date performance of the Magnificent 7 stocks.". We need to carefully analyze the task requirements and only check if the following columns in the refer- ence answer-" "nswer_""ickkrr" "Ticker", "Change_YTi YTD""appear in the predicted answer. This meets the semantic requirements of the abstract instruction. Empirically, we find our evaluation metric is reliable in identifying solutions with alternative output, with a relatively low false-negative rate.
Task
los requanted detailed The thee parformarce of the Magriicent 7 stocks.
Gold Answer
Agene rauuta
S/ARTOAEL
LATULAX
OWAKLYY
1551829913
3082
5000
180
440044777
19855
59.8%
90%%
353447477
751449-77
92
MBYA
25332
81:2100018
34821
38877
AMP
47:02
155
252445-37
303.0
5/60815004
504
2534 07442
2557
3611
MRC
2130009118
15322
01295
222447442
470
151449-7
135
Score
Figure 6: An example of evaluation scripts for table-based evaluation: in this example, the condition.com is (0,5). and the ignore.oroe is true. As long as these two columns predieted correctly, the example can be considered solved.
15

---

Published as a conference paper at ICLR 2025
The evaluation scripts for Spider 2.0 are tailored to the specific format of the model's output. Each script optimized to handle various output types, ensuring precise and contextually appropriate evaluation.
Table 12:
Output Type
Description
Parameters
String w/o number
If the answer is found in the string, it is given a score of I; otherwise, it receives a score of
pred (str): The string in which to search for substrings. gold (List of str): A list of strings to check within the predicted string. conj (str): The conjunction used for matching ('and' or 'or'). Default is 'or'. exclude (List of Str): Strings that must not be present in the answer.
String w. number
For output strings containing numbers. the script captures these numbers and performs num- ber matching for scoring using the number maatchaahhhhhhhh function.
pred (str): The string in which to search for substrings. gold (List[strr float]): list of strings or numbers to check within the predicted string. percentage (bool): Default false. the gold answer is related to percentages, set this to true for more robust evaluation.
precision (int): The number of decimal places to con-
sider. Defaults
conj (str): The conjunction used for matching ('and' or 'or'). Default 'or', and it's typically 'or'.
Table
If the answer is a CSV file table in string format, table- level evaluation is performed.
result (str): Path to the CSV file or result string. gold (str List[str]): Path(s) to the gold file(s), exclud- ing the root directory. Multiple potential gold answers supported.
condition_eam cols (List(int] List[Lis(fint]]: List of col- umn indices to match conditions. For example, [0, 1] uses the Oth and 1st columns in the gold table for match- ing, while ignoring the others.
ignore_ores .order (bool): Whether to ignore the order of rows when matching elements.
Database
If the answer is stored in DB file, database-level eval- uation is applied using the db_match function.
result (str): Path to the DuckDB file containing the result tables.
gold (str): Path to the DuckDB file containing the gold standard tables.
condition tabs (List[str), optional): List of table names to be checked. If not provided, all tables in the gold DuckDB file will be considered.
condition cols (List[List[int]], optional): A list of lists, where each inner list contains column indices used for matching conditions for the corresponding table. De- faults to considering all columns.
ignore orders (List[bool], optional): A list of boolean values indicating whether to ignore the row order for each table comparison. Defaults to [False] for each table.
SQL
If the output an SQL, execution- based evaluation is used. This is primarily designed for Spider 2. 2.telite.
To compare the execution results of the predicted SQL and the gold SQL, table matching is used.
16

---

Published as a conference paper at ICLR 2025
B ANNOTATION DETAILS
B.I SQL ANNOTATION EXAMPLES
In this section, we present several representative examples of the SQL annotation process, includ- ing the original SQL, how the SQL was rewritten to obtain the gold SQL, and the use of external knowledge.
Tab. 1 presents an example based on the Google Analytics database. The task is to calculate the source of web traffic and count the number of sessions for each traffic channel within a given time period.
Tab. presents an example based on New York City public data, where the task is to find Citibike and taxi trips between specified locations and analyze whether Citibike or taxi is more suitable for travel between the two locations. this case, the condition in the original SQL. is to calculate trips between the two locations by Citibike and car. We extend this condition by introducing real-life problem: identifying which routes are faster by Citibike compared to taxi.
Tab. 15 is based on the Google Patents database, which contains large amount of patent informa- tion. The original SQL applied several filtering conditions to retrieve a set of patents. We find a document explaining how to calculate patent's ' originality score, which led to an advanced calcu- lation method. As a result, the final task include additional complex calculation steps.
Tab. 16 is also based the Google Analytics database. The original SQL calculates the Product List Page (PLP) and Product Details Page (PDP). Based on the description in the blog, we define a new task to calculate the conversion rate by determining the probability of users clicking from PLP to PDP.
Tab. 17 presents an example where we merge and rewrite related SQL queries. The first SQL calculates the 50 weather stations closest to downtown Chicago, while the second SQL calculates the number of bike trips on rainy and non-rainy days in New York City. We combine these two tasks, meaning that to determine whether it is a rainy day, we first need to find data from the weather station closest downtown New York City.
B.2 DBT PROJECT ANNOTATION EXAMPLES
Annotation Pipeline of DBT Project. The DBT project be found online resources and is one of the projects with the most SQL scripts. Similar data transformation tools widely used in industrial production. Completing a DBT project requires a comprehensive understanding of both the code and documentation within the project to accomplish the entire task. Fig. shows a cerefforeebbseed project in Spider 2.0. This represents a natural and realistic SQL generation scenario. Using a Fivetran Salesforce transfor- mation package as an example, we transform complex DBT project into Spider 2.0 example through the following steps.
dbt_utils
macros
custam_pereee periss_pors_thrrughccoummmm
salesforce
Intermediate
Int, saalssorce__dat__spphe
int www.ssaarcee_appaee.
intermediete.ymi
staging
balesforee__mmnger__oo
baanager_peroormnnce.. sqi
allesforee_spppottuitty_iii wawpanttyiiin_tee_eeeeeeee hem_enhanced.sd
salesforce__wweer
salesforce_ www.sslllessssottettcoooo
(1) Run DBT project from start to finish, suring it is bug-free and generates a dbt DAG (Fig. 9.. This allows for comprehensive under- standing of the data flow.
selesforce
HD salasforoa.md
po__projcc.ymi
profiles.yml
salesforce.dudke
(2) The DBT project includes yml files and mark- down documents, where the project developers have already planned out the data models and data flow. We will use these as the basis for task instructions.
Figure 7: Codebase for a DBT project, showing models, macros, and configuration files.
17

---

Published as a conference paper at ICLR 2025
name: httpoore__oorrttnttyymooom
description:
table both cleans and erhances
opportunities table with information about spportunity
nsme: oppertunity
name: user
namel user_role
name: oppartunity_jo
description: The unique, system-geterated TD assigned during creation.
tests:
not_natl
description:
account associated aspertunity.
name: account_id
description: 10 of the account associated with this aspertunity.
name: account_type
description:
for anoust. opportunities products, the anount related products.
Competitor, Partner.
description:
name: campaige_1c
description: 10 of related Compaign. This field is defined only for those organizations that have the campaign feature Compaigns enabled.
name: close_date
description: Required. Date appartunity
description: when the apportusity created.
name: petrrtuunty_esscrittt
description: Text description of the apportunity.
name: expected__rveee revenue
description: Read-only field
product the opportunity Anount field and Protability.
description: fiscal years are enabled, the name fiscal quarter peried which apportusity Closedate falls.
name: fiscal_qurrte
description: Represents the fiscal quarter. Valid values are 2, or
fiscal_year
description: Represents
year, exaeple,
description: The category of the state apportunity. Default values are "pipeline', 'forecast', 'bestcase", 'closed', and 'omited"
name: Torecast_catepprt cartstt_ctgory_aae
description: The name of the forecast category.
has__pen_ _attviiiiiyy
description: Indicates
(false).
has www.oooortuiityy_eerre
description: Read-only field indicates whether the apportunity has associated itens. value of neans that Opportunity line
name: mas_overdae__ask
description: Indicates whether opportunity has an overdue task (true) or not (false).
Last iist_activity.de
Value
whichever
the Bue
Last_referencee_Gn
description: The tinestamp when current user last accessed this record, record related this record, or view.
name: Last_viewed__dae date
description: The tinestanp when the current user last vieved this record or List view. If this value is null, the user night have only acces
Lead, source
opportunity,
Trade
This is a common configuration file in DBT projects used to define the schema data model. It represents natural SQL generation scenario, specifying details such as field names, data types, and references for the
Figure 8:
data model.
(3) We remove the sql files corresponding to a specific data flow within the complete DBT project. For example, in Fig. 9 we may delete one to three data flows, as shown in Fig. I0 removing "sales daily activity" and "salesforce contact enhanced" along with their upstream nodes. This turns it into an incomplete transformation project. Note that the DAG figure is only used as an aid for data annotation, and the task does not include any images.
(4) Write the task instructions. For instance, we can draft a prompt like, "I need a daily report on key sales activities-ccoeringg tasks completed, events held, leads generated, and the status of opportunities." Although the data model contains many columns, thanks to the presence of yml files (see Fig. 8 there is need to describe the output columns in detail in the instructions.
Approach to Solving DBT Project Examples. As shown in Fig. 26 completing a DBT project example typically requires the following abilities:
1) Problem comprehension. First, is necessary to fully understand a natural language task.
2) Project reading ability. A real-world data transformation project consists of multiple files, as illustrated Fig. 7 The method needs to explore the codebase and review relevant project files, including -yml, .md. and .sql files. YML files (Fig. () generally define the data models for the data transformation, .md files contain textual descriptions of the data models, and SQL files are the data transformation models themselves.
18

---

Published as a conference paper at ICLR 2025
PLLIMII 200/
Figure 9: DAG (Directed Acyelic Graph) illustrating the data flow and dependencies between various Salesforce tables and models in dbt (data build tool) project. The graph shows stages of transformation, from raw Salesforce data (green nodes) to enhanced and aggregated models (blue nodes), representing different entities such opportunities, contacts, accounts, and events.
Mig
IANORMANT
Jeb Soa. 2ww. JOPERII
Figure 10: In this version of the DAG, several data models missing, including "sales- purce_aaly-actiive and cecesforceccontatteente ates incomplete data flow compared the original.
along with their upstream nodes. This cre-
3) Database exploration ability. The codebase only contains the data transformation code, while the data to transformed is stored in a database. The method must explore the database understand the available source data and identify any missing data models.
4) Problem localization ability. By combining the natural language problem and the YML files, the method should locate where to add or modify the code in the project.
5) Coding ability. The method needs complete complex data transformation code based on the data models defined in the YML files and add the .sql files in the appropriate locations. Visually, it requires completing the data models defined the yml file, transitioning from Fig. to Fig. 9
6) Data transformation execution. Once the SQL is written, it is necessary to run dbt run to execute the data transformation.
7) Debugging. After running the DBT project, if the data transformation is successful, the data models (the tables) the database will change, with tables being added or removed. The method needs to examine the database determine if the transformation was fully successful. If not, the above steps must be repeated until the method meets the problem's 's requirements.
19

---

Published as a conference paper at ICLR 2025
Table 13: Google analyties traffic session examples, using answer format surface rewrite.
Question
Provide the number of sessions and percentage breakdown by channel for December 2020.
Reference Plan
1. First, read the document to understand how traffic is divided into 18 channel groups, primarily based on the metrics of source, medium, and campaign. 2. Extract all visits from the datahase for December, each visit having unique ID and session ID. Retrieve the source, medium, campaign each visit. 3. Rased on classification standards channel groups in the document, write conditional statements to determine which channel each set of datn helongs to, mainly using regular expressions. If the data source (source) contains any of the following: 'badoo', 'facebook'. 'Ib'. 'instagram,, , 'linkedin', 'pinterest', , 'tiktok'. "twiter", or 'whassapp' and the medium (medium) includes 'ep'. "ppe". starts with 'paid', then categorize as "Paid Social". 4. Calculate the number sessions percenlage for each channel based on the channel grouping.
Gold SQL (After rewriting)
VITI prep
ISELECT ve las. ant lus eaar CrreestteeanLLao veerbley "oLsession_id" AS session_id,
tey
1oMonE NaLLS ouee BY event tines camp) Isars perssti1l
.SRAY Mool Ferialia lue, ting
lne Stewent paranc) Marriee "mediux")
*SRAY AGGE LEI wwr.Cauue..sriiiie sueusteing walue Eag STceeent paraas) Wiranayy "campaign1) BY event_ tamp) [SArE OPrseTii1 comsaign
SOHOTE NULLS
FROM bnlequer--ieille co-iinlledddtt.....
WHEEE TABLE SUTFEX RETVERS SETVERR//RREEEE (22201R0 METRRREERSE OVOPP BY ueer_pocudi__e, session_id
SELECT CASE
WEN source "Cftieecc AND Cuediun ('Enot set3', (1oree)133 '13 THEN Teneettret"'
WEN (RSSESP CONTA (cturre, 'at "atibono) mesconlgrogle stopging) shop tyjetry (thay|stripeessteene jnstertt)) CONTAINS compaigh,
Ot REGEXP
(aa-d--2l "aahopl shesp.ng)
Stooplngg
VIEN REGEXP CONTP__COTAAIIIIII source, "baddu cang duck degpl sccesalapop le (yshos (yande))
VIEN REGEXP_ CONTAADTAIIIIO..
Facebook Ifos iastagrre [Linkedsn inintrrest (tkktsk bettter nntetsapp))
AND RBGEXP corcisStsedion "<C_aco-=jpp: (osid.e15)) THEn 'aid Social",
AN REGE CoMTADS inedior "L_cpp--pp -app poi0.41511 THEN 'Poid Video"
MIEN mediun DV ('dtppaay', 'tanter', 'espandable
"apo' TIEN 10isp
dR REGEO CONTAIiiceepaaso "I,elt [P-df-2] shopl hoopprg))**1" Ongante Shopping"
1E1ER AIRSASOSSSS bashoo
rediun ('seetal', "eactal Ierbberr' "socia (irkedie laittrrett (tikt bxitter lehatsapp')
rsecial
'sscial medial) TItN Organtc Secial"
WHEN REGERP CON COPPP_CCMAAASSSSSS.. soorss "aalymution disdinnpppp plus inc byoutse /vincl bet.ee vcccivi soutube)
Wey HEGEXP conrADS "baidu) bang bocckcch lapog lo (yalos) yondor 1I di acdium "orgonic" INEV' organaz Soarch
REGEXE CONTAISSI enarta, tenasl
maill) Aoore cONTAmIs mettun, "ematile-estl (e.ssll rall") TilK frads"
WIN
"heu
Win rediun:
Tim Seternal
VeE4 moliun "oudio" THES Audio" wes rodiue
minmoottee
Unaceigred"
HD As charne Larosping session, CONNTIDAS EOETCORAATTTO.. user oscckkid,, session_id)) As session_count
prep
totallessiens
sEi SiMcoasctin count As Toral count FRMM data
SELECT çd. chamee Loumeeorrrupnng Loroupans_ssaaaea sessken, pt ol.seaalon count, BOU00 www.gaassssoooooooooooooo count to.tot count 18t, 21 pecsontaassf groused
Statn Br waaamasoo__come cesc LIMIT
Original SQL
WITH pres
(SELECT lus.. 1rt va FRIH www.stieeett parrans) N-tRthkey 1i'sessioo_id11 sessios_ie,
ABRAY Aooly ESEiLecTvv lu www.sttingg.v...... veg los FRUM URRESTI spven_paaaas) N-ERE key "acurce")
gncent event value FagM (ReESTtevent parans) Mirarkey "aedtunt) AARAY (50 wew.ealleessstiinnee
tump! (sare
ssurcs,
TQURRE MIl L5 DROER BY event tirestwepl [SarE orrSeTia1] 45 rediue,
AARAY AGi(Esm (S SiTTaee-stiingg value PAOM IRGrSTrevent parans) Waraikey "compaggr cassaign
FROM "biequery bibilic data, gar, obfcscated, sarpla www.erccccccoo.. ovents
WHESE TABLE SUPPaa BETNEER AR00ER20 AND 20001233
OUUP BY user Bor___oud session_id
grospos_data 5F1 CASE
WITH sturce "Le (net) AND Lnediton IN ('Enat wet)', "Lroee) 11 THEN Droweet" win ancExp CONTANIS togg 'emme-rretarrk" THEN wwweer-eeeeeetaee
OR RESEXP CONTAINS (camaaign "nc ("g df 23] laaoos phopping e055 11
AND REGEXP CONTAINS inediu,, ecpp el gpc (poád 415'3 THE IPald Sropping
SEOPOE CONTAIISt la (yshoo
(peid.))")) new 'Peid Search!
NITH REGEOP COSTATE Theino facabook th inatenrra (iikeedi |ptntereenttteeee (riktak tuttter uabs
ConAAnsimmddin, lppp (paid.e15")
'Paid Sociat",
REGENP SooTA dillymotion
AD RECEXP CONTAINE inediu. "*1 Lecppee elppelpppcc. pald. +1211 THEN 'Poid Vidoo"
NTTH mediue IN Undisplay's 'banser', "espendable'
"pon) TIN 1os
shopping iohppity
OR RESEAP CINTATNG mny wil (P-df-r)
x15') Tess/Ggganti Shopping
WHEN REGENP. CONTAIRtssourre, badoc facobook 1o lastogrre oe Lockedle [psnterest] (E3Ktok buitter lehotsoop')
sodiue !'social',
network'. disncpplus
"social network (usaoo)
Modda?! Organic Social"
REGENP CONTAINS1
hysaccl youtune?"
08 REGESP CONTAANN teediuu,, "nL_edddooees"" THEN 'Organic Video
NTER REGESP CONTANSissurroR, "baid: juneetnnngp ee (goog tystes [yansic 1 mesium "argante" mim organti Search!
mewiile-eal je_ssitle
NYER endau lartsllate" TIEn state MEN maduue fraterral THOM Saterral"
WEEH aodiue "oudio" "Haiiuuiio WHEN medium "ses THev
houss 'OR COUNTIDSS CTNCSTTTTTTTTTTTTTTTT
"ebbile inttricattinn THEX Notifications" Unassigned
END www.pggooggeeeeooco session,
1d, session_1d))
FROM
prop GROUP BY channe prwpppooogggggnonoo.o...
GÃDER BY
OONTTOISTIICC CONGATTuser id, session_ (d)) DESC
LINIT
20

---

Published as a conference paper at ICLR 2025
Table 14: Citibike and taxi of NYC public data example, condition parameters surface rewrite.
Question
For the top 20 Citi Bike routes 2016, which route faster than yellow taxis and among those, which one has the longest average bike duration? Please provide the start station name of this route. The coordinates are rounded to three decimals.
Reference Plan
Focus 2016 data to determinc the 20 most popular bike routes based start and end stations, noting their latitude and number trips cach roure. Extract average duration for conveponning taxi routes using the same latitude and longitude for start and end points. 4. Calculate the average taxi duration for the matching routes. 5. Filter the resalts find the hike route where the avengge hike duration is shorter than the average taxi duration. 6. Order the results hy average hike duration in descending order and limit the output one record.
longitude. Calculate average bike duration and
Gold SQL (After rewriting)
WITH top2@route AS
SELECT
start_statee itaar__stationnamee end_station_name,
avg_taxi_duration
FROM
SELECT start_statin sirr__statio__na_e end_station nane,
ROUND(start stttttttttttttonnntt latitude, 3) AS ss lat, ROUND(start cestart_station statiin longitude, 3) AS ss long,
ROUND end station latitude, 3) AS es lat, ROUND station_ longitude, 3) AS es_tong,
(NG(tripduration) AS bike duration, COUNT(*) AS oike_trips
WHERE
EXTRACT(YEAR from starttime) = 2015 AND poort_station_iie name t= end and_statiin stationn caatiion_aeeeeeeee GROUP BY start station end station_nane, Lat, ss long, lat, long trips DESC LIMIT
ORDER
JOIN
SELECT
lat, ROUND(pickup longitude, 3) long, lat, RouND Longitude, AVG SECONDS (ECONDS(dropot datetime) SECONDSSSOS datetimee)) A5 avg taxi duration,
(pickup latitude,
latitude,
leng,
COUNT(+) AS taxi_trips FROM
bigquery new_york. elc_yellow_w__ triiptttiss___. 2015
pu_las lat, pu_long, do_lat, do__ long
bb
ON
a.ss_lat lat b.pu_lat a.ss_long b.pu_long AND a.es_long b_do_looy long
b. lat
SELECT start ita name FROM top2@route
WHERE bike duration taxi duration ORDER avg duration DESC LIMIT
Original SQL
SELECT
poort_station_naat end_station aig_bike, dukktidn avg_taxi_duratioon avg tare_fare FROM
SELECT
_amen, end station name, ROUND(start_staae
httisstatt_stttionnaaattttteeeet latitude,
ss_ lat, RoUND(start_statiinn tow.ittteiinnnniiiineeeeeeeeee longitude, ss_ so_to
ROUND (endd_ttation, _tuu_stttionlltttdd,, 3) AS es
wtweed_staaiooooonittteee 3) AS es long,
COUNT( (x) AS bike_trips
FROM li-iiraa-pull--ctt...eeeee. wwt..//wwwor...tiiikke.ekee witibikk_ttitiii
WHERE Start_station_name != end let__stttionnnen _astionnnmmmee__e
GROUP BY start_station_nan _aaee end_ statiio_naam,,, ss_lat, ss_long, es_lat, es_long
ORDER BY bike_ like_trips DESC LIMIT 100
JOIN ( SELECT
lrppickup_latttud, latitude, 3) AS pu_lat, htnblppicup_loogittuee 3) AS pu ROUND(dropoff) cittroooff_lattttee
(pUND(pickup_
pu_lloo_ long, 3) AS _at, cotooddrooof_onggituueee 3) AS do_ Long ( AS taxi_trips
FROM bigguery-public-data.. yor_y.k. tlc sicc_yellow_tee trips_2016" GROUP BY pu_lat, pu_ lo_gon,, do_ to_lat, do bo_longib ON
a. a.ss_Lat=b.pu lat=b.pu_lk lat AND a.es. a.es_lattes lat=bbdo..ddoo..o lat AND a.ss_ long=b.pu, long AND a.es long=b. do_ ORDER BY bike_trin trips DESC LIMIT 20;
21

---

Published as a conference paper at ICLR 2025
Table 15: Google patent example, advanced calculation surface rewrite.
Question
What is the publication number of US patent granted at January 2018, with the highest origi- nality score based on the diversity 4-digits IPC codes from its backward citations?
Reference Plan
Filter US Patents: Select publication numbers and upplication nunhers from the dataset, including records where the country grant
Additionally, only consider patents specitic kind code pattern (e.g.. %B2%). 2. Extract IPC Codes: Select the publication number and count the unique 4-digit IPC codes associated with each selected patent. 3. Identify Maximum IPC Code Count: Create subset of records that have the maximum count of specific 4-digit IPC code for each patent. Calculase IPC Occurrences Backward Citations: Join the filtered patents with their backward citations. For each backward citation, join with the sabset records the 4-digit IPC codes, counting patent. 5. Compute Originality
January 2018, excluding
grant
occurrences of TPC an originality score based on the diversity of the 4-digit IPC codes from the backward citations, using formula that considers the sun of squared occurrences of each IPC code, normalized by the total number of occurrences. 6. Select Highest Originality Score: Return Resali: Output the publication number of
each patent,
From the computed originality the palent
select the with the highest highest originality scure.
Gold SQL (After rewriting)
WITH pe AS
naapabiicacioonnnmmeer tt.epplicctiooonuuueer "patents-publlc-data pubciations
MIERE
country code lus' AND grent_date between 20180101 MND 20188131 publication
prant_date
interis_tabe terrie As
SELECT peuullicationnnueeeee SUBSTRIipc_ code, 0, 1pc4, SUBSTRI ipc_ code, 42) ipc4_count: FROY contt--pulic---ta.aettetto pub.entts,pbiiiiioon ti, (IcEES((ac) AS ipc_u GROUP BY Lnuber ipc4
chosen_ipct_
SELECT FROM taterinn_tbble
WHERE CONAAtinterinn wtw.tattee.ibiictiiooooooo interin tpp.e.ipcc..cooom IN CONCAT
ORDER pibiipppiiiiiiinnoooooo Lectttionnnubberr MoX(icc4_coun)) interia, table publication_nuater
ipc_counts AS
SELECT peauDliiatioonnuueee 13.1pc4, COUNT(t3. AS ipc_occurrences LEPT patents_sample
SELECT
(2.publlcationn_nueee AS Bitii__publicrrionnnmeee cutittonnuuuuulicctioon AS aecivard_citation
(MEST(citatioo) citation_u)
IN D2..citing_uuliccatie nunber pu. nunber
chosen, ipcd_View view
Liattooooouott t2.bachard_
GROUP ti.publication
Fax tok.oraatttttt AS
SELECT ion nurber;
POMER
POWERSSUHI
originatity
FROM ioc_ counts GROUP BY publication, number
HAVING SUMI www.o_ccccureeecee > ORDER BY arigtnatity DESC LIMIT
SELECT publication
Original SQL
WITH patents_sample AS SELECT
"l.publitationnnuueer clasppiicationnnuaeer FROM cestents-public-dattee patents. pub Lications"
MHERE
country_cade code 'US' IND grant_date beiveen 20180101 AND 20100131 grant_date !- publication,
interin_tabte tabte
SUBSTR(LpC_ SELECT uipe_u.coe, 4JJ 1pr4_com count
4) 1pc4,
(SUNT( (AAT(SUUSTR(ppc.u.coom
FROM
t1, (NNEST(1PC) 1pc_ GROUP BY t..publicationnnunee. ipcl
patents www.ttt.aaattts.uuill....oo
chosen_1ped view
SELECT MHERE CONCAT((nterii) htt.ieneennttilleaaeeeee
interin wablee.pullsattoon.nna
interin, tab ipc4_count)
SELECT COMEAT(pub1icat fpublicationn number, NAXiTpo4_comme coonn)) FROM interin, rable GROUP BY pubTication_numbo numbrr ORDER BY 19c4, ipc4_count DESC
interin_table
SELECT 11. 11.publicatton number, 13.1,c4, COINT(13.11p44) AS 1pc_occurrences aatents_ssnppee
FROM
LEFT JOIN
SELECT
x2. a2.publication ication citation_u, pubticatioo_nubbes baciward_cctattio
aitinggpublied
FROM parents-publie bublic dat.feaaaa.aattttt.... sabtents.ublicattion x2, (IMEST(citatiio) AS citation_u)
ON 12.citingg publication number cl.puhlication_nurae
LEFT JOIN chosen_ipe view ON [3.publiceetion_nmmme 12, (2.beceeerd_
pup Lication number, t3.5pc4
22

---

Published as a conference paper at ICLR 2025
Table 16: Google analytics page conversion rate example, advanced requirements semantic rewrite.
Question
Calculate the conversion rate from product list pages to product detail pages for all sessions at January 2nd, 2021.
Reference Plan
1. query the event data to retrieve all unique event names 2. Selocts events data. from the Google Analytics 4 (GA+) sample e- commerce dataset for the specitic date (20210102) 3. Filter include only events named page.viow, which represent page views. flatten nested event-params array extract values for gassession.d. ga.session.mumbes page.title, and pageJlcationn This allows analysis individual page views within each user's 's session. 5. Further processes the unnested event to classify pages based on URL depth and specific keywords into either Product Detail Pages (PDP) or Product Listing Pages (PLP), 6. Applies window functions to the categorized data calculate the previous and next pages for each session per facilitating analysis of nuvigation paths between puges. 7. Fillers sessions where the current page PLP and the page PDP. 8. Counts the number sessions tramsttioning from PLP to PDP and divides this by the total views PLP pages to calculate the conversion ralle.
Gold SQL (After rewriting)
base table
name, event date, event tieestoep, user psesdo 18, device, geo, trafric_suure,, event paraws, user properties
FROM
bigquery pitppppblllccccppppp ob woscaatted sample wowpe//ccoooooceeeettt...
nrTweru 30710101
('page_vie/
ted_events
SELECT
tiaestase nicreseconts, user preudo 16, MaXTEaSE with coley "po session, 1d THEM cupplue, tit value (00) visitlo, TaL.eesdoo nith cuvs vntuv (io) vasseturber, cokey "pape_ tistal c-vstue. wwwinn. valu AS location base. table.
date ssdate, tinestamp
pape tidle,
MAKLEATE win
TIN
uncur By aolestlevent_.oe paranss
carégorised
SCLECT
case:
AND LLOWERTSPLLT lpuge locolion, 7 1.1r (1rSer0)) CONTATNG (RR location, (B1000TTT000
"electroniss" :
Incttion llettteti arinoarre", "electranics' , "proglesredecinn , ''Lfestyte'' 'nest", 'newr2815-1opp" (taccessori es notare! brases"
"Liesstyee Teead covizeis logs" (ooteborta journals loff aeeee shopiby-brunn"', "smilligoods" "staiionery', "scarasles")
le timesis
(FFSF133)) dh
corfie ihopobyabrard iratleggons "saatineer'' sinSTRIMMY REVEREEEEERSIL SPLTTippp intatior, 1/1115Mt__ "11strt_SrssTTTB)
I. "P8P" 'e'] MD (PoNtRss incattrtr
WIth prrserca)))
(aare
Les aocorel 'brands',
oo sllctctinn' "drinwrre" "eloctraniss' "9200 office "Shop larrard" "sallrgioss "stttieery"" "vearubles' LOVERESPLLT location,
'Lifestyle' 'nest,, 'non lopo'
rotenesks (b12
('ac ('acressoris os". hntarel i, "hrwads's "y "comuseeal teeeeeenneneen "arinoaaee y releerrontes' , "groglesredeeiio (1 (Trfeyyyee' y 'nest', (neen2815-1agg")
"shppbyyerard" cie AS pape_cttegory raa unnested,neets
Euse "other"
SctecT
Bicrossconds! LEAD leaoc catouors, 11 OVER (PART-TIONSS user pical 10, ViRStED ORUER BY event Lamsston,, macroscconns) AS noxiL
ONDER
Krrerorrranssttnn AS SELECT PRpeLTsEts, CUNTIST AS transstions FROM rerkes MIDRE pape_categery "PLP" MD pept pags_titie
PoPain 6 page
piOILEItis, COUNTEST
TRON
NIDE
"PLP" GROUP
Curverrsonifaes 1
FROM IntaliiPvlnue 30TH Pi Praalasttnnci IN pupate Tifle T-dhge ritle
ricle.
elu
conserston_rate
inxconsension
page
DESC LENET 31
Original SQL
base_table
SELECI
aveet,name, event_dote, sinnt_ti user pper_poeuco_16 user_ie, device, geo, traffsc_souree event_sorars, user_propertiee
lssguery-pultc-satt.. gad wtt_oooosstte__ooeeeea
wiile
tobte_suffix BETWEEN'20210100 AMO'202101311 AND event_nare ('poge_vtew")
unrested events
SELECT
event, date ABdate, event_tinesta AS event, tinestono, wicroretonds, wewpoondddduueseeeeeeeeeeeee powuto__dd
MAXICASE WID cakey "Sa_session, number" THEN covalue, valuv B.D) AS visutturber. with cakey "pape 1999 ln #M6) page, fite_
"poje_locstion FROM hase table, saitsT perees)
WALCASE
www.stiing_.ooeee EMB) locationn
GROUP 2, iieerto_evenssccttorrree
SEIECT
cafe
WIEN ARRAY LWWTHISTTTTSP Locattin, Location, 121)) AND CONTADE WWW.SSRRARRAAAAEEEREEEEERRRRR WWW.RREEESLTTTTIRL.OOOOTOOT
DPPSETAF 1e7)
ITiigge locatine, [Kare OPPORTA413)
INC accessoriess asparel" "brando" compusecollectton:. 'drinivere", 1 "electronice' 'sosgle-redesign"
"Iifestyle'
tap" ennoknnkk nurnass' lofficel, "hopettystrand" snal logends" (stat innery' "eeersbless))
dn LONERISPLIT papp location,
(5FFE_0FF Di
( Paccessories " soppera"" brands" cempusecol lectines. fartneerr" V "electronces', 'anogl medeee gn "Tefestyle' Tnest", 100ee0015-- drapesy-brand""
journals
" "Nearables 1)
wir MET rivrAni SANCERjARRAY AnvrRsn(s (Tipage location, 1/ [SAFF (FP5 ABD OWFR(SR Epppage location,
1/"1 11SAAF__OFFEETT
"nsaaboonss journals" "office" "stosssy-brand", snalingoods" 'statiener'', "saarables 1 OR LONERCSPLIT location,
LISAP orFSeT13131
"brands', "cumusecclllcctioo 'drinware', "electronics' "gosgterredssign 'Lifestyte' 'nest', 201
restebonis--jurree to""""""",, "ooffice", "stopsy-bbrnn"" "aeat""ss "eas legoods' "stat tonery y Neearables THEN "PLP"
Bl0 paye paye_categrry FAor envnttedeevnts SELECT ISELECT ECT COUNTINI FROM unnestes events categorited WHRR page_ usjessts: PPP"') 7 (SELECT (OUNT()) FROM uneested Lovettseeeeee.eee.ee categor 1000)
23

---

Published as a conference paper at ICLR 2025
Table 17: GSOD and NYC public data example, merge related SQLs semantic rewrite.
Question
Get the average number of trips on rainy and non-rainy days in New York City during 2016, using data from the closest weather station located near the coordinates (-74.0060, 40.7128). Define "rainy day" as any day where the precipitation recorded is more than 0 millimeters.
Reference Plan
Which days were miny in 2016, and how can we ohtain weather information? The GHCN-D N-D datahase allows us to access weathr data from each weather station. 3. Given that the central coordinates of New York City are (-74.0060, 40.7128), we need select weather station represent the weather data for New York City. 4. Calculate the weather stations closest the center of Obaii precipitation data
precipitation classify the days in 2016 as either raiay or non-rainy. 7. Thee New York Ciibikk databuse stores daily bike rental data, which can grouped based on whether it was a rainy day and then averagdd. 8, Compare the differences in the average number of bike rentals on rainy days versus non-rainy days.
York City based
station.
Gold SQL (After rewriting)
SPI SPLECT ST (-rghointt(77..00n 40.7128) AS center, swon_stattions, 50 AS maxdist
SELECT
ST GeogPaint (1og ilong ittie) AS lec,
S1.DasEance(S7 GT__coooointttt longitude, latifudel, centes. AS dist_neters
FRON
ST_DWithannSTT__ ST.. (eT__GGeoPoonntnooonoeeo Longgitude latitude), parars.center, parans.aadisst.kt 2000)
params
nearest itasions AS I
SELECT *, BÁNK() (AmBR dist_meters ASE1 FROM distance, fistance_frax__eee fras_cent
neorest_nstations AS
station_ ids AS
SELECT 30, dist_neters from nearest_nstations ORDER BY dist_neters ASC LIMIT 58 1, bicysle__rntall
SELECT
COUNT(starttinel as nur_trios, EXTRACTIDATEfron starttine) as trip_date bigquery-pub wwtuurrypuulic-ttttoooo wew.litett...eee.orr... wwtpe/oooor_ctttttott wwwprooooctttiheccceteocc
trip_date
closest AS
station_ids,
id, (WLUEsstttiionntt wtt.__t......teeeeeeee as dist 2016
FROM ps.query-puuilcc-e
JOiN stat fuiiion_idd on wa. idrstal lon, ids. id GioUP BY station, ion.idi.. DIDER BY dist ASC LINIT
rainy_days
6) AS rainy date, COALESCE praX)))))
SELECT wx.date AS date, IF (xtvelement 'PRCP', WK. wwl.a/ee1,, NULL) AS prcp
FROM
bisqury- pat.u---athii-ddaaa.ahhc. dighendd 2016
MHERE
in (SELECT id FROM closest) GROUP date
SELECT
ROURDEAVG
rainy
FROM bicyc rentals J0IN
AS wx.date bk. date GROUP BY rairainy
Original SQL
SOL1: New York City Rainy Days WITH bicycle_rentals AS SELECT
CouNT(starttine) as num, nun_trips,
EXTRACT(DATE from starttine) as trip_der trip_date
FROM publirr-publie-datb... york hwt.yyoooookttttoe wwt.:/it....iibbiieeee.ee..e citsbike_trips GROUP BY trip_date
raysy_ddys AS ),
SELECT date, (NAX(pr<p) 5) AS rainy
FROM
SELECT datd ASdate, IF (wx.element PRCPP, wx.velue/1e, NULL) AS prcp FROM bigquery-public-de suublic-ata..hhen gec.taaggghnndgggnn__eeeeee AS wx wx.id "USW80094728" GROUP date
(OUND(AVG( bk. num trips)) num num_trip trips, rainy bicycle_rentals rentals
rainy_dan days
ON wx.e bk. GROUP rx.ny
-002: Chicago Nearest Weather Station WITH SEL ECT ST ST_GGSgPointtte G 41.88) AS center, 5a AS maxn stations, 58 maxdist_ja ), distance caancc_fra__ceeeee SELECT id, name, state, ST (eooooiintlaangittee latitude) AS lac,
AS
AS
ST ST_pistance(S1 (Asstance(ST_Geeoooiii (etggoonnnt latitude), parans. center) AS neters FROM Exgquery--ublic--ttt...ccc. wew.e_ddhhhnnddtiiiion
params
WHERE ST_ WWithin(ST GeogPoint ( ReogPoint(1ongituunn loogiitude, latitude),
parans.center, traans.maadiss soadiist_kk1000) kn+1000)
SELECT fram distance_ frrp__rrroorerrrrree
24

---

Published as a conference paper at ICLR 2025
B.3 SPIDER 2.0 DATABASE EXAMPLES
Google Analytics 4 serves as notable example of a Spider 2.0 database (see Fig. III. For each Google Analyties 4 property and linked Firebase project enabled for BigQuery export, dataset named ana lyt cicicsss ros.ppoperty..dd is created. Within the dataset, daily tables named events_yYYYYMDD generated when the Daily export option is enabled.
To accommodate latency, Google Analytics 4 updates these daily tables for up to three days with late-arriving events, ensuring proper timestamping. Each column in these tables represents specific event parameters, some of which are nested within RECORDS and may be repeated. For instance, the i teemparams RECORD stores custom item parameters unique to each implementation.
Google Analytics
NESTED RECORDS
app_info:
app_
key
version: STRING
value
smalytics_-propartg
country:
string_volii
metro: STRING
region: STRING
traffic sourte:
sonual camjaggidd STREMG
Flort value double,
STRING
Lomul__comaappnnnne STREMG
Columns
event_ id
event Heestamp
user_ ppor_proporties
pigr, view
page, JocatSce
hctpe://ooargbb. /ww.ar/t....oooooooooo
page, titla
traffic_souree
info stees
uperaeeters...
weltars....
Figure 11: Google analytics 4 database schema with nested record.
Fig. 12 showcases example of an enterprise-level real-world database environment from Spider 2.0, with multiple schemas to navigate through, each of them containing a variety of tables. highlights the complex structure types of Spider 2.0 databses, which exemplifies how our benchmark encompasses a broader and more intricate variety compared to others.
Bigquery Database Env.
MULTIPLE SCHEMAS & TABLES
Noaa global forecast, forecast_system:
NOAA_ GRAA_GF50P25
New_york:
goes
new, nee__orrk_33 yorkk311: 311:
abi, abb_lib 1____11l__r_______a
bobbrraiance
311 requests
abi 12 12_cnip
-nw__yorl hew.yyorrccitibikee
all_files
citibike stations
citibile_stations
citibike citibike_trips
noaa, goes|7:
-ne__yrrk_mp my collisions:
abi 11b radiance
hypd_av_collisee
abi _bi_12_cmip
new_york taxi trips:
taxi sea_zoon___e_e geon
noaa gsod:
tlc_fhv. wtt.ffhyssss__eeeee 2015
gsod1929
gsod2822
tle fhy fhv_trips twt.trips_21eeee
g5od193e gsod2623
tle green_ _twpe__triss_____________
trips__ii______ 2813
g5od1931
g5od2624
stations
Figure 12: Bigquery database environment with multiple schema and tables.
B.4 EXAMPLES OF EXTERNAL DOCUMENTS
In this section, we present the external documents utilized in Spider 2.0. The first is table that outlines the categorization method for traffic channels. The original document provided an HTML table, which present here in Fig. 13 The second document is the Google Page Category as shown in Fig. [4 which demonstrates how to classify a page into categories such as Product List Page and Product Detail Page.
25

---

Published as a conference paper at ICLR 2025
Oureel
Concitiors
redonectve
Source
("Est "Coome"!
Lappaign now contaire "craas iitvee-ieendd andestt-aeen includes deeand San, to narfomacc tax and start shepping.
shoppingse
Sxure satcher shopping tatinads, aaares, geagle dheptrr,, chipify, etay, shay, stripe, farrtt/drooarrdddem
2'1pla prepptrgl
anstttereiteen satckes Saag
oit
eefaid Spciaies
Saurce suticee reges List of social sites facasion instegren Loseedie staterest buttter ahat chatsppe wtetaeee-ttteeeeeeee romddaiiii io rifectian se
youtusel rettt lart sode sltee ieallomttinn coreespt netribuye tite, clme,, tectch
rego
ppaid.
Shopplage
Source natches turt shopping actes teldhebe,
shopphy striae teant midii Compatoo metches
retegentt Secia
Exurce petches rege List of secial sites teetto facebook, ineragerr Loreddts staterent, coese,, buttter whe whet ieddiiieiieeeeee
"oo
eórgenss Vadicc
Sxunce las sode abses ct to uau rama, tuttch y aintiaaatttiiiiitttitit matchea regor
organde
Rediun eaattl Referna
Rotice coaitly
Saacce euuttly motcles se 2ttd 2iidr ricciim coctly natcles ats
"aust Seodaaaaatttteee contares
"satificatier
Miraitigeed Onters
Figure 13: Channel group of Google Analytics, external document for bigquery example.
Product page category
### Refined Page Classification Criteria
#### Overview
To enhance our understanding of user engagement on our e-commerce platform, we differentiate between two types of pages based on the URL structure: Product Listing Pages (PLPs) and Product Detail Pages (PDPs). These classifications crucial for analyzing user behavior and improving site navigation effi- ciency.
#### Product Listing Pages (PLPs)
PLPs are identified by specific characteristics in the URL::
The URL must be divided into least five segments.
Neither the fourth the fifth segment contains '+' sign, ensuring these not detail views.
The fourth or fifth segment must contain one
following category names, indicating broader cate-
gory or collection page rather than a specific product focus:
Accessories Apparel Brands Campus Collection Drinkware Electronics Google Redesign Lifestyle Nest New 2015 Logo Notebooks & Journals Office Shop by Brand Small Goods
Stationery Wearables
#### Product Detail Pages (PDPs)
PDPs, which focus on individual products, are marked by:
A URL split into at least five segments, akin to PLPs.
The presence of '+' sign the last segment, common marker for detailed product pages.
The fourth fifth segment must also include of the specified category names, ensuring that the detail being viewed pertains to one of recognized product categories:
Accessorics Apparel Brands Campus Collection Drinkware Electronics Google Redesign
Lifestyle Nest New 2015 Logo Notebooks & Journals Office Shop by Brand Small Goods
Stationery Wearables
### Conclusion
This detailed classification approach enables more nuanced analysis of user pathways and interactions on our platform. By distinguishing between general browsing (PLPs) and targeted product interest (PDPs), we can tailor our content and design strategies to better meet the needs of our users, ultimately enhancing the shopping experience and improving business outcomes.
Figure 14: Page category document of Google analytics 4.
26

---

Published as a conference paper at ICLR 2025
B.5 EXAMPLES OF CONTEXT SETUP
Besides the context setup method for the DBT project mentioned in App B.2 we will outline the process for establishing the context in example about query database.
For the task, Can you provide the details of the top 5 longest bike share trips that started during the second half of 2017?. "query.py' serves our predefined interface for interaction between the model and the cloud database. This question is inherently ambiguous; without specifying the answer for- mat constraints, evaluating the responses becomes challenging. Therefore, we provide "result.esv', which defines the required answer format.
The deseription query.py # The query intertace
Bigquery cradentials
reault.cov Answer
data in Kovenber 2022
result.csv
trip_id, duratiionn ster date, start start_sttttionn statiinnnnnaee route, saike_numbrr, membrr nember_birth birth yearreeerrrr,oo aeaar,aee age, ag ass, nember, gaanber, regt on_mame
subsortber_tee pype,
For the examples presented in Tab. 1 outline the setup details for the Spider 2.0 example. Additionally, we provide answer examples for specific cases, which not only constrain the answer format but also enable the agent perform self-debugging using these examples.
The task instruction is Provide the number of sessions and percentage breakdown by channel for December 2020. We supply '20201 1.esv' and '202101.csv' as demonstration answers. We envision a real SQL writing scenario where the agent can first query November 2020 to check for consistency with '20201 1.csv'. If discrepancies arise, the agent the necessary corrections. Note that this is not a task requirement; is simply our belief that real SQL writing has such a need, and we will not mandate that the model does this. We believe this approach reflects a natural and realistic setting, although we only provide answer constraints for limited number of examples.
identify that their SQL is incorrect and make
READMR_nd quezy.py BASIC_SQLs
The task description The query incerface SQL examples of google analytics
bi.guerry_eredee
The predefined Bigquery credentials
202012.csv
file,
202101.csv
Answer formaz of data in January 2021 Answer format data Kovember
202011.csv
202011.csv
item_name, itttitttoooo itee__qantitty
Google necal ,103
Gocgle Clear Pen 4-Pack, 81
Google Mash Bag Red,79
Google Mini Kick B6
Google Light Pen R
Google laptop and Cell Phone Stickers, Google Fen Neon Coral,?
Google Metallic Notebook 7 Gacgle Fen 5ilas,5 Google Pen Red,5
The query interface of Bigquery "query-py" is
import
inport pandaa as pd
frem google.cloud import bigquery
query_data (sq)) qsql_yuary, is_save, sovo_path
Queries data from Bigquery based on the provided SQL query and handles the seeult.
Args:
sqi_query (stz) SQL query string executed. save (bool): If True, saves the query results csv file the specified save_path. False, prints reaults to the consale.
27

---

Published as a conference paper at ICLR 2025
patc (stc): the file path where results saved is_eave True. Defaulta to resul..ccv".
rs.enviren APPLICATIOLLCREDEENITII client bigguery.Cliee !)
"biequery_crrdentii..
quary_jab client.quary ingllyagly
try:
results query_jot. tt__datfframe
if reaults.emptyy
primt No data Iound for the specifled query.") else:
if ss_save
ts.to (save_ path, index=False)
print (T"eellt saved to (save_pathi")
print (results)
except Exception as e: print ("Error cccurred while fetching data:
1f m_man__":
your SQL query in sql_query variable interact database, #example SQI query zelated to this Lask La provided below que_y SELECT
FROM
'bigquezy public Mata.gal_oda wtt..eobfuerattteeaaaeeeeee.e.ee eeeetttsmmeee.eee.sss......
TADLE_SUFFIX BETNEEN 202211201' AND 202012311'
LIXIT
..*
query_datt data (sq) query, 1s_aave=True, rave_path="reau reault.esv))
B.6 THE DIFFERENCE IN TASK INSTRUCTIONS BETWEEN SPIDER 2.0 AND SPIDER 2.0-LITE.
During the annotation process, we found that unambiguity and naluralness are two mutually exclu- sive concepts. Therefore, in Spider 2.0-Lite, we emphasize unambiguity, while in Spider 2.0, we emphasize naturalness. The two instructional approaches restore the possible question forms that may arise in real-world text-to-SQL workflows.
Example 1:
Spider 2.0: The company management has requested a detailed report on the year-to-date perfor- mance of the Magnificent 7 stocks.
Spider 2.0-lite: Please show the price change rate of the Magnificent 7 stocks from the beginming of this year today.
Example
Can you provide the details of the top 5 longest bike share trips that started during the second half of 2017?
Spider 2.0:
Spider 2.0-lite: Can you provide the details of the top 5 longest bike share trips that started during the second half of 2017, including the trip ID, duration in seconds, start date, start station name, route (start station end station), bike number, subscriber type, member's birth year, age, age classification, gender, and the region name the start station?
Example 3:
Spider 2.0: What's the no-tip percentage for NYC yellow taxi trips in each borough from January 1-7, 2016, considering valid trips with least passenger and non-negative amounts?
Spider 2.0-lite: For NYC yellow taxi trips between January 1-7, 2016, could you tell me the percent- age of no tips in each borough. Ensure trips where the dropoff occurs after the pickup, the passenger count is greater than 0, and trip distance, tip, tolls, MTA tax, fare, and total amount are non-negative.
28

---

Published as a conference paper at ICLR 2025
B.7 SQL DIALECT DOCUMENTS COLLECTION
The core SQL dialects lies in different advanced functions and subtle syntax variations across SQL versions. To support retrieval-augie augmented agent frameworks, crawled and pre-processed the function documents for different database systems from their official websites. The detailed statistics of the crawled web pages and parsed calegories/functions that, functions belonging to the same category (e.g.. aggregate functions like COUNTIF F and TRRINGAAGG) may be introduced in the same paragraph in some web pages. In this case, re-use the description on this shared function category for different concrete functions.
presented in Tab. T8 Note
Table 18: Statistics of different database systems on Spider 2.0. Notice that, means there is no well-defined function list in the official web pages for Postgres, thus we merely use the summarized document for each function category.
Database
Documentation Website
Page
# Category
Function
BigQuery
34
34
390
Standard
opezatora
Snowflake
719
30
Postgres
/ttp:////ww..oooooaaaaaaaaaaaaaa functions.
30
30
30
Clickhouse
https://ellaalooss..c///bbbee Carence funct ions
226
6
226
SQLite
httpp::/www..slttte.rr/////
6
6
147
DuckDB
https///Sackb..cr//// http:///wr...../////////////////
24
24
513
Total
1039
130
2025
B.7.1 PROCESSED FUNCTIONS FOR DIFFERENT DATABASE SYSTEMS
In this section, demonstrate examples of parsed documents for different database systems. These pre-parsed chunks can be retrieved and inserted into the prompt to compensate agents for their deficiencies in SQL dialect knowledge.
Document of BigQuery Functions
Blaabase="BigQuere function="ST INTERSECTS" "TTTERSEF""TTIITEERSSSSTT"""""" category=" "gogoorr-yyoormphyy-iinttttn""""
## ST
ST J geographyy
** **Decription
Returns TRRE the point intersection of geography "nnd ggograppyy22 2 non-empty. Thus, this function returns TRUE if there at one point that appears both input GEOGRAPHY
ST.NTERSSCTS returns
'TRUE it implies that ST.DIIII returns FALSE
**etuum type**
BOOL
Document of Postgres Functions
29

---

Published as a conference paper at ICLR 2025
database="Postgrrs" wrrrrioocccrrrosmmmmmmmuuurrrrru
For enum types (described in Section 8.7), there several functions that allow cleaner programming without hard-coding partieular values of enum type. These listed in Table 9.35. The examples assume an enum type created as:
CREATE TYPE rainbow AS ENUM ("red", "orange", "yellow", "green", "blue", "purple");
Table 9.35. Enum Support Functions
Function
Description
Example(s)
enum ffusfi anyenum + anyenum Returns the first value of the input enum type. enum houm.fistauul::raibooy
red
last (anyenum) anyenum Returns the last value the input enum type. enum (anliauul:::ainbow - purple
enum enum_range anyenum - anyarry
Returns all values of the input type in ordered array. www.angggeelloo.eeeee..ee
enum @oum_range((nn rangee (rnlrrboww):))
@oum_range anyenum, anyenum anyarray
Returns the range hetween two given cnum values, as an ordered array. values must he from same enum type. the first paramcter is null, the result will start with the first value of the cnum type. If the second parameter is null. the result will end with the last value of the enum type.
enum_rangel "range orange "raange":rainbo"" ::rainbow, enum-range(NULL. "green ":rrainbow) -> wll.orange.yllow...oo enum_rango( ooaage"":""""ooo ::raibow,, NULL) - orange.yellow.gree wrw.ee.buu..purrr....
"green" :::raibbow)
granggeyellow.gree.
Notice that except for the two-argument form of enum range, these functions disregard the specific value passed to them; they care only about its declared data type. Either null or specific value of the type can be passed, with the result. is more common apply these functions a table column function argument than to hardwired type name as used the examples.
Document of Snowflake Functions
30

---

Published as a conference paper at ICLR 2025
Batabase="Snofflake function="ATAN22" cieegoy-""mnnrrc-fuutttoss
Categories: Numeric functions (Trigonometric)
## ATAN2
Computes the inverse tangent (are tangent) of the ratio of its two arguments. For example, x > O, then the expression ATAN2(y, x) is equivalent to ATAN(y/x).
The arc tangent is the angle between:
The X axis.
The ray from the point (0,0) to the point (X, Y) (where X and are both 0).
See also: ATAN
## Syntax
ATAN2( <y> <X>)
Copy Note that the first parameter is the Y coordinate, not the X coordinate.
## Arguments
This parameter is the Y coordinate of the point at the end of the ray. The data type is DOUBLE.
This parameter is the X coordinate of the point at the end of the ray. The data type is DOUBLE.
## Returns
The data type of the returned value is DOUBLE. The returned value is in radians, degrees. The returned value is number the interval [-pi, pil.
## Usage notes
If the data type of an argument a numeric data type other than DOUBLE, then the value is converted to DOUBLE.
the data type of argument is string, the value is converted DOUBLE if possible. the data type of an argument is any other data type, the function returns an error. either argument is NULL, the returned value NULL.
## Examples
SELECT ATAN2(5, 5);
ATAN2(S, 5)
0.7853981634
Document of DuckDB Functions
Satabase="DuckDDi duneei""="dateeddff""" category==date--funtiio
Function: pate,iff(part, startdate, enddate)
The numher of partition boundaries hetween the dates. Alias of date di f.f.
Description: The number of partition boundaries between the dates.
Example: datedi month' DATE 11992-09-15' DATE (1992-11-14"
Result:
Alias: date diff.
Document of SQLite Functions
31

---

SELECT ISONHas( ("a": "hello", [-100, 200.0, 300] 'b')= Published as a conference paper at ICLR 2025
database="SQLite" function="grouu.coom (x, Y) categoy-=agrregleennnniooos
Function: group_concat(X.Y)
Usage: group concat(X) group concat(X. string agg(X.Y)
Descritpion: The proup.concatly function returns string which the concatenation of non-NULL values of X. If parameter present then it is used as the separator between instances of X. A comma (".") is used as the separator if is omitted.
The string agg((. function is alias for group_concat(X.) Y). String-agg( is compatible with Post- greSQL and Ser-Srrver and group.concat() is compatible with MySQL.
The order of the concatenated elements is arbitrary unless ORDER BY argument included immediately after last parameter.
Document of Clickhouse Functions
latabase="Clickhousse" "unction="JSONNee calegory="jsonntuuctioss
## JSONHas
the value exists in the JSON document, will be returned. If the value does exist, 0 will he returned.
### Syntax
JSONHas(json indices or keys]...)
### Parameters
json
JSON string parse. String
indices keys
list zero more arguments, each of which can either string or integer. String, Int
indices.or .o.sss.ooeey.......... keys
type:
String access object member by key.
Positive integer access member/key from the beginning.
Negative integer access n-th member/key from the end.
### Returned value
Returns if the value exists in json. otherwise 0. Ulnt8.
### Examples Query:
SELECT ISONHas(" ("a": "hella", "b": [-100, 200.0, 300] ; 'b', =)
The minimum index of the element is 1. Thus the element does not exist. You may use integers to access both ISON arrays and JSON objects. For example:
SELECT ISONExtract "ettracteey(("""""" ("a": "hello", "b": [-100, 200.0, 1)=
SELECT ESONExxracttey(" ("a"" "hello", ; "b": [-100, 200.0. (2)
SELECT ISONExtractKey(" ("a": "hello", "b": [-100, 200.0.
SELECT COONExtractKe"("""" ("a": "hello", "b": [-100, 200.0, 300]] -2))=
SELECT JSONExtractStringg (""a" "hello", "b": [-100, 200.0.
300]]". 1) "hello"
B.8 EXTEND DATASET STATISTIC
Database scope. As shown in Fig. 15 the databases utilized in Spider 2.0 encompass wide array of domains and real-world scenarios, providing a notable degree of diversity.
Data types. As depicted in Fig. 16 the Spider 2.0 database encompasses wide variety of data types, including text-based data types including STRING and BOOLEAN, number-based like INTEGER and FT FIOAT, structured data STRUCT, JSON, time-related data such TIMESTAMP, and spatial
32

---

Published as a conference paper at ICLR 2025
data like GEOGRAPHY in google bigquery datasets. The diversity and breadth of these data types underscore the extensive complexity and wide-ranging nature of our benchmark database. This variability is reflected in the SQL dialects and the intricacies of data handling. thereby presenting significant challenges for SQL generation.
Keywords. As shown in Fig. 17 due the complexity of the SQL in the Spider 2.0 dataset and its coverage of various dialects, it contains more SQL keywords than any previous datasets.
Number of Tables. As shown in Fig. 17 the databases in Spider 2.0 contain more tables than previous datasets. Additionally, each SQL query in Spider 2.0 requires joining more tables on average.
Data Volume. The databases used in Spider 2.0 contain significantly larger data volumes. com- parison, each database in WikiSQL has only 17 rows, Spider 1.0 contains 2K rows, KaggleDBQA has 280K rows, and BIRD has 549K rows. In contrast, the average database in Spider 2.0 has 5273.42M rows, with many databases reaching TB-level sizes.
TEXT STRUCTIONS
16
SCIDNCE
15.1%
MEGEINE
2003
ORLER
MAShebiE NOLODY ENVIRUNIEN
14.T%
InES
Figure 16: Data types of Spider 2.0 database.
Figure 15: Domain distribution of Spider 2.0 database.
WINSSL
Spider
Kago NDBOA
Spider 2.0
50-
40
30
30
20-
19.71
10
6.89
11.70
0.90
1.00
1.00
3.04
0.00
No. Thble/DB
No. Keywordds
No. JoiNas 50L
Figure 17: comparative statistical analysis SQL queries Spider 2.0 and previous text-to- SQL benchmarks.
C DETAILS OF EXPERIMENTS
C.I SPIDER-AGENT FRAMEWORK
Inspired by React Yao et al. 2022) and Intercode (Yang et al. 2023) we developed an agent frame- work called Spider-Agent, which primarily focused database- related coding tasks and projects.
33

---

Published as a conference paper at ICLR 2025
Spider 1.0
Spider 2.0
SELECT T2.name, 12.buiget
instructor 101u departuent
MITI cohorts As SELECT
12.departreet__c
User_te Id, DaTE(crested_ it) as order_date,
OATE(RATE, (FURC( (creaateed
NAVING svlar).
OVER(PARTETION usen cconmorce
created ac), month)) cohort
(SELECT axg(salary) FROM Instructon)
bigquery public-
ANTE(created
KaggleDBQA
activity
calculate
SELECT
SELECT school district
User id, cohort,
DATE_ (ATE__IFFForddrnco cohort, north) AS nonth_since sontt_cincc_Frrtttovve Frrer
FINREY FED 17 12 ON Ti.stato 12.state "Misconsin"
FROM FTNaEV,
T2, ststtt__cook WHERE
10TH
),
sint OATE(cuhort) "20z0-un-e1"
FROM cohoris
counting the mumber of untiue users esch colort new users A5 SELECT
oRDER: BY T3. feed BESE LIMET 1
cohort, sonth_since, farstcc_frrttooddrr CONTTOISTENCT uson_id) nev_vsor activity
SEDE
first,
5e count (v.ot(v.po)dddd) 'Total Votes', w.postid as (Post L1nk]
calculate total customer
cohort_usere users AS
SELECT
w.postid
cohort, sonth_since, worth_stnce_Firsstoodde Earst_onder,, noKLuser,
PoxtTynetd
OVER(PMIT1110N By cchort ORDER BY nor narth,since, order)) As cohort, user FROM nea_users
FI8ST VALUE (nea, user)
entragarnc DeTeron(eonth,, GTTONTE())
CROUP DV v.powtid
users parcentage
Votes'
conort, http__stcc__Sssstoorrre
FROM WHERE nonth, since, First_ordon Excluce Jonuory data (month
coachin FROM noaches MErE 1ptn unxx unx! AMD
1-0
MDD Losses I- AND toochED IN
(SELECT coschID FROM coaches WHERE 1gI0= WBA cRouP uv coachin
oroea BY
catort,
hAvanG count(
sonth since. www.hssseeseeeesoooooeoeeeo first.crderr order;
post Josses LTMIT 1;
Figure 18: comparison of SQL examples selected based the median token length for Spider 2.0 and previous text-to-SQL benchmarks. Spider 2.0 examples were selected with token counts at the median, while examples from the other four datasets were selected from the original papers.
34

---

Published as a conference paper at ICLR 2025
The framework allows for multi-turn interactions with the database via command-lin line interfaces til the final answer is obtained. To ensure the agent focuses solely on interacting with the database, shown in Tab. I9, we designed set of specialized actions. We use temperature of 1.0 and top-p of 0.9 and truncate from the beginning of the input still exceeding the max tokens limit required by the models.
The model automatically terminates it outputs the same result three times in a row or any action takes longer than 120 seconds. The prompts used in the experiments are provided in App We heuristically request the agents to complete the tasks within max step limit of 30, which is enough for most tasks.
Table 19: The action space of Spider-Agent, agent baseline method for Spider 2.0.
Action
Description
BASH
Exccutes shell commands, such as checking information, running code, or executing DBT commands.
CreateFile
Creates file with specified content.
RditFile
Edits overwrites the content of existing file.
ExecuteSOL
Exccutes SQL query on
with an option to print save the results.
Get
Retrieves all table names and schemas from a specified BigQuery/Snowflak dataset.
GatTooTnfo
specific table BigQuery/Snow mnooy///.w.lake Samples a specified number rows from a BigQuery/Snowfiak table and saves them as JSON.
Retrieves detailed column information
SampleRowa
PAIL
Agent decides the task infeasible.
Termi
Agent decides the task finished.
The number of joins does not have a direct correlation with model performance. For Tab. 19 there no clear correlation observed between performance and the number of joins. We speculate that this due to the fact that during the SQL annotation process, we ensured that all examples were quite complex, which made performance independent of the number of tables in SQL involved.
Action analysis of Spider-Agent. We analyze the results of Spider-Agent. For all correctly com- pleted tasks, the agent needed an average of 9.0 steps (with maximum of 17 steps and minimum of 6 steps) within the defined action space to achieve the desired result. We also analyze the fre- quency with which actions invoked each turn by Spider-Agent, shown in Fig. 20
253
Figure 19: The effect of the number of Join on
performance.
Figure 20: The frequency with which actions are invoked at each turn by Spider-Agent w/ ol- Preview for task instances that it solved on the Spider 2.0 (286 trajectories).
C.2 DETAILS OF SPIDER 2.0-LITE EXPERIMENTS
Details of baseline methods. LLM-based text-to-SQL methods have demonstrated exceptional zero-shot reasoning and domain generalization capabilities. DIN-SQL (Pourreza & Rafiei 2024)
35

---

Published as a conference paper at ICLR 2025
employs task decomposition and adaptive prompting strategies tailored to task complexity. DAIL- SQL (Gao et al. 2024) achieves the best EX on Spider through elaborately designed prompt opti- mizations and in-context learning. CHESS (Talaci et al., 2024) integrates schema filtering based on entity and context retrieval, and SQL. revision, achieving the best EX on BIRD. CodeS (Li et al. (0244) fine-tunes open-source code generation models extensive text-to-SQI. corpora, obtaining performance comparable to methods that are based on prompting LLMs.
The treatment for sampled cell values. Spider 2.0-lite contains various complex data types, such as nested structures (RECORTED or array (REPRATED) in BigQuery. if we only provide data type indicators, it is challenging for models to correctly process these types by appropriate SQL functions. Therefore, we provide sampled cell values (in markdown format) from each table in the prompt for all evaluated methods.
The treatment for value linking. During evaluation, we do not perform value linking (entity re- trieval CHESS, value retriever CodeS) when solving instances from BigQuery, as the API cost of retrieving all values from a terabyte-scale cloud database is prohibitively expensive. Since value linking crucial for identifying entities filter conditions, its omission may hinder performance. Exploring cost-efficient methods for value linking or alternative approaches is important direction for future work.
LLM. Given the extensive length of prompts after serializing large-scale schemas, default to using GPT-4o, which supports 128K context window, as the base LLM. Specifically for CHESS, use GPT-3.5-turbo for column filtering to reduce costs.
Temperature. For all methods, we set the temperature of the LLM to 0 to ensure the reprodueibility of the results.
Database Schema
Reference Plan
GRATE TABLE "stoedhare.crigg criipe"
"erdp set STaEes coepan "Manerie bike trip".
plan that useful for guiding the generation of components of complete query:
"start
coress" "Start staettupp tria",
"duration attutts" 1ul6e comeast "Tame of trip in sdratec"
*iccie Dop Scurcasm respecilve sourts
by selecting
CLEATE TABLE "aiteeheenrataateee""
Trp Deem
"crandee cat Datied, ST100G "cranuc" sreres
W thin the hip dos. crsure popper data sees by saldy conve ding socessary fickss to intgers and
"ucdress" STIDIG.
Oracle SQL Functions
Sampled cell values
eed besshha has
[stat_sttiion duerkonmaaeee
M Potentially useful special functions with their usage: +/
1420370 2205-33 23 33.4
CAST
CAST cistrerree typeuere Horre chusee
end settar statzes
home
piddress
"Decrppasst
castspnza eaery to cos thathh tpe osressionscced converted
Paese Park) cleend 1355 Ringbtrry
Wher uaing query can Gasgardge unable pertorntte cvet you wertte from
Fow shot Example
"6 "Exorizles
The tolliing paary woult
"brae" aal "Talee" for other non- 'MULL' Vakk,, and
mair tee mavoured in yean??
tiiee Texur, COUNTIC
Task Instruction with Specific SQL Dialect
ROUN www.iiigguttttttl..... Aug Aepuradios
Genertte Gasls BigQuery SOL statmmanto snower tolloainng pander, ereatng tht RigQuers
GROUP ge Urow Tanuw
Texure
External Knowledge
Question
bemencc understand the tystcal pattoraa bike syptema Theae systers oten see giter néterhip during perk cerkkciiiuuiiggg 1au5
richh spiaan in the Ausin Weshore syste tas oness bia ouraton? war:
An example of prompt organization given by DAIL-SQL. prompt components that tai- lored to Spider 2.0-lite are highlighted. All these prompt components are similarly implemented for all other evaluated bascline methods including DIN-SQL, CHESS and CodeS.
Figure 21:
36

---

Published as a conference paper at ICLR 2025
C.3 DETAILS OF ERROR ANALYSIS
We summarize the descriptions for all error categories in Fig. 22
Error Type
Subcalagoery
Description
Example Predicted SQL
Esample of Gold SQL
Soase the requssed tabes are incurtot or da net eeeter ecceativ belse en repaetedum por winee ground trth teilee ere miowing
Wrnne Table
contoertt wrb pelfi
Wmming Linking
Wronng
requeend oohomas
MITA
QUACE cariteed www.eeeeeeeeeeeeeeeeoooooooo.
Incomect naissing use of dalect specie
haamet D elect
ferediore for shing mertpsleiin lng, CIONCCT))
Functicn Usage
tate procestigg lt.g DATZ TRUNG), 2e
geograglis le DISTAXCE),
Adhrmooe dats calos abore mork rntended eggregatian g.
Iaurmel Data
grorging
Caladadea
BLKCITSON NTTLE)
application (a CORN, STADEV).
The poid SUL incohees nested guotes, intermection
boorect Plansig
read proc askng thoouh CIEs, or operatons mvrging sab query mests. model ether fais moogtiza cilscs clanors..
Patee 1tg. 7031
Hat
Jeta tonr
The JIN condrton wdechs the tabos cceclemes.
Canditos
The Eltzing trcometly
clase
Misandrrtandin Drienal Kazwledge
mottergns boowledge
The iepe proppe ewoech the LLM'ss masimaa lagga, casite aton makkng the anove tnaooesthl
Excasive Procept
Syste:
the presetts
The percutad qaary custain invdlid spstas
Figure 22: Descriptions and examples for all error categories.
C.4
OTHER ANALYSIS
Current LLM-bbasd methods exhibit limited capability in addressing enterprise text-to-SQL tasks. Tab. 5 illustrates that Spider 2.0-lite and Spider 2.0-snow present significant challenges. The highest performing method, DAIL-SQL- + GPT-4o, achieves an EX of only 5.68%, which is markedly lower compared to its score of 86.6% Spider 1.0 and 57.4% BIRD datasets. With ef- ficiently filtering the minimal sufficient schema, CHESS + GPT-4o is able to tackle more instances than DIN-SQL. Despite being extensively fine-tuned, SFT CodeS-15B is far from solving Spider 2.0-lite, with an EX score of only 0.73%, which further reveals the significant complexity gap be- corosuoooorruu......
tween Spider 2.0-lite and the current text-to-SQL corpus. For Spider 2.0-snow, even the best method achieves only 2.20% EX, highlighting the increased challenge due to SQL dialect differences.
LLM-agent frameworks face challenges handling different SQL dialects. As shown in Tab. 20 we analyze the examples in Spider 2.0 based on database type. The results indicate that examples of the Snowflake type in Spider 2.0 are the most challenging. To assess the impact of SQL di- alects performance, we randomly select 180 examples and hosted them on both BigQuery and Snowflake, using the same set of questions. We find that performance on BigQuery is 12.78%. while on Snowflake it is 6.6%. highlighting that subtle differences in SQL dialect syntax can lead to variations in SQL generation performance.
C.5 EXPERIMENTS COST
We summarize the average of API calls for each instance across different methods in Tab. 21
37

---

Published as a conference paper at ICLR 2025
Q Question:
Please find out what percentage of the page on Januaryyy 2021, were for type pages.
Gold SQL:
unnestel, events, catageriaze As
External K nowwledge:
categorizie Leingin rage ritles into rars and rus
NLTH tabse_talle
SELECE
Relined Page Clasifiiation Ceitecia
Overviex
SELECT
relevant dates
CASE wen
ARRAY, LEMITHESSLEECCNNEE Jocation, 1233
To enluaaee our codestariing of swe engi gennent on womemeeee pladiams we diffemetade betwees two types of pape bard on the URL structure Product Eadiag
prent_nnne,
COMIAESS SJASTREERRYY SOMESSECPLLTTPEE (eppsr plccttoooo
Pages (MUP and Product Desall Pages (h0es) lheee claeticatins are crucall for ynagytng sse betonor and imprening site sargrtion effireeccy.
'7"33 [18F6 cepst ((9)]. '+") Mn
TROr
(SPURRSSPTTTPPP location..
Product Lating Page (PLPs)
bigguery public-
(FPSRT(4]])
idectiteed spaciti dusacteriatis UBL.
Bccossocilael, [apearel): ('bradts "saarables
weaerer
lilagaa, falloniigg calagary
events_
DR. LOWFACSK (page Jararton,
wHE
["][S (4FSFT(3)) tpparell, spp
indioatioiia looader calagary callaction page specific product
Lable "22111222"
Accessntes
unsestedd evencs A5 unisste event paran stors
Lon, """Msee_ (4') and (r)s(()))))))
NeTEcoNTanss SUSTTRERRRa SEVERSE ITEpage Jocat
44F
Broads
Carpps Collectoz
Apgami
Offiice
2013 Logp
kays
(LSMEN(FLLL
Trinskaoom
Shop Brand
Elactronics os
Small Goonds
( accesssrllss' "ippoee"" "tradds' cea (weorablesss
Google Bedenign
Statonary
MARECASE kukey
UNER:S (otpe "/" (SSStt_ (5)3)
Lebestyle
title c.value valle 1MD) A5
"accessories' 'apporel,, 'brands' youry (storalles))
PDPs, whith focus on indiv products am marlord by:
Product Detail Fages (PDPs)
MAX/CAGE c.esy
pape tiple pege_sitle_arrjeseee
TTh URL musi be divided lnto at least live segunnts.
Thhe pr presnce of 'n' sign the first segueet, a contraoo caaleer datalld prodatt pages
c.valus atriig (ND)
FRO
Rape tacation!
usrested everts
The fourth fitt segnent mosd elso inderde ane of specified category being stewed petaatss of the mooquizzd
FRar
names creuzing
SRIECT (SRIETT couNT(Y) FRom
product categortes
UNIET (eraat)
usnested exents canegoelioe sesee)
CROUP BY 2,4,2
Baps ritla adjutred seall PSFEECT cOUN()) FRo
events canagoni
Weembles
Predicted SQL:
Gold Result:
SELECT
(COUNTPP
event dinersiass hostraes
17.49112426035.
I' ''Accsssor ans Appural Itrands compas Gillccion/orareeee fornbwerr wwwerrelllacionieee lboogl
hedessihh Loftlll Nes Yow 2035 Loplloocbbokk & zourall 10ffice by trand (bell ossis ossisl sttlooorrryyyyooooo (waratlle)
Predicted Result:
A5 pep_perconaage
adp_percentage
1938
biguery public gat obfuscated Jo0 somple,.ee....eeeeeeeeeee cow.rrco.ccoooooooooooooo... events 2001099
event dote '20030822)
An example Misunderstanding External Knowledge. The error in predicted SQL stems from the failure to correctly interpret the external knowledge provided for classifying PDP and PIP pages. While the predicted SQL. uses a simple pattern-matching approach with regular expressions, it overlooks key aspects of the classification rules, such as the specific URL structure and the position of a 4 sign, which indicates the misunderstanding when trying to leverage the external knowledge.
Figure 23:
p on Spider2.0-lite
12.1
Table 20: Model performance with different
DB types.
10.6
(X)
Task Subset
#Examples
SR (1)
8.8
Spider 2.0
100.00%
17.0%
w/ Bigquery 33.86%
24.07%
5.8
w/ Snowflake
31.33%
7.14%
DAIL-SQL GPT-4o
w/ SQLite
21.36%
20.74%
4.8
12
5
10
20
w/ DuckDB
10.76%
17.69%
w/ Postgres 1.58%
12.82%
n
w/ Clickhouse 1.11%
57.14%
Figure 24: pass @ (n) results.
38

---

Published as a conference paper at ICLR 2025
Table 21: Average cost per instance across all methods.
Method
Avg. Cost (J)
Spider + ol ol-preview
0.75 $
Spider-Agent + GPT-4-Turbo
0.58 $
Spider-Agent + GPT-4o
0.32 $
DIN-SQL GPT-4o
0.14 $
DAIL-SQL GPT-4o
0.09 $
DAIL-SQL -preeview
0.32 $
CHESS + GPT-4o
0.43 $
SFT CodeS-15B
0.00 $
C.6 CASE STUDY OF SPIDER-AGENT
In this section, we present the success and fail cases on Spider 2.0 using the op-review and Spider- Agent framework.
Fig. 25 shows a success case of google patents data. The Spider-Agent receives feedback from executed queries, allowing to iteratively refine SQL queries until achieves bug-free result. Automated iterative debugging is one of the key advantages of Spider-Agen,
Fig. 26 shows success case where Spider-Agent successfully solves DBT project example. first explores the codebase and database, locates where the code needs to be added, writes two SQL scripts, debugs, and runs the DBT command, ultimately completing the data transformation successfully.
Fig. 27 shows failure case where the agent fails explore the codebase and locate key information about monthly recurring revenue, resulting in an incorrect SQL. query.
Fig. 28 presents a failure case where the model reads the document but fails transform the complex document strings into an accurate SQL query.
Fig. 29 illustrates failure where the model cannot comprehend the database format with nested columns, leading to multiple unsuccessful attempts at writing the correct code.
39

---

activit.sll: with date iitecs as Published as a conference paper at ICLR 2025
Success case about patents analysis
### 17 Steps
Action Bash(code="II")
Action 1: Bash(ccode="cat README.md")
Action 2: W...GE.TTBLLESdaaabbaeee haa.roossssnaaaeeeetttttttteee public---aaaiii"a"" dataset.name="ppatee
path=="daaastt_mtttaaaaaa
Action BQ GET TABLE INFO(database name="patents-pubiiddaaa""" dataset name="patents"
ble= "lc="ubbicaatios"" save auth="publication info.csv")
Action 4: BQ EXEC (sL( query=.... is save=False)
Observation-4: Error occurred while fetching data: 400 No matching signature for function DATE Argu- lypes: INT64 Signature: DATECTIMESTAMP [STRING]) Argument I: Unable to type INT64 expected type TIMESTAMP
Thought-4: The error indicates that I'm trying to use the DATE function on an INT64 field, which is invalid. The date fields in the dataset are stored as integers in YYYYMMDD format. need to convert these integer date fields to DATE using the P ARRS_DAAE function. I'll update my SQL query properly parse the dates and include the condition for published applications.
Action-14: BQ EXEC SQL(sql query=.... is "NNe="/worsspaccrrsuu..sss/
Observation- 14: Error occurred while fetching data: 400 No matching signature for function DATE Argu- types: INT64 Signature: MIMESTAME [STRING]) Argument I: Unable to coerce type INT64 expected type TIMESTAMP
Action-15:
query=............ is swv=""wootspacchhsull..ss> Observation-15. Now that the results are saved to /workspace/reeult..cot
BQ EXEC SQu.ss...uerr...........
Action-16: (emnintt(ouuuu=="esuu..ee"
Figure 25: Success case 1. After multiple rounds of SQL writing and repeated debugging, the correct answer was obtained.
Success case 2: DBT data transformation task
### 20 Steps
Action Bash(code="1ls models/salesforee)
Action 4: Bash(code="'cat modelb/sllesfrrre/sssssfoeee,
Action 5: Bash(code="ceat hatpll/sssssteersssssoeeemaannno manager.perffmmacessq"""
Action Bash(cnde="Is mad//salessfncc//neemmcnaae"
Action Bash(code="catt matekvaaleefforc//memmeiiiiii salesforce opportunity by owner.sql")
Action 8: 1.OCAL. DB SQL.(fie data command="SHOw TABLES;", out-
out="direct")
L.C.............l.p.ppprrrrabse. command="SHOw TABLES.". out- put="direct")
Action 9:
Action 10: Craaerffffiooaiimmmmddddcasassse
select date date_day day as activity... ...)
Action 11: Bash(code="dbt run"))
Observation 11: noticed warnings during 'dbt run' indicating that 'salesforce_ daillayattttilllttyiyelaa is not found.
a model named "salesforce_.daill..... _aitivityttviiitt.... should remove references "salesforce yml'. Additionally, I need to add the new model 'salesforce "salesforce_ddilyy daley folce.yml with appropriate descriptions
Since we don't
sales aacivityy to 'sales-
Action force picc__ddilyssllssacttvee LIMIT 5:", outputzdireet)
"SHMEnd="SELECT
FROM sales-
Action 18: "ash(code="dhtt run")
Action 19: airiintteooupuue"ssllsssooeer
Figure 26: Success Case 2.After multiple rounds of file and DBT operations, the data transformation task has been successfully resolved.
40

---

Published as a conference paper at ICLR 2025
Failure case
Action 1: Bash(code="Is- -R")
Action Bash(code="cat models/customer www.eeerreeeeeeeeeeeeeeeeeeeeeee www.m.hhothhhhc..eeeeee mmml")
Action Bash(code="caat models/usstomer.ccmm wwtp:utwwwwtrtt.....oeeeeeeeeeee wwt.rcchtthhsshs..eeeee _month.sql")
Action 4: Edltrilcillpchh="nddeeeksssoons www.rchhhuuuaeeeeeeeee month.sqll)):
Action 5:Bash(code="cat models/mmr.yml"")
Action ".Bashlcode==cat models/mmr.sq!"))
Action "eadifiledieppaaa=mmdoeeeennined
with unioned
select Zrom zef (ttssttorrrcovaanooyyyy 13
uniion all
solect from zet castomer_churrnnontt"
LaggerLivalsg
coalesce(
ligtis act ive) over (partitior by customer id order by date morth), fal www...otttiiiiuueeo..... conierce lag previcus unioned
(pertition eustoner_id date_nonch),
sw wet
date_menth,, custones_ié, nes, la_active, case
Kaea aeriooss_ootth_ss_cctio falso and is_active true then "Kav_Busiine Businnsss
when previoux wonthtttthhhh...oooo is true and ix ive true ner anr_ehhagee 0 then "Eeparaiion
isgaot
acr_change Lhan "Conssact
scteeceve when previoux ouxioux,nn mon..oothh_ttii 1x true and ixcact true arr_change then "Remewall
previcus
xaсa provious_mone itte__oott____etttte faico and is_active taise then "Reurrocctoo""
"Cther"
from Lagged_vall values
Action 8: Bash(code="'htt run"))
Action LOCAL DB SQL((fie murh=dm cOMMand="SELECT FROM mrr LIMIT 5", out- put="ddrect")
Figure 27: Failure case 1, Reasonable data transformation process, but did not follow our predefined category type.
41

---

Published as a conference paper at ICLR 2025
Failure case
Action 1: Bash(code="Is R*)
Action Bash(code="caa README.nd md ga4.page www.ppaaecccceeec category.md")
Action 3: DO_.GE..TBBLESCCC htw.:aaaabbbaaaaamremieeeeeeeee name="bigquery "eagquery-ubblic-dttt""
Action 4: Bash(code="cat httpe/dssppppppemceeeeeeeeeeee eccommeree save.path="datase aaaah=="dtaaet_mettdaaa metadata.csv""
Dataset_name="gaat saa__ooffscatee__aaaaa
Action BQ GET TABLE INFO(database nane="higqueryyuuiiccaaaa"" ,ataaset
"ga4 obfuscated sample ecommerce", table="events 20210109", save path="ga4 info.esv")
WICH pdg_pages AB SELECT
event parans ve ue. ce-string rining ve loe AS page. I locat
FRO0
Valgueeyy publ io del.agad_ pete_abbbaaatteeeeenng wwt.m//ww.oooo....o...ooo 2tt.
UNNEST Leveett__
paent_
WHERS
"page "pace_iocation
preat_ppranoo key
VI" (V/]=///// ("V (MVV V/-//""MM (V/]+++ (*V11>>)
AND RESEXP CONTAINS (event, parans, val uel string value, e/A/ (Acpessories Appare IBrands Campus, Col eet ion
p Redesign Lifestylo Bost 1REM_2015__00 (Nozobooks A,. Journali Ortiies
Shep.py Brandd Small, Bepli1,,Goo Goode Stationory (Mcorables) V/1
para___ar__
SELECT
COUNT;-] total_riows
aigquurr--publ viex!
KHERS
SELECT
(SELSCI COUNT
103.0 total__age, pagee__sge_voooss.oo www.wriwww.ttttieeeeoo tiees AS percentagg
FROM
total
Action Temmintteoutppus="rssllls"""""""
Figure 28: Failure case 2. Cannot ground the complex document into SQL.
Failure case 3
Action 1: Bash(cnde="1s -R")
Action 2: BQ GET dAaaaaee nume="tigquery--uulioodda
dataset name="ga4 obfuscated sample ecommerce" save path= metadata.csv")
Action 3: cashhcode==cca cat iatiuusee.mttdat.ccs...
Action 4: BE..ETT.TABBBBBE BEETT.AALE.INTT
TALL.TTTAAETTOOOtaaatttt.... (dao(daaaaase waaaaaaaeeaaaeaaeiiiieeeee maan=="biguury-publiic.. www.blii-daaaaddatateee... ddaaaset_names
table="'evenss 2eteeeeett_220000000 wwwppatthhaaaiiiiitt
wrent_timestten timestanp,
pornt_sorane.oo Key,
event_parans, www.:a/.........trioooaa.l.... page_iocation
F30M
"aiguurry public data.gee__ob wew.ggee_ooouestteeee serpll _er_reeeelleeeeeeeecooooooe wcw.necce..cc... wwt.eeeeeetet111111_11 x_20210128
ST (areat)_aaaaann) WHENE
parans
'1362220' AND daeet_ddate 0202
oRDEa BY tvont_timestane
Action Temminte(outpuuF/"ssullss"""
Figure 29: Failure case Cannot understand the complex nested schema.
42

---

Published as a conference paper at ICLR 2025
C.7 PROMPTS SPIDER-AGER
Following the ReAct Yao et al. 2022 prompting framework, which guides models in breaking down complex problems into manageable steps through an iterative chain of reasoning (Reasoning). action (Action), and observation (Observation) phases, where the current state is observed after each action before proceeding to the next step of reasoning, we build the Spider-Agent. Below, we will introduce the system messages and actions used within this framework.
System Prompt
System Prompt
You are a data scientist proficient in database, SQL and DBT Project. You are starting in the {work.dir) directory, which contains all the data needed for your tasks. You only use the actions provided in the action space solve the task.
### [Action space]: {action prompts}
### [Notice]
1. First, run "Is" to check the current folder for files. there are other markdown files, read them as they may contain useful information.
Examine the database schema folder, you fully understand the structure schema of the database.
Use appropriate SQL execution action to run queries.
Be prepared to write multiple SQL queries to find the correct answer. If an enror occurs, revisit the database information and previous queries to adjust your SQL accordingly.
Ensure the results valid. If the result.esv file is empty only contains header, the SQL query is incorrect. The final result should be cither saved CSV or directly provided text answer, an intermediate step or SQL statement.
After completing the task, verify the output data against the definitions. For dbt projects, after writing the SQL,, run dbt run to update the database and confirm the new data models meet the YAMI. file definitions.
### [Response format]
For each task input, your response should contain:
1.
One analysis of the task the current environment, reasoning to determine the next action (prefix
"Thought: ").
2. One action string in the ACTION SPACE (prefix "Action:
### [Example interaction]
Observation: (the output of last actions, provided by the environment and the code output, you don't need to generate Thought:
Action:
### [Task]: {Task}
Action Space Prompt
Bash
## Bash Action
Signature: Bash(code="shell command") Description: This action string will execute command the code field. Only non-interactive commands are supported. Commands like "vim" and viewing images directly (e.g.. using "display") are not allowed.
Example: Bash(code="Is -1")
CreateFile
## CreateFile Action
Signature: htaeeiillcoode=seelllooooee
Description: This action string will execute valid shell command in the 'code" field. Only -on-interective commands supported. Commands like "vim" and viewing images directly (e.g., using "display") are not allowed.
Example: CreateFile(codle==kk..
43

---

Published as a conference paper at ICLR 2025
EditFile
## EditFile
Signature:
File_content
Description: This action will overwrite the specified in the filepath field with content wrapped in paired symbols. Normally, you need to read the file before deciding to use EditFile to modify it. Example: Eahifille(ilepat=="hhllll "hw.lo.worl....p"""
print Hello,, world!")
BIGQUERY EXEC.SO.
## BIGQUERY _EXEC_SQL
Signature: BIGQUERY WEEEEE__EIUUI AXXCC.SQUIssLLperrr=SSS.....
FROM your_table", is_save=True,
save hath=///worsspcc/outuuu file.esv")
Description: Executes SQL query on Google Cloud BigQuery. If "is save' True, the results are saved to specified CSV file; otherwise, results are printed. If you estimate that the number of returned rows is small, can set Faasave=F,sse, to directly view the results. If you estimate that the number of returned is large, be sure is save True. The 'save-path" CSV must be under the '/workspace" directory. Examples:
Example1: BIGQUERY EXEC WEEEECSLL(sql..pe. SQL.ssp..uerr>"SELEEEE count(") FROM sales", (s-save=False)
Example2: BIGQUERY EXEC EXEC_SQL
user_id, sum(purchases) FROM transactions
GROUP BY user id", , is ss_save=True,
GET,TABLES
## GET DABLES
Signature: GET
GGT.TABLESKdabbee_nmeeeVoo waw.:abboeeeaameeoouuuuuuuu.a namee. lataset.name= www.aaammmeyyouuudttttttm.mmmam dat.adetattemmeee
Fxecutes query fetch all table names and their corresponding DDI. from the specified dataset Google Cloud BigQuery. The results saved the specified CSV file.
Description:
BigQuery of table usually in form database name.datase, name.table table name. This action mainly focuses on the tables under nataset.name.
Examples:
Examplel: GET
dataset_name="nnw.ooff
save_path="
pat.paattttssstttttaaaa ddtaaset www.mmmetaaaaa""""
GET TABLES INFO
## TABLE. INFO Action
Signature:
GET.TABLE_NG
datasetlaame=""yuurddtttttttt
ble="table _ameee,
auth=="path/oo/ouuptt..iii
Description: Executes query to fetch all column information (field path, data type, description) from the specified table in the dataset in Google Cloud BigQuery. The results are saved to the specified CSV file.
The BigQuery id of table is usually in the form of database name.dataset name.table name.
Examples:
Examplel: GET CEL.TABLEJIFFOOLEE database daaaooooedataabeeeaaaaaaaaa wwwpbbee--uuuli---aaeee "sssshaaespprre"" save "saah===shasppearee www.opraaareiiirrr...
SAMPLE_RWW
## SAMPLE ROWS Action
Signature:
SAMPLE ROWS(database name=""our maaeaayyuurdatabaseee
database _naee".
datasel_nam=="your wawp-naammmeyuuuuudataaeeeee wlwaaaet-naaeee
ble= Mc==ltable_nt _amme,, watppaapaahhto/ouuuuee httptoo.uuuuuuuuusscrr)r) filc.csv")
Description; Executes query to fetch all column information (field path, data type, and description) from the specified table in the datasct in Google Cloud BigQuery. The results saved the specified CSV file. natase dataset habl. name.
The BigQuery id of table is usually the form of database name
Examples:
44

---

Published as a conference paper at ICLR 2025
Examplel: htw.://wWSS/aaaaaeaaeeeeeeeegeee "le="shakespeare" sawve_path="shaksppeee
nataset_ame="sampllee ta-
45


Reviews:
[{"review_id": "rl0WaJfIOc", "reviewer": "Reviewer_mvyZ", "summary": "The paper introduces Spider 2.0, a new NL2SQL benchmark designed to address the challenges of real-world enterprise text-to-SQL workflows. Unlike previous benchmarks, Spider 2.0 focuses on 595 complex workflow problems derived from real-world enterprise use cases involving databases often exceeding 1,000 columns and stored across diverse systems like BigQuery and Snowflake. These problems require models to handle intricate SQL workflows, understand long contexts, explore database metadata, manage dialect variations, and interact with project-level codebases. The tasks frequently involve generating multi-step, nested SQL queries exceeding 100 lines, encompassing diverse operations from data transformation to analytics. Spider 2.0 thus pushes beyond traditional text-to-SQL challenges, demanding higher levels of reasoning and adaptability from language models.\n\nThe evaluations reveal significant gaps in existing language model capabilities, with the introduced code agent framework solving only 15.1% of Spider 2.0 tasks, compared to 91.2% on Spider 1.0 and 73.0% on BIRD. This highlights the difficulty of handling real-world enterprise scenarios, such as schema linking in large databases, navigating diverse SQL dialects, planning complex query sequences, and effectively leveraging external documentation. The insights from Spider 2.0 underscore the need for substantial advancements in autonomous code agents to meet enterprise requirements.", "questions": "No questions", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1) Originality: Spide 12.0 significantly extends traditional text-to-SQL benchmarks by focusing on real-world enterprise scenarios.  It incorporates a broader and more complex range of challenges, such as handling extremely large databases, diverse SQL dialects, and intricate workflows. This originality lies in redefining the scope of text-to-SQL tasks to better align with practical, enterprise-level applications.\n\nS2) Quality: the benchmark created and curated 595 real-world text-to-SQL workflow problems derived from enterprise use cases, ensuring relevance and rigor. The inclusion of diverse database systems like BigQuery and Snowflake, along with tasks involving database metadata, codebases, and nested SQL queries, reflects meticulous data collection and a comprehensive representation of real-world complexity.\n\nS3) Clarity: the paper is very well organized and carefully written. The content is very rich and deep. By providing detailed analysis and explicit examples of schema linking, dialect handling, and multi-step query generation, it offers clear insights into the difficulties faced by language models in enterprise scenarios.\n\nS4) Reproducibility: the paper provides source code.", "weaknesses": "I don't find any major weakness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FsBybIUTr", "reviewer": "Reviewer_XhZP", "summary": "The paper introduces a new benchmark, Spider 2.0, which consists of 595 real-world text-to-SQL workflow tasks over ~1000 columns. The  text-to-SQL tasks are significant more challenging compared to the commonly used Spider and Bird benchmarks. The authors evaluate multiple LLMs using Spider 2.0 and present a set of interesting findings.", "questions": "Q1. Is CodeS-15B fine-tuned on Spider 2, or the fine-tuned model from the repo was directly used for evaluations?\n\nQ2. Among dialect function usage errors, what is the ratio between simple syntax errors (i.e., LLMs not knowing the specific syntax) and completely incorrect chosen function?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1. The new benchmark dataset is publicly available and critical for the community and enterprises that develop LLM-based text-to-SQL solutions.\n\nS2. The dataset creation is solid and clearly presented in the paper. The SQL complexity including functions and dialects is closer to the real-life enterprise setting.\n\nS3. The analysis provides some interesting findings, especially the error analysis of SQL generation.", "weaknesses": "W1. The agent aspect of the benchmark should be emphasized more as it largely separates Spider 2 from the other benchmarks. It is reasonable to use execution results as the metric for evaluation. However, the other important aspect is the ideal multi-turn process for each task. For example, what is the typical number of turns and the sequence to complete the task.\n\nW2. The difficult level in Sec. 3.1 is acceptable but not comprehensive. The SQL complexity reflects one aspect of the task difficulty. The other aspects include the data complexity (e.g., whether the task requires access to nested schema, any question/data ambiguity, etc.). If these are not as straightforward as the number of tokens, associating some tags to each task would help one to understand the specific challenges associated with the tasks.\n\nW3. It seems that SQL is the only \"code\" allowed to execute the task. However, certain tasks such as data transformation might be easier to be executed through script languages (e.g., Python). IMO, a more realistic setting should allow LLMs to decide the best way to execute a (sub-)task. This will enable the LLMs, such as GPT-4o, to better leverage their strength in code generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hq04N5XFMw", "reviewer": "Reviewer_Bdrh", "summary": "This paper introduces Spider 2.0, a benchmark aimed at assessing the capabilities of large language models (LLMs) in handling complex text-to-SQL tasks within enterprise environments. Unlike previous datasets, Spider 2.0 involves real-world databases containing thousands of columns and diverse SQL dialects, with tasks that require interaction with database metadata, project-level codebases, and external documentation. The benchmark emphasizes challenges such as processing extensive schemas, using advanced SQL functions, and generating multi-step SQL queries.\n\nKey findings highlight that current LLMs, even state-of-the-art models like GPT-4, struggle with Spider 2.0, achieving a success rate of only 15.1%. This reveals significant gaps in their ability to handle industrial-grade SQL workflows, which involve complex data transformations and analytical tasks. Spider 2.0 aims to push the development of more capable models for practical enterprise applications by providing a realistic and challenging benchmark.", "questions": "1. W3\n\n2. May I ask where can I find the proper json file for Spider 2.0 and Spider 2.0 Light in your repo?\n\n3. Please figure out the difference between your work and MLE-Bench made by OpenAI", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "S1. It is important to have Spider 2.0 for real enterprise Text-to-SQL. Spider 1.0 and BIRD focuses on multi-table tasks, while WikiSQL and GeoQuery only focus on single table tasks. These datasets are popular, but they still represent a very small part of the whole data analytics/science workflow. The emerge of Spider 2.0 tries to fill this gap.\n\nS2. The collection of Spider 2.0 complies the data manipulation workflow in real enterprise scenarios. Detailed types of queries and error analysis are shown in detail, which indicates the future direction of improvement.\n\nS3. Spider 2.0 is a valuable dataset for the future agent of data analytics, which will be an important future trend of the data science domain. This paper also provides Spider-Agent, which can be a new baseline of data analytics agent.", "weaknesses": "W1. There should be an opportunity to have an end-to-end benchmark for enterprise data analytics/ data science workflow with both SQL query and Python. Also, it would be better to also consider queries on Hadoop/Cassandra, which are also widely-used storages.\n\nW2. In this paper, schema linking and table mapping results are shown. It would be better to open-source these two related datasets to extend the gap of LLM-based Text-to-SQL schema linking.\n\nW3. It would be better to show the number of tokens of Codebase+DB Info+Documents, which should be an important parameter for picking up proper LLMs to construct agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ffYzEz68U", "reviewer": "Reviewer_c1AS", "summary": "The paper presents Spider 2.0, a new benchmark for Text-to-SQL systems that introduces more complex and realistic SQL tasks compared to benchmarks like Spider 1.0 and BIRD. Spider 2.0 tasks require models to leverage project-level information, such as codebases and documentation, for generating accurate SQL queries. It also includes more SQL dialects such as DuckDB, Snowflake, and BigQuery, and utilizes advanced SQL functions such as ST_DISTANCE and CORR. \n\nThe benchmark demonstrates its difficulty by showing that even state-of-the-art language models achieve a maximum accuracy of only 15.12%. This result highlights significant gaps in current Text-to-SQL approaches that perform well on enterprise-level SQL generation tasks. Spider 2.0 can potentially advance Text-to-SQL systems, bringing them closer to being practical for enterprise applications.", "questions": "1. How many tables per database in Spider 2.0? The paper only reports the number of columns per database. \n2. The paper's abstract states that it can \"generate multiple SQL queries with diverse operations, often exceeding 100 lines.\" However, the queries average 144 tokens, which seems unlikely to translate into over 100 lines. Am I misunderstanding something here?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Spider 2.0 is the first Text-to-SQL benchmark to incorporate complex contexts — such as data pipeline code and SQL dialect documentation — into the SQL generation process, reflecting an essential aspect of real-world SQL writing in a data engineer’s workflow. This marks a significant advancement over BIRD, which only includes simple evidence (1-2 sentences) as external knowledge.\n2. SQL queries in Spider 2.0 are derived from actual projects and tutorials, enhancing the benchmark's realism and quality. Each database instance is selected to ensure sufficient complexity, with a minimum of 200 columns per database.\n3. The evaluation in this paper is highly insightful, offering readers a clear perspective on the current limitations of LLM-based text-to-SQL approaches.", "weaknesses": "The goal of this paper is to simulate real-world enterprise text-to-SQL workflows. While the authors have made substantial progress toward this aim, several important aspects of enterprise workflow characteristics remain unaddressed.\n\n1. Context Complexity: The paper has included codebase and SQL docs as the context. However, in the enterprise setting, business context is also crucial. For example, different companies have different ways to compute daily active users or user retention rates. There can be a page-long documentation to explain how a metric should be calculated or even a page-long formula that defines a metric directly. This kind of context is missing from the benchmark. \n2. Schema Complexity: The paper says it selects databases where the number of columns is larger than 200. However, that does not guarantee that the schema is complex enough. For example, these 200 columns can be stored in one table. It is not clear how complicated the table relationship is. How many joins are involved in the SQL?\n3. Question Complexity: The questions are generated by the authors based on the SQLs crawled from forums or tutorials, which might not reflect the nature of real-world BI questions, which often ask for trends or patterns. Including real data engineers to audit the benchmark or participate in the benchmark generation process can make it more convincing.\n\nDespite these limitations, I recognize the authors' efforts in taking an important first step toward an enterprise setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)