Evaluating Cross-Domain Text-to-SQL Models and Benchmarks
Authors: Mohammadreza Pourreza
Davood Rafiei
Venue: EMNLP 2023 Main (2023)
Keywords: Text-to-SQL
Natural language interfaces to databases
Benchmarks
Evaluation
TL;DR: A study revealing limitations in text-to-SQL benchmarks and the need for additional independent evaluations

Abstract: Text-to-SQL benchmarks play a crucial role in evaluating the progress made in the field and the ranking of different models. However, accurately matching a model-generated SQL query to a reference SQL query in a benchmark fails for various reasons, such as underspecified natural language queries, inherent assumptions in both model-generated and reference queries, and the non-deterministic nature of SQL output under certain conditions. In this paper, we conduct an extensive study of several prominent cross-domain text-to-SQL benchmarks and re-evaluate some of the top-performing models within these benchmarks, by both manually evaluating the SQL queries and rewriting them in equivalent expressions. Our evaluation reveals that attaining a perfect performance on these benchmarks is unfeasible due to the multiple interpretations that can be derived from the provided samples. Furthermore, we find that the true performance of the models is underestimated and their relative performance changes after a re-evaluation. Most notably, our evaluation reveals a surprising discovery: a recent GPT4-based model surpasses the gold standard reference queries in the Spider benchmark in our human evaluation. This finding highlights the importance of interpreting benchmark evaluations cautiously, while also acknowledging the critical role of additional independent evaluations in driving advancements in the field.

Evaluating Cross-Domain Text-to-SQL Models and Benchmarks
Mohammadreza Pourreza
Davood Rafiei University Alberta
University Alberta pourreza@ualberta..cm
drafiei@ualberta.
Abstract
proved from 63.6 (Wang et al., 2019) 74.0 (Li et al., 2023a). Measuring such progress is hinged on reliable benchmarks and evaluation metrics.
Text-to-SS SQL benchmarks play erucial role in evaluating the progress made in the field and the ranking different models. However, accu- rately matching model-generatee SQL query to a reference SQL query in benchmark fails for various reasons, such as underspecified ural language queries, inherent assumptions in both model-generated and reference queries, and the -entemeeriiiittic nature of SQL output under certain conditions. In this paper, we con- duct an extensive study of several prominent cross-domain text-to-SQL benchmarks and re- evaluate some of the top-performing models within these benchmarks, by both manually evaluating the SQL queries and rewriting them equivalent expressions. Our evaluation re- veals that attaining perfect performance these benchmarks is unfcasible due to the multi- ple interpretations that can be derived from the provided samples. Furthermore, we find that the true performance of the models underes- timated and their relative performance changes after re-evaluation. Most notably, our evalu- ation reveals surprising discovery: a recent GPT4-based model surpasses the gold standard reference queries in the Spider benchmark in our human evaluation. This finding highlights the importance of interpreting benchmark eval- uations cautiously, while also acknowledging the critical role additional independent eval- uations in driving advancements in the field.
Two standard metrics for evaluating the perfor- mance in this domain have been exact set match accuracy and execution accuracy. The former mea- sures if a model-generated SQL query lexically matches reference SQL query, whereas the lat- ter measures if a model-generated SQL query pro- duces the same output as a reference query ($ 4).
What are the names and ids all makers with
Model
SELECT FullName T1.ld CAR,
car makers JOIN model list AS T2 ON TI T2Maker GROUP BY Tıkkd HAVING count(")
JOIN MODEL LIST AS T2 ON T1.ld 12.Maker GROUP BY T1.ld HAVING count")
[[Gannoral Motora', 4). ('Chrysler,, 6)]
[[4, 'gm'), [6, "chrysier\)
An example question with two correct SQL queries, each corresponding different interpretation. There is an ambiguity in schema mapping, with two different database columns describing the name.
Figure 1:
1 Introduction
Consider the example in Figure 1, which con- sists of a model-generated query (shown on the left) and a reference query (shown on the right). Both SQL queries return the id and name of mak- ers that have more than 3 models. However, the model-generated query returns the column Full- Name, which gives the full name of maker (e.g.. "Ford Motor Company"), whereas the reference query given in the benchmark returns the column Maker, which gives the short common name of a maker (e.g., "Ford"). The model -enerreee query
Significant progress has been made in translat- ing natural language text to SQL statements over the past few years. The execution accuracy on the hold-out test of Spider (Yu et al., 201880 8b) large-scale cross-domain text-to-SQL benchmark- has improved from 53.5 in May, 2020 (Zhong al.. 2020b) to 85.3 in March, 2023 (Pourreza and Rafiei, 2023). The exact set match accuracy, without considering database cell values, on the same benchmark and over the same period has im-

---

fails an exact set match since the column names in the select clause are different. The query outputs are also different and the model-generated query fails the execution accuracy as well. The natural language utterance not specific about the type of name to be returned, and a human evaluator tags both queries correct.
the identified issues by updating these benchmarks requires considerable human effort. Additionally, benchmarks in the Text-to-SQL domain, like Spi- der and BIRD, have holdout test sets used for offi- cial leaderboards and comparisons of text-to-SQL. methodologies. We only have access to the de- velopment and training sets of these benchmarks, which limits our capability to alter the test sets. As result, making changes only the development and training sets would not completely address the benchmark's ' inherent problems, given that final performance gauged using the problematic test sets.
As the models improve, these types of failures make up most of the errors, and the performance metries become less relevant, as shown in our evaluation. In particular, we re-evaluated all de- velopment set queries of Spider on which two top-performing models, one using fine-tuned model (Scholak et al., 2021) and another using large language model (Pourreza and Rafiei, 2023), failed. We found out that 25% of the queries gen- crated by one model and 87% of the queries gen- erated by the other model were indeed correct but were wrongly evaluated by the benchmark. For the same set of queries, our re-evaluation of the ground truth queries found 33% of the SQL queries incor- rect, which was more than the number of incorrect queries generated by one of the models. This eval- uation places one of the models above the ground truth queries in this re-evaluation.
2 Related Work
Limited research has been dedicated to assessing the reliability and effectiveness of Text-to-SQL. benchmarks. The authors of SQL-PaLM (Sun et al., 2023) note in their qualitative analysis of their model that some queries, labelled as incorrect by execution accuracy, were considered correct by human annotators. Similarly, Lei et (2020) conduct an analysis highlighting the discrepancy between automatic evaluations and human anno- tations. They emphasize that certain queries pro- duced by the models were labeled as incorrect SQL queries but human annotators labelled them as rect queries. Generally, query that is equivalent (but not identical) to ground truth may be mistak- enly classified as incorrect by automated evaluation metrics. Another study by Zhong et al. (2022) iden- tifies limitations within the Spider benchmark, such as issues with ties and certain syntactic problems. Their analysis is primarily focused on a subset of Spider, without quantifying the extent impact of these limitations conducting assessment of other benchmarks.
We further re-evaluated two well-known bench- marks, Spider (Yu et al., 2018b) and Spider- DK (Gan et al., 2021b), and a newly released bench- mark, BIRD (Li et al., 2023b), and found similar problems in all three benchmarks that affect the evaluation. Our evaluation reveals that 18% of the queries in the train sets and 20%-23% of the queries in the dev sets of these benchmarks are subject to ties in the dataset and which one of the tied rows are returned. This means a model-generated query will be deemed incorrect if it does not return the same row, among tied rows, as the ground truth query. This can severely impact the evaluation, es- pecially when there is tight race among models. Considering these observations, it is crucial to phasize the significance of additional independent evaluations when utilizing these benchmarks. To enhance the evaluation process further, a poten- tial solution to incorporate multiple SQL queries as the ground truth, each representing a different interpretation that may be valid.
Text-to-SQL Benchmarks
Benchmarks have played a crucial role in advanc- ing the field and providing a platform for evalua- tion. WikiSQL (Zhong et al., 2017) consists of over 24,000 tables from Wikipedia with SQL queries generated based on some predefined rules and tem- plates. The queries in this dataset are considered casy since they are all single-table queries. Spi- der, introduced by Yu et al. (2018b) (b)),, consists of 200 database schemas of which 160 schemas are published as train and dev sets and schemas are kept hidden for testing. The queries are written on those schemas by Computer Science students
Our objective in this paper is to provide comprehensive evaluation of existing Text-to-SQL benchmarks, underscoring the inherent issues they possess. We refrain from introducing a new dataset due to several considerations. First, addressing

---

without using templates. This is considered chal- lenging dataset. Some other benchmarks are devel- oped based on Spider, including Spider-Syn (Gan et al., 20) which replaces schema-related words with synonyms and eliminates explicit mentions be- tween NLQ and schema, and Spider-DK (Gan et al., 2021b), which introduces rarely observed domain knowledge into the Spider development set. Other benchmarks include FIBEN (Sen et al., 2020), cre- ated for the financial domain and BIRD (Li al., 2023b), which comprises 12,751 queries over 95 databases spanning 37 professional domains.
matching of literals can be challenging since predicates such as rationality="Candda" and nati matiioallty=="Candian"" will not match. How- ever, accurately generating those literals without accessing database content may not be possible. Under exact set matching without values, which is used in Spider (Yu et al., 2018b), matching of literals is not required.
Two equivalent SQL queries can have different expressions and may not match under an exact set match. An alternative metric that can reduce the number of false negatives is the execution accu- racy. Under execution accuracy, the cquivalence between model-generated query and a reference query is established if they both produce the same results all possible databases instances (Yu al., 2018a). While testing all instances is impractical. running queries on subset of instances can help identify candidates that are not equivalent to the reference query. Although execution accuracy can detect queries that are equivalent but not identical, it may mistakenly identify queries as equivalent if they produce the same result tested instances. Therefore, effective execution riccution-based evaluation requires finding instances that cover various edge cases and can detect queries that are not equivalent to the reference. Test suite accuracy (Zhong et al., 2020a), which is simply referred to as execution ac- curacy in Spider benchmark and in our work, aims to minimize false positives by evaluating queries on a carefully selected collection of database in- stances, known as test suite. Nevertheless, execution-based accuracy cannot capture all cor- rect SQL queries, highlighting the limitations and the continued importance of human evaluation for reliable assessment.
Our study in this paper focuses on cross-domain large-scale benchmark Spider, its variants Spider- DK and Spider-SYN, and more recent cross- domain large-scale benchmark BIRD. The selec- tion of these benchmarks stems from their resem- blance to real-world datasets, which a crucial fac- tor in conducting comprehensive research and anal- ysis. One notable advantage of these benchmarks is the availability of a large training set, which plays pivotal role in training and fine-tuning large-scale models. The inclusion of substantial amount of training data enables the development of more ro- bust and powerful models that can better handle the complexities and nuances present in real-world databases.
4 Evaluation Metrics
The performance evaluation of text-to-SQL. sys- tems involves comparing them to reference sys- tem, typically gold standard set of known correct SQL queries. Generating reference can be chal- lenging due to multiple interpretations of natural language questions, while SQL queries are based on logic and tend to cover only one interpretation. Even if an interpretation is fixed, detecting if a model-generated query is equivalent to reference query is challenging, due to the halting problem which is undecidable (Davis, 2004). Nonetheless, assess progress, proxy measures of performance have been developed the literature. As two such metries, we review exact set match accuracy and execution accuracy this paper.
Execution Accuracy Failures
A model-generated query can be correct but still fail the execution accuracy. We classify these fail- ures into three categories: (1) failures due to ties in output, (2) ambiguity in schema matching, (3) wrong assumptions made about database content.
5.1 Failures Due to Ties in Output
Under exact set match accuracy, SQL queries are evaluated by matching the query clauses and components independently, such the se- lect, where, having, group by, and order by clauses. The matching is based on compar- ing columns and predicates, disregarding the or- dering of columns and predicates. An exact
SQL queries can lead to ties and subset of the tied rows may be returned. The selection of tied rows can vary between queries and this can affect the execution accuracy. We identify a few sources for such ties, as discussed next, and study their impact on benchmark evaluations in Section 6. Table

---

provides detailed breakdown of the number of queries that can potentially yield tied rows in both train and development set of Spider, Spider-DK, and BIRD benchmarks.
What Independence year, surface area of country with the smallest population?
Model
5.1.1 Top with Ties
SELECT Name, SurfaceArea, Indep Yoor FROM country
BELECT Name SurfaceArea IndeyYear FROM country
WHERE Population (SELECT
Sometimes the query asks for top rows that satisfy some conditions (e.g., the student with the highest GPA, or the youngest student). When there is tie for the top position, and the query in natural language is not specific on how the ties should be handled, the corresponding SQL query may return all ties or only one. This becomes a problem in eval- uation if a model -eneertdd query and the reference query treat the ties differently. Figure 2 provides a concrete example from the Spider dataset, illustrat- ing this issue, where the reference SQL query in the benchmark fails to account for ties and returns only one of them using the LIMIT keyword.
miin(Population FROM country):
ORDER BY Population LIMIT
[VAnntacctica', 19120000.0, (French Southern territoriee' 7780.0, None), (Bouuvet loland'; $9.0, Monejl
l'Antarctica', 131200000,Nomo))
An example question that can have two correct SQL queries, each corresponding to a different interpre- tation. The SQL query on the left returns all tied values, while the SQL query on the right returns only one of the tied values.
Figure 2:
both cases, multiple records can be associated with the same grouping column or aggregation value, whereas each group can only return one record. Some database systems including Oracle and DB2 prevent these cases by treating them as syntax er- rors. However, other database systems such as SQLite and MySQL take a more lazy approach (sometimes for efficiency reasons) and allow these cases to happen. Many text-to-SQL benchmarks follow SQLite syntax and suffer from this issue. The affected queries in our benchmarks were iden- tified after migrating from SQLite to PostgreSQL., as detailed in Section 6.4, and checking for queries that failed during PostgreSQL. execution. Figure 3, illustrates one example of such problem from the Spider dataset.
5.1.2 LIMIT N
The problems associated with using the LIMIT clause in SQL queries not limited to the top posi- tion, as discussed above. The use of this clause is problematic for evaluation in general. Firstly, with- out an explicit ordering, the result of a SQL query is expected to be set. Two equivalent (but not identical) queries return the same set of results, each listed in different orders, but selecting the first n rows from one ordering will not necessarily match the same selection from different ordering. Secondly, with query results sorted, there can be a tie on row n with multiple rows having the same values. The ordering among tied rows can vary between two queries, and so is the first rows that are returned. All benchmarks studied in this paper (Spider, Spider-DK: Spider-SYN, BIRD) use the limit keyword and suffer from the aforementioned problems associated with ties.
5.1.4 ORDER BY
Another subtle ambiguity with tied values arises in queries where the SELECT clause incorporates the "distinct" keyword, paired with an ORDER BY clause referencing a column absent in the SELECT clause. Consider the exemplary query from Spi- der train set: SELECT DISTINCT district_name FROM district ORDER BY city_area DESC. The ordering of the output, well as the result of com- parison with reference query, becomes uncertain if single 'district_nanne _ame" value maps multiple coity_ared values. Similar to GROUP BY, the af- fected queries in the benchmarks were identified through SQLite to PostgreSQL migration($ 6.4).
5.1.3 GROUP BY
Many text-to-SQL benchmarks encounter a differ- ent type of issue associated with ties, particularly arising due to incorrect usage of non-aggregated columns in both the SELECT clause and the GROUP BY clause. Within the benchmarks, these ties manifest in two scenarios: 1) column appears in the SELECT clause without being inside an ag- gregation function and without being included in the GROUP BY clause; 2) the SELECT clause contains a mix of aggregated and non non-aggnee aaggrggggt columns without utilizing a GROUP BY clause. In

---

Benchmark
LIMIT
LIMIT N
GROUP BY
ORDER BY
Total
Dev set
BIRD
255(16%)
42(2%)
20(1%)
4(0.2%)
321(20.86%)
Spider
171(16%)
10(0.9%)
51(4.5%)
2(0.2%)
234(22.63%)
Spider-DK
94(17%)
2(0.3%)
30(4.5%)
2(0.3%)
128(23.85%) (55%%)
Train set
BIRD
1558(16%)
211 (2%)
23 (0.2%)
4(0.04%)
1792 (18.22%)
Spider
989(14%)
106(1%)
254(3%)
10(0.1%)
(159(18.1%)
Table 1: The number of SQL queries having specific type of limitation together with the pereentage on both development set and train set. The Spider-DK dataset does not have any training set.
5.3 Wrong Assumptions on DB Content
What is the lenguage spoken by the largest percentage people
Lastly, one type limitation in text-to-SQL bench- marks stems from incorrect assumptions regarding cell values. It is common to make assumptions about database content and constraints when writ- ing SQL queries, but those assumptions may not be supported by the database schema content. This issue arises when the database content is created under assumptions that do not align with those in queries, leading to potential failures the evalua- tion process. Text-to-SQL models often lack access to full database content due to limitations such as the context window problem and the inability to pass all cell values to the models for reasons such as privacy and cost. These models typically rely the provided database schema and selected sample of database rows to represent potential val- ues (Pourreza and Rafiei, 2023; Liu et al., 2023; Rajkumar et al., 2022: Sun et al., 2023; Li et al.. 2023a; Lin et al., 2020). Consequently, the assump- tions made by these models may not align with the actual ground truth, resulting in SQL queries that are correct under the assumption made but do not match the reference query in the benchmark.
country?
Model SELECT LANGUAGE, CountryCode max/Pereentagel FROM countrylanguage GROUP LANGUAGE, CountrCode
Ref SELECT LANGUAGE, CountryCode max(Porcantago) FROM countrylanguage GROUP CountryCodo
Pashto', 'AFG: 82 4).
['Pashtto', 52.4), I'Dutch', 'BEL', 59.2).
(Pashto' 'PAK', 13.1),
I'Dutch', 'ABW', 6.3).
('Dutoh'. 'NLD', 96.6),
I'Dutch' (Dutch", 'BEL', 59.2),
Figure example question that can have two correct SQL queries, each corresponding to different interpre- tation. The SQL query on the left returns all languages of each country, each pair of country and language in separate row, whereas the SQL query on the right re- turns one of tied values for the column LANGUAGE.
5.2 Ambiguity in Schema Matching
Schema matching refers to the task of establishing the correspondence between natural language question and the tables, columns, and cell values in the database ((Cao et al., 2021; Pourreza and Rafiei, 2023; Wang al., 2019; Li al., 2023b). Ambiguities arise when there are multiple columns in the database that can represent the same semantic meaning, and the information needs of a query may be satisfied using any of those columns. As a result, there exist multiple SQL queries that can produce the correct answer, yet most benchmarks only provide one query among the many possible correct answers. Figure illustrates an example question that can be satisfied by two different SQL. queries, both of which are valid responses to the question at hand.
One observed case is when certain condi- tions (e.g., PerType=='ogl queries due the erroneous assumption that the condition holds for all rows the database. Figure 4 exemplifies this issue using an example from the Spider dataset, where both queries yield the same answer on a specific database instance. However, changing the database values could result in failure, especially when evaluating performance using test- suite accuracy, which involves querying different database instances. Another case observed in the benchmarks is when the ground truth SQL queries assume a specific column has unique values, but in reality, that column does not possess that unique constraint. Figure 5 depicts an example of this
omitted from SQL

---

problem from the Spider dataset.
6.1 Evaluation Through Query Rewriting
In this experiment, our focus on ties and how a tie breaking strategy affects the benchmarks and our evaluation. This is done through query rewrit- ing. Automating query rewriting faces inherent challenges, particularly when dealing with failures stemming from schema ambiguity, erroncous sumptions about the database content, and the am- biguity of natural language utterances. These chal- lenges arise because there no specific structure to address the failures systematically. Successful query rewriting in these cases necessitates a deeper understanding of table and column semantics to identify ambiguities and erroneous assumptions. In cases of ambiguity, human expertise is essen- tial to disambiguate the context, these situations often lack clear guidelines. Detecting erroncous assumptions often involves introducing new data to the database and meticulously reviewing and correcting failed queries on a case-by-casc basis. Therefore, our efforts have been channeled towards rewriting queries concerning tied values, which ad- here to a specific syntax structure, and the problems associated with the ambiguity in schema matching and wrong assumptions on database content are studied in the next section.
Find the weight of the youngest dog.
Model
SELECT weight FROM WHERE Petlype 'do'' pet age (SELECT MINipet age) FROM Pets WHERE Pettype dog")
SELECT weight FROM pets ORDER BY pet_age LIMIT
[[9.3]]
[19-91I
Figure 4: An example of question and SQL pair with a wrong assumption on the cell values. The SQL query on the left does not make the same assumption.
What are the names of students who have no friends?
SELECT name FROM
Highechooler WHERE NOT IN (SELECT FROM
name Highschooler EXCEPT 12.ame FROM Friend AS T1 JOIN Highachooler AS 12 ON T1, 11..sueent. id T2.d
Highschooler JOIN
Many benchmark queries use "LIMIT 1" to find top rows that satisfy some conditions. If there are ties top, one arbitrary row among ties is returned. An alternative is to return all ties. We rewrote all queries that used "LIMIT 1" to return all ties. This was done by introducing min() and max() aggrega- tion functions within nested queries to accurately identify extreme values. An example of such rewrit- ing is shown in Figure 2. Breaking ties for queries that used "LIMIT n" for n > 1 was not straightfor- ward, and those queries were left unchanged.
T1JD 12.rriend id)
('Brittany') ('John'))
No student
Figure 5: An example of question and SQL. pair with a uniqueness assumption on the "name" column, which is not supported by the schema. The SQL query the left does not make the same assumption.
For resolving ties introduced by an incorrect us- age of GROUP BY in benchmark queries, we in- cluded all non-aggregated columns from the SE- LECT clause in the GROUP BY clause. For exam- ple, if the SELECT clauses included id and name, but the GROUP BY clause only included name, we added id the GROUP BY clause. This change will not affect queries where there is a one-to-one mapping between and name, but will resolve the ambiguity when such mapping does not hold.
6 Experiments
To understand the extent at which the aforemen- tioned problems affect the benchmarks, our evalua- tion and the ranking of the models, we conducted three benchmarks: Spider, Spider-DK, BIRD. Our findings here apply the SYN dataset as well, which employs the same SQL queries as in the Spider dataset. For the same reason, we did not conduct a separate analysis of that benchmark.
three types of evaluations
With these two changes, 16% to 20% of the ref- erence queries in our benchmarks were affected. Under a perfect evaluation scheme, the accuracy should not be affected with these changes that sim-

---

Benchmark
Affected Queries
Exec Ace
Set Match Ace
two annotators for labelling. The annotators had access to the database schemas and were tasked with identifying the queries they deemed correct for each question, without knowing which model generated which query or the query was from the ground truth queries. Annotators could also create databases and validate queries. ensuring thorough evaluation.
Spider
206 (19%)
92.3
81.6
Spider-DK
112 (20%)
95
83.9
BIRD
252 (16%)
96.87
Table 2: Performance of the revised SQL queries on the development set of the benchmarks.
ply resolve the uncertainty. Table 2 displays both the execution accuracy and the exact set match ac- curacy for the reference queries from the BIRD, Spider, and Spider-DK benchmarks after our modi- fications. It's important to highlight that the perfor- mance metrics provided in this table encompass the entire development set of these benchmarks, com- bining both modified and unaltered queries. For clarity, in the Spider dataset, out of 1034 queries, 206 were modified. The performance assessment took into account mixed set of predicted queries: 206 that were adjusted and 828 that remained as originally presented. This culminated in an execu- tion accuracy of 92.3 percent.
Following our initial labelling process, we wanted to minimize the potential impact of human errors in our evaluation. For this, we identified queries with inconsistent labels among the anno- tators and presented them to the annotators. Each annotator was asked to provide an explanation for their assigned labels. In the final stage of evalua- tion, each annotator was presented the inconsistent queries and the explanations provided by the other annotator. They were then asked if they would revise their labels based on this additional informa- tion. The results of this experiment are presented in Table 3. This table presents the outcome of hu- man evaluation on a sample of 102 queries that both DIN-SQL and T5+PICARD methods were deemed incorrect in terms of execution accuracy. SQL experts conducted this evaluation, with 81.6% of these queries judged as correct for DIN-SQL. and only 25.5% for T5+PICARD. Notably, among the reference queries, only 67.3% were deemed cor- rect. Even after the second round of annotation, a few queries (more specifically, four question-queet pairs) still exhibit inconsistent labeling by the an- notators. The main challenge with these particular pairs is the inherent ambiguity in the questions the subjectivity of interpretations, which leads a lack of definitive answer. Figure 6 demonstrates one example of such a question with two possible SQL query as answers.
can be noted that the execution accuracy is not as adversely affected as the exact set match accuracy. We hypothesize that this could be at- tributed to the absence of ties in the test data used for these benchmarks. An evidence of this is the following two queries, (Q1) SELECT name, capacity FROM stadium WHERE average (SELECT maxerrg) FROM stadium), and (Q2) SELECT name, capacity FROM stadium ORDER BY average DESC LIMIT 1, labelled as a correct match by the test scripts of Spider.
6.2 Human Evaluation
To gain deeper understanding of the limita- tions within the benchmarks, we conducted an experiment focused on the widely-used text-to- SQL benchmark, the Spider dataset. Specifically, we evaluated two top-performing methods from the Spider leaderboard: DIN-SQL (Pourreza and Rafiei, 2023) and Ta-le + PICARD (Scholak et al., 2021). This experiment involved running these methods on the development set of Spider, which comprised 1034 question-quee query pairs. From the results obtained, we extracted the questions for which both methods failed to produce correct an- swer, based on the execution accuracy, resulting in 102 pairs. We then presented these questions, along with the SQL queries generated by the meth- ods as well as the ground truth SQL queries (treat- ing them the same as modell-generated queries), to
An intriguing observation emerged from this ex- periment: the DIN-SQL method, powered by GPT- 4, produced the highest number of correct answers, surpassing even the ground truth SQL queries. This finding sheds light on the limitations of the current benchmarks and raises doubts about the reliability of current leaderboards and performance metrics.
6.3 Error Analysis of Human Evaluation
We performed an error analysis of the SQL queries that were labelled as incorrect in our human evalua- tion to better understand the error types and causes and to provide insights into areas for improving the
'Tee human annotators are the authors of this paper.

---

What are the countries that have grealer surface any country Europe?
M7.219)
MD126%)
Model
M2(62%)
Schema
(50%)
M2(4409)
SELECT Name FROM country WHERE SurfaceArea max(SufaaceArea) FROM country WHERE Continent = Europe')
country SurfaceArea (SELECT min(SurfacsArsa) FROM country WHERE Continent 'Europe")
(A(1099)
GROUP BY (8%)
MO(50%)
Fallures
Cond
M2(60%)
MD(44HS)
(7%)
(19%)
Nested (15%)
M1
[TArube'). (Afghanlsstan', ('Angola), ('Anguillar)
M0(3399)
M0(26%)
M1(1119)
No country
M2(55%)
('Albenia'' J.
Figure 7: Distribution of SQL queries across error groups for the two models being evaluated and the ground truth. MO refers SQL queries the reference (ground truth) set, MI refers to the DIN-SQL. method, and M2 refers to T5+PICARD.
Figure 6: An example of a question with two possible SQL queries as the answers. Both these SQL queries correct under different interpretations.
Method DIN-SQL
Acc
Incon
group, where the tables and columns were incor- rect, in this group, the correct tables and columns used, but the conditions in the WHERE clause are erroncous. This error primarily manifested in the queries generated by the T5-PICARD method, but was also present in the reference set. The T5 model's 's tendency to introduce additional columns or omit necessary conditions could be attributed to its smaller size relative to larger models like GPT-4, limiting its grasp of intricate SQL syntax.
81.6
4
(Pourreza and Rafiei, 2023)
T5-large + Picard
25.5
4
(Scholak et al., 2021)
Ground Truth
67.3
4
Table Accuracy of the SQL queries generated by two methods and the ground truth SQL queries based on human evaluation. In four cases, the two annotators did not agree on label even after a second round.
The source of this problem is using a non- unique column for the nested SQL query, as also discussed in Section 5. Figure shows example of such error in SQL query. This more common in the SQL queries provided in the reference set as well as those of T5-PICARD.
Nested
ground truth SQL queries. Additionally, we com- pared the errors in ground truth queries with those of fine-tuning and prompting approaches. The iden- tified errors, categorized into five groups, are briefly discussed next. The distribution of SQL queries across these groups is depicted in Figure 7.
This category includes queries that incorrectly used GROUP BY, resulting in ambi- guity or uncertainty in the result as discussed Section 5. Notably, the reference set showed the largest number of errors, closely followed by the fine-tuned T5-PICARD, DIN-SQL exhibited the least number of errors.
GROUP BY
The primary issue responsible for the majority of errors, affecting both the reference SQL queries and the two methods, is the incorrect age of schemas, which arises when the SQL query utilizes incorrect tables or columns to answer the given question. These errors indicate ambiguities in the database schema and/or questions, as dis- cussed in Section 5. Notably, the reference set shows the least number of errors, which is closely followed by DIN-SQL.
Schema
As highlighted in Section 5, one of the eтror scenarios involves not properly handling ties when using the LIMIT keyword. The DIN-SQL method demonstrates lower incidence of this type of error, attributed to its prompting nature. Conversely, TS-PICARD exhibits identical perfor- mance to the ground truth in this particular case.
LIMIT
The second-largest group of errors observed pertains to the usage of incorrect condi- tions within the SQL queries. Unlike the schema
Condition

---

Benchmark SyntaxErr
UndFune
UndCol
Order By
Group By
Spider
4
69
211
2
51
Spider-DK
134
62
80
2
30
BIRD
5
103
4
20
Table 4: Breakdown of SQL errors observed in Spider, BIRD, and Spider-DK, following migration to PostgreSQL.
6.4 Standard SQL validation
accuracies presented in Table serve as a lose up- per bound for the achievable accuracy by models. is lose because our rewritings were unable to address cases that required manual intervention to reconstruct correct query. Thus, the upper bound is expected to be lower considering other issues such wrong assumptions the database content and ambiguity in schema matching.
We undertook an extensive review of the develop- ment set of Spider, BIRD, and Spider-DK bench- marks through the lens of standard SQL validation. The objective was to identify some of the problem- atic queries discussed Section and assess the portability of the benchmarks. As part this analy- sis, we migrated the databases and queries these three benchmarks from Sqlite to PostgreSQL. Our decision to use PostgreSQL, a widely recognized RDBMS, stemmed from its rigorous adherence to SQL standards. Following the migration, ecuted every query from the development set on these PostgreSQL databases, with keen focus on identifying queries that failed during PostgreSQL. execution. Table 4 provides a breakdown of queries by error type across all three benchmarks. Notably, errors such as UndefinedColumn, SyntaxError, and UndefinedFunction emerge due to the different SQL formats supported by Sqlite and PostgreSQL. These variances necessitate adjustments to make the queries compatible with PostgreSQL standards. For instance, the Spider dataset frequently show- cases errors stemming from PostgreSQL's strict typing conventions. While SQLite allows for com- parisons of int with text, PostgreSQL does not. Also, some queries run into problems because of SQLite-exclusing functions, such as strftime and iff, because PostgreSQL interprets literals in double quotations as column names.
Our human evaluation (3 6.2) further supports our claim and provides more insight into the limi- tations within one of the benchmarks studied. The results in Table 3 demonstrate that prompting meth- ods, such DIN-SQL, inherent limitations of the training set in the bench- marks. However, they are not fully immune be- cause of the few-shot input-output demonstrations that are taken from the train set. On the other hand, fine-tuned approaches, such as T5+PICARD, per- feetly mirror the distribution of errors seen in the ground truth queries for types nested, LIMIT, and GROUP BY. The largest number of wrong queries in schema and condition classes belong to fine- tuned model, due to inability of the model gen- erate correct SQL queries.
less affected by the
Conclusions
The reliance on standard text-to-SQL. evaluation metrics, namely exact set match accuracy and ex- ecution accuracy, has become less reliable the model performance approaches human-level per- formance. Our work is the first systematically study the limitations of these metrics and bench- marks through both human evaluation and query rewriting. Our re-evaluation of well-known bench- marks (Spider, Spider-DK, and BIRD) uncovers common systematic issues that affect the evalua- tion process and performance estimates, revealing that a significant portion of queries in the train and dev sets are impacted by these issues. Incorporat- ing multiple SQL queries as the ground truth and representing different interpretations of queries of- fer a promising solution enhance the evaluation process and achieve a more comprehensive and accurate assessment of Text-to-SQL models.
The two other types of failures, group by and Order by, included queries that introduced ambigu- ities to the benchmarks, as discussed Section 5. It should be noted that these benchmarks present a range of issues that are not solely confined to syn- tax. Challenges related to wrong assumptions on DB content and ambiguities in schema matching are notably pervasive.
7 Discussion
Our analysis (5 6.1) reveals the limitations of ma- jor text-to-SQL benchmarks, highlighting the fact that even with a perfect model, achieving a perfect accuracy on these benchmarks is not possible. The

---

Limitations
hanced text-to-sql model with mixed local and non- local relations. arXiv preprint arXiv:2106. 01093....
In this study, our focus was primarily on cross- domain text-to-SQL. benchmarks and models. The failure cases identified in this domain are likely to be present in other Nooaann-specinn text-to-SQL. benchmarks and models as well. It is essential to conduct further analysis to identify specific fail- ure cases within domain-specifie benchmarks and models.
Martin Davis. 2004. The undecidable: Basic papers on undecidable propositions, unsolvable problems and computable functions. Courier Corporation.
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward. Jinxia Xie, and Peng- sheng Huang. 2021a. Towards robustness of text- to-sql models against synonym substitution. arXiv preprint arXiv: 200.::0006.0665.
Furthermore, is worth mentioning that our work has a limitation regarding the analysis of failure cases that lack specific structure and re- quire manual effort for detection. Identifying and addressing such problems necessitates extensive work. The purpose of our study was to highlight these failure cases; more in-depth analysis of their prevalence can provide a clearer understand- ing of their impact on the overall performance of Sext-to-SQL systems.
Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. Exploring underexplored limitations of cross- domain text-to-sql generalization. arXiv preprint arXiv:2109.05157.
Wenqiang Lei, Weixin Wang. Zhixin Ma. Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. 2020. Re- examining the role of schema linking in text-to-sql. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6943-69555 6954.
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023a. Graphix-t5: Mixing pre- trained transformers with graph-aware layers for text- to-sq] parsing. arXiv preprint Xiix:3300.07500
Ethics Statement
In this paper, we acknowledge the importance of ethical considerations in conducting and present- ing our research. We affirm our commitment to comply with the ACL Ethics Policy and adhere ethical guidelines and principles throughout the entire research process.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang. Bowen Li. Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023b. Can Ilm already serve a database interface? big bench for large- scale database grounded text-to-sqls. arXiv preprint arXiv:2305.0311
We have taken necessary measures to ensure the privacy, confidentiality, and consent of individuals or entities involved in our data collection, experi- mentation, and analysis. Any personal or sensitive information used in this study has been appropri- ately anonymized and safeguarded.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. arXiv preprint a:Xiv:2012.16627.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. 2023. A comprehensive evaluation of chat- gpt's zero-shot text-to-sql capability. arXiv preprint N.iv:2303311547
Furthermore, we have made efforts to minimize any potential biases and discrimination in our re- search design, data selection, and interpretation of results. We have strived for transparency, accu- racy, and fairness in reporting our findings, and we have provided appropriate citations and acknowl- edgments to give credit to the work of others.
Mohammadreza Pourreza and Davood Rafiii. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint arXiv:234.11015.
By including this ethics statement, we aim to demonstrate our dedication to conducting research with integrity, respecting ethical principles, and contributing to the responsible advancement of knowledge in our field.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabil- ities of large language models. arXiv preprint 224.v:2204.00498
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. arXiv preprint
References
Jaydeep Sen, Chuan Lei, Abdul Quamar, Fatma Özcan, Vasilis Efthymiou, Ayushi Dalmia, Greg Stager, Ashish Mittal, Diptikalyan Saha, and Karthik
Ruisheng Cao. Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. Lgesql: line graph en-

---

Sankaranarayanan. 2020. Athena++ natural language querying for complex nested sql queries. Proceed- ings of the VLDB Endowment, 13( (3(12)2777722759
Ruoxi Sun, Sercan O Arik, Hootan Nakhost, Hanjun Dai. Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023. Sql-palm: Improved large language modeladaptation for text-to-sql. arXiv preprint arXiv:2306.00739.
Bailin Wang. Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. Rat-sql: Relation-aware schema encoding and linking for text- to-sql parsers. arXiv preprint arXiv: 19.4994..
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 18. Syntaxsqinet: Syntax networks for complex and gross-domaintexttooogg task. arXiv preprint arXiv:/810.05237.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, et 201 Spider: large-scale humen-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint S0Civ:18000086000
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jason Eisner. 2022. Active programming by example with a natural language prior. arXiv preprint arXiv:2205. 12422.
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020a. Semantic evaluation for text-to-sql with distilled test suites. arXiv preprint arX.v:2010.02840.
Victor Zhong, Mike Lewis, Sida Wang, and Luke Zettlemoyer. 2020b. Grounded adaptation for shot executable semantic parsing. arXiv preprint arXiv:2009.07396
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, Abs/1709000001.


Reviews:
[{"review_id": "VQWKkSgaJr", "reviewer": "Reviewer_ReVM", "summary": "", "questions": "1. Table 2 shows results with revised SQL queries. Comparing with unrevised queries should also be shown.\n\n2. Is query rewriting possible to automate for all types of queries or does it involve significant human expertise?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and would like to clarify the intent and limitations of our paper. Our primary objective was to provide a comprehensive evaluation of the existing Text-to-SQL benchmarks, highlighting the inherent issues within them.\n\nRegarding the creation of a modified benchmark, we can make two comments: (1) Updating these benchmarks to address the mentioned issues necessitates a substantial human effort. (2) Many benchmarks in the Text-to-SQL domain, such as Spider and BIRD, incorporate holdout test sets for official leaderboards and comparisons of text-to-SQL methods. Our access is limited to the development and training sets of these benchmarks, and we are unable to modify test sets. Thus, even updating the development and training sets alone would not fully resolve the underlying problems within these benchmarks, since the final performance comparison would be reported on the test sets with inherent issues.\n\nIn essence, our paper's aim was to draw attention to the issues within existing benchmarks rather than proposing a new benchmark. We believe that the responsibility for benchmark updates rests with the benchmark creators, and we encourage them to consider our findings when revising their benchmarks to ensure a more accurate evaluation of Text-to-SQL methods in the future.\n\nIn Table 2, we presented the execution accuracy and exact set match accuracy of the revised queries, using the unrevised queries as references. The purpose of this comparison was to highlight that even with correct SQL queries manually crafted by experts, we cannot achieve 100% accuracy on the benchmark tasks. In such a scenario, comparing the results with unrevised queries would not provide meaningful insights, as both exact set match accuracy and execution accuracy would be 100% for unrevised queries.\n\nAutomating query rewriting faces inherent challenges, particularly when dealing with failures stemming from schema ambiguity, erroneous assumptions about the database content, and the ambiguity of natural language utterances. These challenges arise because there is no specific structure to address the failures systematically. Successful query rewriting in these cases necessitates a deep understanding of table and column semantics to identify ambiguities and erroneous assumptions. In cases of ambiguity, human expertise is essential to disambiguate the context, as these situations often lack clear guidelines. Detecting erroneous assumptions often involves introducing new data to the database and meticulously reviewing and correcting failed queries on a case-by-case basis.\n\nWe hope this clarifies our paper's scope, and we welcome any additional suggestions or questions from the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper highlights the gaps in current Text-to-SQL benchmarks and categorically highlights the issues in the benchmarks. \n\n2. The paper would serve as an useful tool to the community in coming up with better benchmarks. ", "reasons_to_reject": "1. While the paper is useful in understanding the issues with Text-to-SQL, I was expecting modified benchmark would be shared with community. No such information is mentioned. \n", "questions_to_authors": "1. Table 2 shows results with revised SQL queries. Comparing with unrevised queries should also be shown.\n\n2. Is query rewriting possible to automate for all types of queries or does it involve significant human expertise?\n\n", "paper_topic_and_main_contributions": "In this paper, the authors undertake a thorough examination of the Text-to-SQL benchmarks, specifically focusing on SPIDER, SPIDER-DK, and BIRD. The study sheds light on various concerns pertaining to the evaluation of these benchmarks, particularly with regard to handling ambiguity in generating queries. By doing so, the authors aim to identify the primary reasons behind the decline in exact-set match accuracy and execution accuracies observed in state-of-the-art models.\n\nThe paper dedicates Section 5 to categorically highlight the underlying causes of inaccuracies, which include failures arising from ties in the output, ambiguity in schema matching, and erroneous assumptions about the database content. This in-depth analysis enables the readers to grasp the complexities associated with evaluating Text-to-SQL models.\n\nTo address the identified ambiguity, the authors propose insightful modifications to the development set, leveraging human annotations. The results demonstrate that state-of-the-art models perform remarkably well on these modified benchmarks, underscoring the effectiveness of the proposed changes in mitigating ambiguities.\n\nMoreover, the paper goes beyond the evaluation aspect and sheds light on the creation of Text-to-SQL benchmarks. By outlining the issues in their inception, the authors encourage the development of more robust and comprehensive benchmarks for future research.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The citations seem adequate.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper reads well. "}, "extra_text": {}}, {"review_id": "v4xU7uMpHu", "reviewer": "Reviewer_FJpx", "summary": "", "questions": "I mostly derive the questions from the points raised in the \"Reject\" section above:\n\nA. Please clarify the interpretation of the experiment reported in Table 2 - also provide examples (in Appendix if necessary) of the min/max/top transformation.  \n\nB. Please clarify the interpretation of the experiment with human evaluation reported in Table 3 - confirm these are accuracy figures over 102 samples selected to demonstrate the fragility of original Spider queries. \n\nC. What is your recommendation on how to improve Text-to-SQL datasets based on the analysis reported in the paper?  You mention adding multiple versions of SQL for the same question - but these SQL queries could return different result sets - is this your recommendation? Or would you prefer to disambiguate problematic cases based on inter-annotator agreement?\n\n[Post rebuttal]\nI acknowledge good answers to these questions.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review; your meticulous attention is greatly appreciated.\n\nTo address the systematic analysis issue raised by the reviewer, we conducted a comprehensive analysis of the Spider, BIRD, and Spider-DK benchmarks using standard SQL validation. Specifically, we migrated the databases of these three benchmarks from Sqlite to Postgres, a renowned RDBMS known for its strict adherence to SQL standards. Subsequently, we executed all development set queries on these Postgres databases, identifying queries that produced execution errors. Here are the results:\n\nSpider dataset: Total 1034 queries\n\nUndefinedColumn: 211\nUndefinedFunction: 69\nGroupingError: 51\nSyntaxError: 4\nInvalidColumnReference: 2\n\nBIRD dataset: Total 1534 queries\n\nSyntaxError: 134\nUndefinedFunction: 103\nGroupingError: 20\nDivisionByZero: 5\nInvalidColumnReference: 4\nDatatypeMismatch: 1\nCardinalityViolation: 1\nUndefinedColumn: 1\n\nSpider-DK dataset: Total 535 queries\n\nUndefinedColumn: 80\nUndefinedFunction: 62\nGroupingError: 30\nSyntaxError: 5\nInvalidColumnReference: 2\n\nMost of the observed errors, including UndefinedColumn, SyntaxError, and UndefinedFunction, stem from differences in the accepted SQL formats between Sqlite and Postgres, necessitating modifications to align the queries with Postgres standards. For example, queries in the Spider dataset often failed due to strong typing in Postgres (e.g., comparing int with text allowed in SQLite but not in Postgres), SQLite-specific functions (e.g., strftime, iff) and Postgres not accepting literals in double quotations, interpreting them as columns.\n\nThe issues related to InvalidColumnReference, GroupingError, and CardinalityViolation are primarily linked to the misuse of GROUP BY and ORDER BY statements, potentially leading to ambiguities. This experiment complements our paper's analysis, and we will be happy to add it to the paper. However, it should be noted that the benchmarks suffer from many issues not captured by a syntactic analysis, such as the problems with ties and ambiguities.\n\nRegarding the performance results in Table 2, we agree that the caption is not clear. The reported results are for the entire development set. For instance, in the Spider dataset, 206 out of 1034 queries underwent modifications. Performance evaluations considered a combined set of predicted queries, comprising 206 revised queries and 828 unchanged queries, resulting in an execution accuracy of 92.3 percent. Reviewer asked for some examples of the min/max/top transformation, below you can find some of them (we include these examples in appendix)\n\nOriginal Spider query:\nSELECT Name, SurfaceArea, IndepYear FROM country ORDER BY population LIMIT 1\n\nTransformed query:\nSELECT Name, SurfaceArea, IndepYear FROM country WHERE population = (SELECT min(population) FROM country)\n\nOriginal Spider query:\nSELECT name,  capacity FROM stadium ORDER BY average DESC LIMIT 1\n\nTransformed query:\nSELECT name, capacity FROM stadium WHERE average = (SELECT max(average) FROM stadium)\n\nRegarding Table 3, we agree that the caption should be improved. As noted, this table presents the outcome of human evaluation on a sample of 102 queries that both DIN-SQL and T5+picard methods were deemed incorrect in terms of execution accuracy. SQL experts conducted this evaluation, with 81.6% of these queries judged as correct for DIN-SQL, and only 25.5% for T5+picard. Notably, among the reference queries, only 67.3% were deemed correct. An important aspect is that annotators had access to databases to run queries and validate responses, ensuring a thorough evaluation.\n\nIn the context of enhancing the text-to-SQL dataset, our proposal involves using multiple SQL queries as references for a single question rather than making the SQL queries more deterministic and compliant. This approach aligns with real-world scenarios where question and database ambiguities are common. We advocate retaining the ambiguity in questions while addressing all interpretations to facilitate a fair comparison. However, there is no reason for a non-determinism behavior of ground truth queries in the benchmarks (e.g., those attributed to GROUP BY and ORDER BY), and we strongly agree that they should be fixed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. Good empirical analysis of popular text-to-SQL datasets showing their limit and calling for prudence when interpreting model accuracy.\n\nB. Empirical results demonstrating the potential level of noise in the dataset caused by systemic problems supported by human analysis.\n\nC. Suggestion to include multiple queries per question to address question ambiguity is well taken and will improve best practice.", "reasons_to_reject": "A. The analysis of the SQL \"quirks\" indicated in the paper relies on non-standard SQL implementation of SQLite which is used in Spider (mostly for convenience).  SQLite (as well as MySQL) has non-standard typing system which makes foreign keys in Spider quite random, it also fails to enforce many of Standard SQL semantic constraints.  For example - looking at https://blog.jooq.org/a-beginners-guide-to-the-true-order-of-sql-operations/ which lists many of the sources of non-determinism in SQL execution - the list includes:\n\n* In the presence of a GROUP BY clause, only expressions built from GROUP BY expressions (or functional dependencies thereof), or aggregate functions can be used in HAVING, SELECT, and ORDER BY clauses.\n* There are cases in which GROUP BY is implied. E.g. if you write a “naked” HAVING clause\n* A single aggregate function in the SELECT clause (in the absence of GROUP BY) will force aggregation into a single row\n* This implicit aggregation can also be implied by putting that aggregate function in ORDER BY \n* You can ORDER BY expressions that reference any columns from the FROM clause without SELECTing them. But that’s no longer true if you write SELECT DISTINCT\n\nNone of these constraints are enforced in SQLite - and all of them lead to non-deterministic output (that is, running the same query on different servers with the same schema and same content will output different outputs).\n\nGiven this observation - it would have been useful to propose a more systematic test of the Spider queries through \"standard SQL validation\" - including types, keys, and all forms of distinct, aggregate, order cases and correlated subqueries.\n\nIn other words, the work calls for a vigorous re-validation of the standard text-to-SQL dataset - but does not quite follow through with a systematic analysis.\n\n[Post rebuttal]\n\nThe preliminary analysis included in the rebuttal addresses this point convincingly - I recommend the authors include a deeper version of this analysis in the final version of the paper.  This addresses this point convincingly.\n\n\nB. I find the experimental analysis confusing: table 2 shows the impact of two semantic transformations (replace TOP 1 with embedded min/max queries and extend group by to dependent columns) - how many queries are impacted (up to 20%) and the execution accuracy and set match accuracy of the reference queries after these changes.  The figures are 92-97% for exec accuracy and 81-84 for exact match accuracy.  The fact that exact match accuracy is low when the query is modified should not be surprising - exact match measures query similarity.  I infer from this that only about 20% of the 20% impacted queries (about 4% altogether) are actually modified in a significant way.  Accordingly, the execution accuracy impact is 8% of the 20% impacted queries - which would be altogether 1.6% of the overall dataset.  I am not discussing here the importance of the finding - just the way it is presented which is hard to interpret.  The caption of the figure does not make it clear what the reported numbers describe and the text (425-430) is also hard to interpret.\n\n[Post rebuttal]\nThe clarification of this Table in the rebuttal is satisfying.\n\nC. Table 3 is also problematic - it shows the result of a manual analysis of 102 queries (line 453) out of the 1034 samples in the development set of Spider.  The caption of Table 3 does not mention the fact that this is sampled on only 102 queries - making the \"surprising result\" of the ranking reversal less convincing.\n\n[Post rebuttal]\nThe clarification of this Table in the rebuttal is satisfying.\n\nD. In the human evaluation, annotators had access to the schema and the queries - but apparently not to the results (392-398).  There is also no indication of the complexity level of the respective queries (Spider has 4 levels of complexity - simple to extra-hard).  These two factors are important to assess the reliability of the human judgment - because one can call into question the capability of programmers and non-SQL experts to assess the accuracy of a complex SQL query without looking at the execution results or experimenting with the query in an interactive manner (as supported for example in the motivation of the design of the multi-turn dataset SparC).\n\n[Post rebuttal]\nThe clarification of this point in the rebuttal is satisfying.\n\nE. The conclusion of the analysis (592-597) calls for incorporating multiple SQL queries in the dataset.  This is indeed a good direction - but another direction that should be considered is to fix the non-standard SQL queries in the dataset to make them more deterministic and more compliant.  A full analysis of leading models vs. such fixed queries over the whole dataset would be the most informative piece of data missing in this version.\n\n[Post rebuttal]\nThe clarification of this point in the rebuttal is satisfying.", "questions_to_authors": "I mostly derive the questions from the points raised in the \"Reject\" section above:\n\nA. Please clarify the interpretation of the experiment reported in Table 2 - also provide examples (in Appendix if necessary) of the min/max/top transformation.  \n\nB. Please clarify the interpretation of the experiment with human evaluation reported in Table 3 - confirm these are accuracy figures over 102 samples selected to demonstrate the fragility of original Spider queries. \n\nC. What is your recommendation on how to improve Text-to-SQL datasets based on the analysis reported in the paper?  You mention adding multiple versions of SQL for the same question - but these SQL queries could return different result sets - is this your recommendation? Or would you prefer to disambiguate problematic cases based on inter-annotator agreement?\n\n[Post rebuttal]\nI acknowledge good answers to these questions.", "paper_topic_and_main_contributions": "The paper provides an in-depth analysis of SQL queries in Text-to-SQL benchmarks (Spider and variants) and demonstrates that systemic semantic aspects of the SQL queries make the benchmark ambiguous or fragile in about 10% to 20% of the samples in the dataset.\n\nThe specific analysis relates to:\n\n1. Aggregates (group by) where the list of fields that are aggregated does not include all selected fields.  (For example, \"select id, name from T1 group by name\").  \n\n2. Extrema queries (min, max) implemented through LIMIT (TOP n) - in these cases, depending on available data in the tables, TIEs may change the returned values.\n\n3. Ambiguity in schema matching - when multiple columns (sometimes in different tables) may correspond to the focus of a question - this leads to systemic ambiguity in interpreting the meaning of the question.\n\n4. Wrong assumptions on db content - for example, assuming that all rows in a table satisfy a condition (which may even be true given a snapshot of the content) but which does not correspond to a generalizable query.  (Example with \"petType = 'dog').\n\nExperiments are run to assess the impact of these phenomena in execution match metrics for Spider variants (Spider, Spider-Syn, Spider-DK, BIRD).  They show that up to 20% of the samples may be affected by these considerations - and that models that \"overfit\" to the interpretation embodied in the Spider gold SQL fail to produce correct execution results when the query is disambiguated or simply modified syntactically in a way that should be neutral to semantics.\n\nThese factors are shown to lead to \"leaderboard reversals\" - where some models rankings are changed when considering query rewriting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7ly65yyJuM", "reviewer": "Reviewer_5LcL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your valuable feedback. We appreciate your insights regarding the evaluation methods.\n\nIn addition to drawing attention to the issues within existing benchmarks, we proposed a novel approach to address the evaluation challenges in text-to-SQL models. Specifically, we suggested using multiple SQL queries as references, which can better capture the inherent ambiguities present in natural language questions. This approach reflects the complexities of real-world scenarios, where databases often contain numerous tables and columns with various interpretations.\n\nWe believe that by incorporating multiple SQL queries as references, researchers and authors can ensure a fairer and more comprehensive assessment of text-to-SQL methods. This approach enables a more accurate evaluation, considering the diverse interpretations that users might have when querying databases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The workload of this paper is quite heavy and the work is meaningful.", "reasons_to_reject": "-They point out the issues with the existing evaluation methods, but not propose a new method to assess text-to-SQL models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors study the limitations of text-to-SQL evaluation metrics and benchmarks through both human evaluation and query rewriting. They also summarize issues that affect the evaluation process and performance estimates.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers liked how the paper highlights current issues with existing text-to-sql benchmarks through experiments and human analysis. The paper might have created expectations for new benchmarks and evaluation metrics that it did not deliver, but also never claimed to. Other concerns have been addressed in an exemplary fashion by the authors during the rebuttal.
Decision: Accept-Main