Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness
Authors: Shuaichen Chang
Jun Wang
Mingwen Dong
Lin Pan
Henghui Zhu
Alexander Hanbo Li
Wuwei Lan
Sheng Zhang
Jiarong Jiang
Joseph Lilien
Steve Ash
William Yang Wang
Zhiguo Wang
Vittorio Castelli
Patrick Ng
Bing Xiang
Venue: ICLR 2023 notable top 5% (2023)

Abstract: Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries. However, recent studies reveal that text-to-SQL models are vulnerable to task-specific perturbations. Previous curated robustness test sets usually focus on individual phenomena. In this paper, we propose a comprehensive robustness benchmark based on Spider, a cross-domain text-to-SQL benchmark, to diagnose the model robustness. We design 17 perturbations on databases, natural language questions, and SQL queries to measure the robustness from different angles. In order to collect more diversified natural question perturbations, we utilize large pretrained language models (PLMs) to simulate human behaviors in creating natural questions. We conduct a diagnostic study of the state-of-the-art models on the robustness set. Experimental results reveal that even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop on the most challenging perturbation. We also present a breakdown analysis regarding text-to-SQL model designs and provide insights for improving model robustness.

Published as a conference paper at ICLR 2023
DR. A DIAGNOSTIC EVALUATION BENCH- MARK TOWARDS TEXT-TO-SQL ROBUSTNESS
Shuaichen Chang Jun Wang', Mingwen Dong", Lin Pan", Henghui Zhu", Alexander Hanbo Li, Wuwei Lan', Sheng Zhang", Jiarong Jiang", Joseph Lilien?, Steve Ash', William Wang', Zhiguo Wang", Vittorio Castelli", Bing Xiang", Patrick Ng²
Ohio State University, AWS Labs chang. 1 69 1022osuu.edu {Juwanga, zhiguow, (ttricnn)@amazon.c.
ABSTRACT
Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries. However, recent studies reveal that text-to-SQL. models are vulnerable to task-specific perturbations. Previous curated robustness sets usually focus on individual phenomena. In this paper, we pro- pose a comprehensive robustness benchmark based on Spider, cross-domain text-to-SQL. benchmark, to diagnose the model robustness. We design 17 pertur- bations on databases, natural language questions, and SQL queries to measure the robustness from different angles. order to collect more diversified natural ques- tion perturbations, we utilize large pretrained language models (PLMs) to simulate human behaviors in creating natural questions. We conduct diagnostic study the state-of-the-an models the robustness set. Experimental results reveal that even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop the most challenging perturbation. We also present breakdown analysis regarding text-to-SQL model designs and provide insights for improving model robustness.
INTRODUCTION
Large-scale cross-domain text-to-SQL datasets facilitate the study of machine learning models for generating SQL query given natural language question (NLQ) and corresponding database (DB) as input. Neural text-to-SQL models encode NLQ and DB schema and decode the corresponding
SQL (Wang et al. 2019 Lin et al. 2020 Scholak et al. 2021), which have achieved remarkable results on existing benchmarks (Zhong et |017 Yu et al. 20188 Shi et al.] (020)). However, those results are obtained in the setting where test data are created with the same distribution as training data. This setting prevents the evaluation of model robustness, especially when the data contain spurious patterns that do not exist in the wild. For example, previous studies (Suhr et al. 2020 Gan et al.) [021aa] Deng et al.] 2021) I) have found spurious patterns in the Spider (Yu et al.) 2018). widely used cross-domain text-to-SQL benchmark, such NLQ tokens closely matching DB schemas, leading models to rely on lexical matching between NLQs and DB schemas for prediction instead of capturing the semantics that the task is intended test.
Figure D shows examples where the state-of-the-an art (SOTA) text-to-SQU models are vulnerable to perturbations. (1) DB perturbation: replacing the column name wi vine name with champ- name leads the model to miss the intent of "3 youngest winners": (2) NI.Q perturbation: the model confuses the selected column rinner_name with winner_age given a paraphrased NLQ which uses "Who" to imply the selected column; (3) SQL perturbation: simple change to the number of returned items (from LIMIT
LIMIT 8) fails the model to detect the right intent. Recent studies created data to reveal the robustness problem of text-to-SQL. models via perturbing DBs or NLQs MM & Wang 202T; Pi et al.) 2022, Gan et al.) 2021a Deng et al. [021). However, they usually focus individual linguistic phenomena and rely rule-based methods few
Work done during internship at AWS Al Lahs OOur
data and code are available di lagnosticc robustness--eeeee teatssttnees--t--ttt-----
https: //gi ht//gittuu thub. www.b.como///....ooo com/aws labs/

---

Published as a conference paper at ICLR 2023
Pre-perturbation
Ranking
DB Perturbation
Ranking
Ranking
Find the name and rank of the youngest winners across all matches.
Player First
Find the name rank the 3 youngest winners across all matches.
Winner WemeeWnner
Loser Name
Winner Rank oser Name
Seawss Wilaare
Sinvra Hoep
Swewr lens
Sinera Help
SRIRCC n2. c2-mh nome, #rom ma matches as D1 Join playara as T3 on fd p3.player_ _ 1
points
SELECT w wine name, winner _anner_rank rank matchea ORERR BY winner age LIKIT
FROM
Correct
Incorrect:
about youngest
NLQ Perturbation
Benkin Pale
Rankin
SQL Perturbation
Backin Date
Ra Name
Who are the three youngest
Player Find Name
Find the name and rank of youngest winners
Player Name
Winner NameWeneer Rank Loser Name
Winner Name Winnie Rand
oser Name
winners all matches?
Sanasa WiBlrro
Senura Hales
Saer win
Snnr Hikp
And their ranks?
across all matches.
Picard
Picard
SELECT
Winner age,
winner rank FROK matches LIMIT
SELECT winner name, winner rank FROM
WHERE winnez
Incorrect select winner_age insteed of winner name
Incorrect: predict age= instead of LIMIT
Figure I: An example of the SOTA model Picard (Scholak et al. 2021 against DB, NLQ, SQL perturbations on the database WTA. Picard predicts correct SQL on pre- pre-perturbation data but fails on post-perturbation data. The blue and gray areas highlight the modification on input and the errors of predicted SQLs respectively.
annotators, which cannot cover the richness and diversity of human language. For example, the NLQ perturbation example in Figure requires sentence-level paraphrasing, which cannot be generated by previous perturbation methods.
In this paper, curate comprehensive Diagnostic Robustness evaluation benchmark (Dr.Spider), based on Spider (Yu et al.) 2018)) via 17 perturbations to cover all three types of robustness phenom- ena on DBs, NLQs, and SQLs. Dr. Spider contains 15K perturbed examples. We perturb DBs with set of predefined rules by taking advantage of their structural nature to represent data in differ- ent ways. For NLQ perturbations, we propose a collaborative expert-crowdsourcer. by prompting the pretrained OPT model (Zhang et al. (2022) to simulate various and task saekispecific linguistic phenomena. For SQL perturbations, we programmatically modify the local semantics in SQLs and their corresponding tokens in NLQs while minimizing the surface-level changes to measure model robustness to local semantic changes.
framework
We evaluate the SOTA text-to-SQL SQL models 2021 Yu et al.) 2021 Scholak et al. 202T). The experiments demonstrate that although the SOTA models achieve good performance
benchmark (Wang et al. 2019 Rubin & Berant
the original data, they struggle have consistently correct predictions in our robustness sets. Even the most robust model suffers from a 14.0% performance drop overall and 50.7% performance drop against the most challenging perturbation. We also present a breakdown analysis of model robustness in terms of model architectures, including model size. decoder architecture, and the entity- component. We analyze the advantage of different model designs, which provides insights for developing robust text-to-SQL models in the future.
2 RELATED WORK
Robustness in NLP
The evaluation of model robustness an important step toward the develop-
ment of reliable models. (Nie et al. 2019 Wang et al.) 2021 Goel et al.) 2021). Dia & Liang (201 7); Iyyer et al. 2018), Wang et al. (202 measure model robustness against semantic-presse preeervving pertur- bations, and Kaushik al. (219); Gardner et al. (2020) evaluate the decision boundaries of models against semantic- semantic-changing perturbations, which change local semantics while minimizing modifica- tions in the surface-leve patterns. Previous studies robustness data for text-to-SQL only consider semantic- semantic-preserving preserving perturbations on DB or NLQ and rely on rule-based methods or handcrafted examples, which limits the naturalness and diversity of perturbations (Gan al. 2021a b Deng
2

---

Published as a conference paper at ICLR 2023
perturbations (on DB and NLQ) but also cemantic-changinn perturbations (on SQL) to evaluate the robustness of models in distinguishing closely related but different semantics.
et al. 2021 Pi et al. 2022 Our benchmark not only contains
Data Generation with Pre-trained Language Models Large pre-trained language models (PLM), such as GPT-3 (Brown et al.) 2020), OPT (Zhang et al. 2022). PaLM (Chowdhery et al. 2022. have achieved human-level pertormance on many NLP tasks. Recently, they have been used to generate data with prompts that consist of few similar examples (Schick & Schütze 2021 Lee et al. 202 2022: Liu et al., 2022). Liu al. (2022) has motivations that most parallel ours, which uses GPT-3 to generate a new data example with several existing data examples that share a similar pattern. However, instead of generating a new example, we leverage PLMs to paraphrase existing NLQs to create task-specific perturbations.
3 DATA CREATION
Dr.Spider aims comprehensively evaluate the robustness of models with perturbations each component of the text-to-SQL task, i.e. DB, NLQ, and SQL. We apply task-specific perturbations create our benchmark based on the Spider development set, the Spider test is not public. Dr.Spider contains 3 DB perturbation test sets, 9 NLQ perturbation test sets, and 5 SQL perturbation test sets simulate various task-specific phenomena. Each test contains parallel pre-perturbation and post-perturbation data to measure model robustness against the perturbation. In total, Dr.Spider contains 15K pairs of pre- prepperturbation and post- pesttperturbation examples.
Data Creation Principles We create our data following three principles. Task Specificity: Huang
et al.] (202T]: Ma & Wang ) y uses general perturbations in NL.Qs, such as word substitution and back translation, making difficult to provide fine-grained analysis in addition the general robust-
ness problems found in other NLP tasks (Nie et al. 2019 Ribeiro et al., 2020 Goel et al.) 2021).
We curate our NLQ perturbation data within 9 task-specific categories to reveal unique robustness challenges in the text-to-SQI task. Linguistic Richness: Previous studies rely on rule-based word- level perturbations (Gan et al. 2021a] Ma & Wang. 202I) or manual perturbations provided by a few experts (Gan et al. 2021b Deng et al. 2021)) which cannot provide high degree of linguistic richness. For example, "teachers whose hometown are" is perturbed to "teachers whose birthplace are" with word-level perturbation (Gan et al.] (021a), however, other common linguistic phenomena are not covered in previous data such as the sentence-level perturbation "teachers who were born in" from Dr.Spider. We leverage large pre-trained language models (PLMs) for NLQ perturbations instead of relying on rule-based methods or few human annotators. Diagnostic Comprehensive- ness: For systematic evaluation, a benchmark should provide various diagnostic angles. Previous work only considers cemaatti-preserving perturbations on either DBs or NLQs with few perturba- tion methods, such as replacing the database schema with synonym (Pi al. 2022) and replacing schema indicator in NLQs with synonyms Gan et al. (202 aa). We apply various perturbations, in- semantic-preserving preserving perturbations DBs and NLQs and cemantic-changeen perturbations to SQLs, to curate 7 perturbation test sets focusing on different robustness phenomena.
cluding semantic-
3.1 DATABASE PERTURBATION
Database perturbation aims to simulate the scenario where data can be represented in different ways in DB. We consider three DB perturbations: synoona, and co column-egu hawm-eeuiialleeeeeeeeeeeeee ianuvvalens lence.. Schema- its synonym. For instance, the column named country can be replaced with nation. achema-abbreviatee
replaces the name of column with
replaces the name of a column with its abbreviation. For example, the column named wwwnkkinn--iiitt can be substituted with rank-pts. First, we create a schema synonym and abbreviation dictionary for each database. We use the synonym dictionary in Pi et al. (2022) and obtain an abbreviation dictionary from a public abbreviation website Unlike sample subset from
Pi al. (2022) who replace all possible columns with their synonyms, the synonym and abbreviation dictionary to replace the sampled columns with their synonyms or abbreviations. The samplings are repeated five times to create diverse perturbations with duplicate samples removed. Finally, we programmatically modify the database and affected SQLs to generate
hhtpp:/wwww.alllccooms.ccmm
3

---

Published as a conference paper at ICLR 2023
new data based sampled perturbations each time. As Ma & Wang (2021) discovered that models are generally robust to DB perturbations that do not affect gold SQLs, we only select SQLs that mention any replaced columns in perturbations.
errrontent-equivaleneee simulates the scenario that data can be represented in different for- mats in a database by replacing a column (column name and content) with one multiple semantic- equivalent columns. We first ask task experts to provide possible semantically equivalent substitu- tions for each column. (1) For text column content, we consider to split compound column into multiple columns, such as replacing ful Lame with firstname and lastname. (2) We con- vert columns with fixed string values into boolean column(s) or vice versa. For instance, boolean column is male is equivalent to a text column sex with all gender options, and a text column degree is equivalent to three boolean columns is bachelor, is master, and is PhD when there are only three degrees in the database. (3) For numerical columns, we replace them with other numerical columns with equivalent semantics. For example, the column bi birthyear is semanti- cally equivalent to the column age since one can be inferred from the other. Similar to the other DB perturbations, we sample a subset from the substitutions at most times to ensure perturbation diversity and programmatically modify SQLs according to the changes in DBs. Note that remove SQLs that require unseen grammar to modify according to the database perturbation. To the best of our knowledge, we are the first consider representing database content in different ways create database perturbations.
3.2 NATURAL LANGUAGE QUESTION PERTURBATION
We propose a collaborative expert-crowdsourcee by leveraging the advantage of each: experts (we) provide task-specific analysis, crowdsourcers have natural and diverse language patterns, and Al models are scalable. Our framework involves the following phases: (1) Crowdsourcing annotators from Amazon Mechanical Turk I paraphrase sampled questions from the Spider dataset. (2) Task experts review the paraphrases and categorize them from the text-to-SQL task perspectives. (3) A pretrained language model (PLM) generates categorized paraphrase perturbations based on the collected human paraphrases in each category. (4) A pretrained natural language inference (NLI) model and rule-based filters are used to select factual paraphrases in each category. (5) Task experts review the paraphrased questions in the end to ensure the high quality of our dataset.
framework collect task-specific perturbations
Crowdsoureing Paraphrase Study NLQs in Spider were collected in a setting that users (ques- tioners) are familiar with SQL and have the access to the DB schema and content. However, in reality, the general users are not familiar with DBs or SQLs in most applications (Lee et al., 2021a). To understand the linguistic phenomena that NLQs could have in a realistic scenario, we recruit annotators on Amazon Mechanical Turk to paraphrase 100 sampled questions in Spider and collect five paraphrases per question. We encourage annotators create diverse paraphrases by not directly providing the database schema or content to the annotators. Instead, we provide a short descrip- tion of each database, which reveals the topic of the database and the key tables and columns in the database. Note that we omit the description of naturally occurring tables and columns (e.g. a student table contains columns name, grade) because they do not provide the necessary infor- mation for understanding the database context and also make annotators tend to follow the structure of schema during paraphrasing.
Expert Analysis We find only 270 out of 500 paraphrases are factual as a small change to a ques- tion could alter its meaning in the text-to-SQI. task. We remove nonfactual paraphrases and analyze linguistic phenomena in crowdsourcing paraphrases regarding how the paraphrase is different from the original question. We focus on three types of tokens in NLQ that can be aligned with SQL: SQL keyword indicators, column name indicators, and value indicators, where SQL keywords include sort keywords, comparison, and aggregation operators; column names refer the mentioned col- umn in SQL except for JOIN clause, and value tokens include database content and other numbers.
We find 9 categories based whether the paraphrasing contains a task-specific linguistic phe- nomenon, including 8 categories that modify the aforementioned indicators and one category that does not, T shows the categorized examples. Note that a paraphrase may fall into multiple cat- egories if it contains multiple phenomena. The category bultitype also include such examples.
hhtps//wwww.mur.ccom/
4

---

Published as a conference paper at ICLR 2023
We believe that the first categories can diagnose models against each task-specific NLQ perturba- tion, and we regard the last category as a control group to evaluate the robustness of models against general syntactic perturbations in sentence-level paraphrasing.
Paraphrase Category
Example
Scyword-synonye (Replace
Question: What is the code airport that has the highest number of flights?
SQL keyword indicators with synonyms)
Paraphrases: Show me the code for the airport that currently has the most flights.
Keyword-carries (Imply SQL. keyword indicators by car- rier phrases)
Question: Show the name and theme for all concerts and the number of singers in each concert.
Paraphrase: List the names and themes for all concerts and how many singers are in each.
Columm-synonym (Replace
Question: List the name teachers whose hometown not Little Lever Urban District.
column indicators with syn- onyms)
Paraphrases: Find the name of teachers who were not born in Little Lever Urban District.
Col column-carries (Imply col- umn indicators by carrier phrases)
Question: Show the name of teachers aged either 32 33? Paraphrases: Which teachers are aged either 32 33.
Column-at triute (1mply column indicators by aggregated attributes)
Question: What is the name of the conductor who has worked the great- est number year?
Paraphrases: Who has worked longest as conductor?
Col vilumnvaaluse (Imply column indicators by values)
Question: What are the ids of students who not own cats as pets? Paraphrases: Find the IDs of students who don't 't own cats.
Value- synoonym (Replace value indicators with synonyms)
Question: Find airlines that have at least flights. Paraphrases: Show the number of airlines that have at least ten flights.
Multitype (Contain multiple
Question: Find number of pets owned by students who are older than 20?
phenomena above)
Paraphrases: How many pets are owned by students over 20?
Others
Question: what is the name and nation of the singer who have a song having 'Hey' in its name?
Paraphrases: Which singers have 'Hey" in their song's name? List their name and nation.
Crowdsouring paraphrase categories and examples. The red tokens in the original ques- tions highlight the eyyoor/cclumn//alue indicators that are paraphrased, and the blue tokens in the paraphrases represent how the indicators are replaced or implicitly mentioned.
Table I:
Categorized Paraphrase Generation Although crowdsourcing paraphrases can be used as NLQ perturbations, it is costly to scale and categorize them for curating large-scale dataset. To tackle this issue, we generate paraphrases with large PLM and prompt it with crowdsourcing paraphrases in each category. We choose the OPT (Zhang et al. 2022) with 66B parameters as the PLM model. We manually select five examples from the crowdsourcing paraphrases in each category to form the in-context examples as the prompt prefix.
For each category, the input of OPT consists of category speccffcc instruction, 5 question-parappe paraphrase pairs in the category, an NLQ that needs to be paraphrased, and its SQL tokens (keywords/column: s/values) whose indicators are required to be modified in the category. Inspired by Wei et al. (2022); Paranjape et al. (02I). ) we employ the OPT model to generate an explanation along with the para- phrase. We overgenerate 20 paraphrases per question to create diverse paraphrases (however, most 5 examples will be kept after the filtering stage). Our prompt prefixes and implementation details can be found in Appendix A.2
Paraphrase Filtering and Quality Control After generating paraphrases in each category, first remove the duplicate questions and use BART-large model (Lewis et al.) 2019) trained on Multi-NLI (Williams et al. 2018)) filter out nonfactual paraphrases. Then, simple and high- recall rule-base based classifier is built to remove out-of-category paraphrases based on the alignment between SQI. tokens and their indicators in NL.. In particular,
the annotations from Lei
5

---

Published as a conference paper at ICLR 2023
et al. (2020) to build the column and value indicators. And to obtain SQL keyword indicators, collect the common indicators (covering over 80% examples) of each SQL keyword in the NLQs and programmatically label NLQs with those indicators. The common indicators of SQL keywords can be found in Appendix A.3 Finally, we filter out the paraphrases where all the indicators for SQL keywords, columns and values remain the same as the original NI.Q except category others.
To ensure the data quality in Dr.Spider, we request three annotators with SQL expertise to carefully review paraphrases with gold SQL labels and select the final data. We split the filtered data into small chunks and each chunk is reviewed by at least one annotator. 63% generated paraphrases are labeled factual and belong to the correct category, while only 54% of human paraphrases from the previous crowdsourcing are actually factual. This shows the effectiveness of our proposed frame- work for generating categorized paraphrases. 5% data (evenly from 9 categories) are reviewed by all three annotators. The Fleiss Kappa score between the three annotators is 0.61. Randomly sampled generated paraphrases in each category can be found in Appendix A.3
3.3 SQL PERTURBATION
SQL perturbations evaluate models' robustness against local semantic changes. We consider five perturbations regarding SQL tokens: (I) Comparison replaces a comparison operation from <<,>,<,>><) to another; (2) Sort-order switches ascending sort and descending sort; (3) NonDB-number replaces a non-DB number n with another number in [n - 10, n + 10), includ- ing the number in the LIMIT clause (e.g. Limit n) and criteria about counts (e.g. HAVING count (+) > n); (4) DB-text replaces a mentioned DB content text to another in the same column; (5) DB DB-number replaces mentioned DB content number to another number in [m- - 10, m + 10). Similar to DB and NLQ perturbations, we perturb an example at most five times to increase the diversity of perturbations. Examples for each category can be found in Appendix A.4
Since SQL. perturbuubaa involve modifications on both SQL tokens and their indicators in NL.q,, we want to minimize the surface-level changes introduced to NLQs disentangle SQL perturbations from NLQ perturbations. For comparis aaparisson and sort-ordez we replace an indicator in NLQ only with common indicators in the Spider training data. For example, an NLQ "...more than one car" corresponds to the SQL. "....... count (+) >1 We perturb the SQL by replacing operation ">" with ">=" as well as the NLQ indicator "more than" with "at least". We choose "at least" rather than uncommon indicator phrases (e.g. "no less than"") because SQL perturbations focus on evaluating models against local semantic changes on SQLs and replacing the SQL keyword indicator "at least" with its synonym "no less than" is covered in NLQ perturbation keyword-synoe synonnym. More details about the replacement rules can be found in Appendix [A.4] For number perturbations, mentioned number in NI.Q will only be replaced with another number in the same format, e.g., 3=>4, 3th, and three= four, and we skip numbers and 1 since they usually have non- number tokens indicators in NLQ. We only consider the DB content text that is mentioned in the exact same format in NLQ to disentangle it from NLQ perturbation va value weluussyynn synonym.
3.4
DATA STATISTICS
Perturbation Data
TS
I.
DC
DB Perturbations
NI.Q Perturbations
SQL Perturbations
Huang et al
2021
x
Ma & Wang 2021
v
Pi
2023
al. 2021a
Gan al. 20216)
x
5
Deng al
2021
Dr:Spider
9
5
Table 2: Perturbation attribute and the number of perturbation types for DB, NLQ, and SQL in existing studies and Dr.Spider. TS, LR, and DC represent task-spec,,,ccty, linguistic richness, and diagnostic comprehensiveness, respectively.
Shooss the covered principles and the number of perturbation types for DB, NLQ, and SQL in previous work and ours. Dr.Spider is the only benchmark covering all principles and perturbing

---

Published as a conference paper at ICLR 2023
all DB, NLQ, and SQL. Sp.der also contains various task-specific paraphrases for NLQ pertur- bations which contain a higher degree of linguistic richness than word-level perturbation methods. Similar to ours, Ma & Wang (2021) consider various formats to represent data in a DB. However, their perturbations require NLQ and SQL not to mention the perturbed DB columns, which fail to detect the lack of robustness of models (2.5 points performance drops average). We report the number of examples in Dr.Spider and the original Spider development set in the column # in Table 3 The number represents the size of each post-perturbation test data in Dr.Spider. To measure the naturalness of our generated NLQ perturbations, we sample 500 questions from each of the original Spider development set, human paraphrases (we use all 270 factual paraphrases), and our perturbed questions. Crowdsourcing annotators are hired to score the clarity and flueney of sampled questions from 3. Each question is evaluated by five annotators. The clarity scores of Spider questions, human paraphrases, and generated paraphrases are 2.7, 2.6, 2.6, and fluency scores are 2.7, 2.6, 2.7, which shows generated paraphrases have similar fluency and clarity to the original questions and human paraphrases. The annotation interface can be found in Appendix A.3
4 EXPERIMENTS
Models
The SOTA text-to-SQL. models follow an encoder-decoder framework that encodes an QLQ and a DB schema jointly and decodes the corresponding SQL. We evaluate multiple represen- lative text-to-SQL SQL models on Dr.Spider, which are trained on the Spider training set: (1) RATSQL (Wang et al. 2019): The encoder pretrained with BERT-large (Devlin et al., 2018)) and augmented with relation-aware attention (Shaw et al. 2018), and the decoder is a down docoding with ab-
stract syntax tree (AST). We use the implementation from [Scholak et al. (2020). (2) GRAPPA (Yu et al. 2021): RatSQL with GraPPa embedding as pretrained encoder. GraPPa finetunes RoBERTa- large (Liu et al. 2019) on synthetic table-text pair data, (3) SMBOP (Rubin & Berant 202T): An
encoder-decoder model with the same encoder GRAPPA and a bottom-up decoder with AST. (4) T5-FAMILY (Raffel et al. 2020): We use T5-BASE. T5- T5-LARGE, and T5-3B which are fine-tuned on Spider. (5) T5-3B LK (haw et al. 2021): A T5-3B model with an entity linking between ques- tion and DB content. (6) PICARD Scholak et al. (201) : T5-3B LK with a constraint decoding. It is the SOTA model on the Spider benchmark. (7) GPT-3 CODEX Chen al. 2021): A large language model without finetuning on Spider. We present the evaluation of CODEX in Appendix B
Robustness Evaluation Metrics We follow the Spider Benchmark to use the two SQL evaluation metries evaluate the correctness of predicted SQL with gold SQL: exact set match (EM) measures a predicted SQL with gold SQL on each clause ignoring values; execution accuracy (EX) compares the denotation answers from a predicted SQL and a gold SQL. We use EX as our main evaluation metric it evaluates the correctness of SQL values, which is an important part of model robustness, while we only report EM for GRAPPA which omits value predictions.
To evaluate robustness, report 3 metrics: (I) pre-perturbation accuracy: the accuracy pre- perturbation data, (2) post-perturbation accuracy (or absolute robustness accuracy): the accuracy on post-perturbation data, (3) relative robustness accuracy: The ratio of the number of correct predictions on both parallel pre-perturbation and post-perturbation data over the number of correct predictions on pre-perturbation data. While the post-perturbation accuracy measures the absolute performance of a model on perturbation data, relative robustness accuracy is to evaluate the regres- sion rate of models against perturbation, which more precise to compare the robustness of models by considering their performances on the pre-perturbation data.
RESULTS
5.1 DIAGNOSTIC RESULTS
Table the execution accuracy changes of the SOTA models against perturbations, where we report the macro-average scores for all perturbations. The EM accuracy can found Appendix B In general, all models suffer from significant performance drops in DB and NLQ perturbations. Picard is the most robust model but still has 30.0% and 14.5% relative performance drops. The wewponttettteutiiaaeeeaee
most challenging perturbation for DB is
Even PICARD faces 50.7% performance drop. We believe it is because the way that schemas and contents are mentioned in NLQs is different from that in DB, which requires additional common-sense reasoning to align
7

---

Published as a conference paper at ICLR 2023
Perturbation
*
RAYSQL
SMBlOP
TS-BASE
TS-LARGE
TS-38
T5-38 LK
PICARD
Spidkr-dvv
1.034
72.8
78.0
$7.0*
67.2
71.7"
74.4
79.3
Schema synotym
2.619
65.7
45.4
20.8 53.
48.3 25.2
39.836688
64.2 41.6
06.4 46.9
73.0 56.5
DR
Schema-athreviation
2,53
67.0
44.2
72.2-201
48.5-256
61.2-39.2
66.0-50.7
69.5-33.3
74.9 64.7
DBcossen--euivvllsee
382
79.8 12.0
51.2-572
560-17.5
71.5 34.0
78.5-36.4
84.6
88.7
Averge
70.8-
74.7-50.
51.1-223
64.0-36.7
69.5
73.5-
78.9-
Key synonym
953
69.9 69.9-53.7
75,9 61.3
53.9-55x 45.8
66.5 66.5-57.7
57.7
68.3 60.3
70.2 62.6
22.6 66.3
Keyword-carier
309
83.2
8277-79.2
68.9-66.2
74.9-70.9
82.2-76.9
82,7-76,4
85.0 82.7
synomm-
563
63.8 42.6
65.9-48.77
45.3-28..
54.5-40.1 40.1
63.1 46.5
63.9 51.3
71.0 57.2
NLQ
Colume-carier
579
58.0
85.1 65.1-64.
72.9-57.3
76. 76.7-59.6
83.1 83.1-61.7
86.9 64,9
Column attiibue
119
53.8
42.9
75.6 58,0
52.1-40.3
40.3
47.9 12.9
53.8-52.1
49.5 18.7
58.8 56.3
Colame-value
304
73,7-526
29.6-589
52.-30..
64.1-45.4
66.1-500
58.6
82,9-69. 69.4
Value-yyweyy
506
62.3 18.6
68.6-29
45.8-26.3
60.5-36.6
64.0-15.8
68.6 68.6-46- 46.4
72.5 53.0
Multitye
1,351
70:2-10.7
76.2 16.1
51.4-33.
63.8 44.0
47.8
70.1-51.1
24.4 57.1
2
67.2
58.0 51.3
75.3
78.3
Average
70.2
76.6-58.1
539--00.5
63.6-5099
68.2-54.9
70.3
26.0 65.0
Comparioon
178
62.9
68.5 68.5-65.
50.0-32.6
57.3-57,3
65.2-66.1
62.9-62.4 62.4
68.0 68.0
Surt urder
192
72,9 68.2
25.3 76.6
62.0 60.4
74.5 68.8
74.5 73.4
75.0 70.3
79.3 74.5
SQL
NonDB- number
131
77.9-58.8
77.1-7188
66.4 62.6
74.8-73.3
81.7 82.4
77.1-733.
83.2 77.1
DH8-text
911
57.5-51.2
65.1-43..1
38.1-38.1
48.0
53.0-52.1
59.5 58.3
64.7
D8--mmbee
410
72.9-74.4
68.0-654
76.8-76.3
80.0-79.3
83.0-83.7
86.3 85.1
Average
68.8
62.1
74.7
72.2
56.9 51.8
65.5 64.7
70.9 68.5
71,7 69.6
76.3 74.0
AlI
09.9-51.2
75.2
20.8
54.3 40.6
64.2 52.4
69.2-57.1
71,3-39.9
76.6 65.9
Table 3: Data statistics and the execution (EX) accuracy of SOTA text-to-SQU models on the original Spider development set (Spider-dev) and Dr.Spider. The column contains the size of Spider-dev and post-perturbation data in each perturbation test set in Dr.Spider. represents the model we trained with official codes as their models are not public. x-y represents the pre-pertun pre-perturbation accuracy and post- pestturrubbtoon accuracy (i.e. absolute robustness accuracy). We report macro- average scores over multiple perturbations. Bold number is the highest accuracy in each category and the underscore stands for the second best result.
them. This phenomenon is common in reality when the actual users are not aware of DB schemas. Prediction samples of PICARD can be found in Appendix B
Although all NLQ perturbations are based on sentence-level paraphrasing, models are more vulner- able to perturbations that contain task-specific phenomena than others, which demonstrates the value of our perturbation categories. PICARD is more vulnerable to replacing keyword indicators with synonyms than question carriers. For column-related perturbations, PICARD is vulnera- ble to the perturbation that implies columns with question carriers or values than replacing column names with synonyms. The most challenging perturbation among NLQ is the value- synonym where PICARD has 26.9% performance drop. Since most SOTA models rely on string matching to align the DB value with its indicator in NLQ, this perturbation illustrates the limits of such methods when values are represented in different formats.
Surprisingly, although most SOTA models are relatively robust to local semantic changes (SQL perturbations), except for RATSQL and T5-BASE, PICARD still has a 7.3% performance drop in NonDB-number. Figure [ contains an example that PICARD understands "3 youngest winners"" but confuses "8 youngest winners" with "age is 8", A robust model should recognize that the number in the question corresponds to a logical meaning LIMIT BY and be robust to the changes in the number value.
5.2 DIAGNOSTIC INSIGHT FOR MODEL DESIGNS
This section presents experiments to answer some questions about the robustness in terms of model designs and provide insights for developing more robust models.
Are larger models more robust? Figure 2 shows the results of T5-FAMLLY models. Not surpris- ingly, the larger models have better pre-perturbation accuracy. In addition, we find that the larger
8

---

Published as a conference paper at ICLR 2023
80
TS-base
Exact set Match Accuracy
Pre
Execution Acturacy
70
TS-large
b
GraPPa
60
. T5-3B
65
SmBoP
60
50
Post
40
55
GraPPa
50
SmBoP
30
20
10
35
30
Pre
Post
Relative
DB
NLQ
SQL
Figure 2: Pre-perturbation, post-perturbation, and relative robustness accuracy of T5 T5-BASE, T5-LARGE, and T5-3B in terms of EX.
EM accuracy of GRAPPA and SMBoP on pre-perturbation and post- perturbation data of DB, NLQ, and SQL.
Figure
models have both a higher post- perturrbttrooon accuracy and relative robustness accuracy, which indicates that using large pre-trained models is one solution for improving the robustness of models.
What are the advantages of top-down and bottom-up decoders? Figure shows the robustness of GRAPPA and SMBoP which have the same encoder but with a top-down decoder and bottom- up decoder. We find that the bottom-up decoder model SMBOP is more robust to DB perturbations, and the top-down decoder model GR APPA is more robust to NLQ perturbations. We believe is because the DB perturbation modifies local inputs of the database schema and the bottom-up decoder decodes the leaf nodes first, which learns better alignment between NLQ and databases. However, NLQ perturbations are via sentence-level paraphrasing, where the top-down decoder decodes the structure of SQL first, which is more robust than sentence-level paraphrases. This implies that a model combining the top-down and bottom-up decoders could be solution to improve models' robustness on text-to-SQI. tasks.
Does question and DB content linking benefit model robustness? The SOTA models leverage an entity linking feature between question tokens and DB content based string matching to im- prove value prediction performance, which concatenates column names with mentioned DB contents in input (Lin et al.) 2020) Schollak et al. 202T). This feature provides two benefits: (1) indicating which column should be predicted in SQL by its mentioned DB content and (2) providing the for- mat of a mentioned value in its DB. We compare T5-3B and T5-3B LK against NLQ perturbations
and report their relative robustness accuracy. Using the entity linking significantly improves model robustness against col umn-valus (68.7 78.1) and value-a synonym (35.8 46.1) pertur- bations in terms of EX accuracy. The entity linking benefits the SQL value prediction by giving how the value presented in the DB especially when the value appears in a different format in NLQ (ralue-ssnonym)) However, this feature slightly hurts the SQL prediction in terms of EM accuracy (ignoring values) on value model overfits the string-matchin maalue-synonyn synonym (66.2 vs 67.2). We believe that is because the feature to indicate the mentioned column and does not generalize well when the matching fails due to perturbation. Therefore, it is beneficial to provide the formats of mentioned values in the DB via the linking. And better linking mechanism should be adapted for developing robust text-to-SQI. models. A detailed analysis can be found in Appendix
6 CONCLUSION AND FUTURE WORK
We curate a diagnostic text-to-SQL evaluation benchmark Dr.Spider, which contains various pertur- bations on DB, NLQ, and SQL to simulate diverse task-specific robustness challenges. We diagnose the robustness of SOTA text-to-SQL. models regarding their fine-gained components with our bench- mark. Our experiments reveal existing models are vulnerable to perturbations, while some model architecture designs are more robust against certain perturbations. We envision our findings could benefit future model design for robust text-to-SQL models. In the future, we want to explore differ- ent strategies of prompt examples selection besides handpicking for paraphrasing NLQ. We also plan to improve the robustness of text-to-SQU tSL models with the insight from this work, such as combining top-down and bottom-up decoder and using our perturbation methods for data augmentation.

---

Published as a conference paper at ICLR 2023
7 ETHICS STATEMENT
We acknowledge the importance of the ICLR Code of Ethics and agree with it. An unrobust text-to- SQL system may cause harm by misinforming its users. Our work aims addressing by providing a diagnostic robustness evaluation platform and insights to improve the robustness of models. We construct our benchmark Dr.Spider based on Spider, an public and free text-to-SQL benchmark. We recruited annotators from Amazon Mechanical Turk (MTurk) for the crowdsourcing paraphrase study as well as the clarity and fluency annotation for our data quality control. We took great care to pay fair wages. Crowdsourcing annotators are required to paraphrase question or label the clarity and fluency of a question. No sensitive information about annotators was asked and the only personal information we collected was the worker IDs on Mturk, which will not be released. We leverage large pre-trained language models (PLM) to paraphrase questions as our NLQ perturbations. We acknowledge that PLM may generate toxic language (Sheng et al. 2019 Gehman et al., 2020). Three expert annotators carefully review all generated paraphrases to ensure no offensive language in our dataset.
ACKNOWLEDGEMENT
We would like to thank Eric Fosler-Lussier, Michael White, Micha Elsner, Srinivasan Parthasarathy, Huan Sun, Yu Su, Amad Hussain, Pengfei Liu, and others the OSU SLaTe lab and the Clippers seminar, well as the anonymous reviewers for their valuable feedback on this work.
REFERENCES
Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer, Ilya Shnayderman, Noam Slonim, and Liat Fin-Dor. Quality controlled paraphrase generation. arXiv preprint arXiv:2203. 10940, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Xiang Deng, Ahmed Hassan, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure- Structure-rroundd pretraining for text-to-sql. In Proceedings of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language T'cchnologees pp. 1337-1350, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward, Jinxia Xie, and Pengsheng Huang. Towards robustness of text-to-sql models against synonym substitution. arXiv preprint ar 1r1.:22006.0165,, 2021a.
Yujian Gan, Xinyun Chen, and Matthew Purver. Exploring underexplored limitations of cross- domain text-to-sql generalization. arXiv preprint arXiv:2109.05157, 2021b.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. arXiv preprint arXiv:2004.02709 2020.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. In Findings of the Associa- fion for Computational Linguistics: EMNLP 2020, pp. 3356-3369, 2020.
10

---

Published as a conference paper at ICLR 2023
Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. Robustness gym: Unifying the nlp evaluation landscape. arXiv preprint 0rXf:2201100840, 2021.
Shuo Huang, Zhuang Li, Lizhen Qu, and Lei Pan. On robustness of neural semantic parsers. arXiv preprint wrXiv2202..05663 2021.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1875-1885, 2018. 2018.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.
2017.
Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes data. arXiv preprint arXiv: 194444999...44,..... 2019.
difference with counterfactually-augnne
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05833 05858, 2019.
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. Kaggledbga: Realistic evaluation of text-to-sql parsers. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp.
2021a.
Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural data augmentation via example extrapolation. arXiv preprint 212.22100,,1355,, 2021b.
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
Wenqiang Lei, Weixin Wang, Zhixin Ma. Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking in text-to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6943-6954, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequunee pre- training for natural language generation, translation, and comprehension. arXiv preprint 11X1v:1910.13461, 2019.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. In Findings of the Association for Computational Linguis- tics: EMNLP 2020, pp. 4870-4888, 2020.
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. Wanli: Worker and ai collabora- tion for natural language inference dataset creation. arXiv preprint arXiv:2201.05955 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Pingchuan Ma and Shuai Wang. Mt-teql: evaluating and augmenting neural nlidb on real-world linguistic and schema variations. Proceedings of the VLDB Endowment, 15(3):569-582 2021.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial nli: A new benchmark for natural language understanding. arXiv preprint arXiv: 1910.1 1910.4999, 2019.
11

---

Published as a conference paper at ICLR 2023
Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, and Luke Zettle- moyer. Prompting contrastive explanations for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4179-448 47992 2021.
Xinyu Pi, Bing Wang. Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. Towards robustness of text-to-sql models against natural and realistic adversarial table perturbation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2007-2022, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 21(140):1-67, 2020.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. arXiv preprint orXiv:2204.00435 2022.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Be- havioral testing of nip models with checklist. arXiv preprim arXiv:2005.00 041 18, 2020.
Ohad Rubin and Jonathan Berant. Smbop: Semi-auttreerrssivee bottom-up semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Techmologies, pp. 311-324, 2021.
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. arXiv preprint urXiv:2104.07540, 2021.
Torsten Scholak, Raymond Li, Dzmitry Bahdanau, Harm de Vries, and Chris Pal. Duorat: Towards simpler text-to-sql models. arXiv preprint arXiv:2010.11119 2020.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. Picard: Parsing incrementally for con- strained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9895-9901 2021.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp- 464 468, 2018
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional gen- eralization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pp. 922-938, 2021.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. The woman worked as babysit- ter: On biases in language generation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3407-3412, 2019.
Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lexico-logical alignments for semantic parsing to sql queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1849-1864, 2020.
Alane Suhr, Ming-Was Wei Chang. Peter Shaw, and Kenton Lec. Exploring unexplored generalization challenges for cross-database semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2372-83883 2020.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat- sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint 1eXiv:191.00444 2019.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:211 1.02840, 2021.
12

---

Published as a conference paper at ICLR 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL http: //cl web. org/ antho 1 ogy/N1 8 11101]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's 's transformers: State-of-the th-tt---ttt--tttt natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, 2018.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir R Radev, Richard Socher, and Caiming Xiong. Grappa: Grammar-augmented pre-training for table semantic parsing. In ICLR, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint 1rXiv:1709.00103, 2017.
13

---

Published as a conference paper at ICLR 2023
A
DATA DETAILS
A.1 CROWDSOURCING PARAPHRASE STUDY INTERFACE
To understand the linguistic phenomena that NLQS could have in realistic scenario, we recruit annotators on Amazon Mechanical Turk to paraphrase 100 sampled questions in Spider and collect five paraphrases per question. Figure 4 contain the instruction and interface on Amazon Mechan- ical Turk for crowdsourcing paraphrase collection. Annotators are required to rephrase a question in their own words which do change the meaning of the question given the context. We provide a short description of the corresponding database as the context of a question.
Paraphrase Question
In this task, will given English question and its context. Please paraphrase the question words. The to parsptrased the original queatlon).
question
the givan question omit
Below are good persphrase exemples:
Context: There is database containing airines, flihht and airports.
Question: What the code of airport that has the highest number of flights?
airport the flights? List Good Paraphrase What the sirport code the largeol amount flights??
Paraphrase
Question: What the abbreviation of the ainino has the fewest flights and what country it in?
Paraphrase What
of that country of airlino has least flighss.
number flights
dois airilna oparate?
Paraphrase Show abbreviation
Balow are unacoopted example:
Question: What the abbreviation of the airilne has the fewest flights and what country in?
Paraphrase @tt is good peraphrese beceuse the leria feweet flights the original question ornitted)
abbreviation
country
Bad Paraphrase 2: What is the detais of the airinne with the fewest amount of fights?
(It is not good paraphrase because detailss does not mean the same as abbreviation of the airline and country the original question omitted)
Figure 4: The instructions for crowdsourcing paraphrase collection on Amazon Mechanical Turk.
Paraphrasa without changing meaning givan context: given question)
paraphrased question
containing players information, ther ranking winner played greatest
Paraphrasa:
Type your paraphrase here. Thank you
The interface for crowdsourcing paraphrase collection Amazon Mechanical Turk. An- notators are given a free text box input their paraphrased question.
Figure 5:
A.2 PARAPHRASE GENERATION DETAILS
Each NLQ paraphrase category contains an individual prompt prefix which comprises 5 paraphrase examples and question be paraphrased, except for Multitype which includes examples that involve multiple phenomena selected from other categories. Each paraphrase example contains parts: (1) an original sentence, (2) the mentioned SQL tokens that need to be rephrased implied, (3) paraphrased question, and (4) an explanation of how the question is paraphrased. We use OPT- 66B to generate paraphrases given an original question. Inspired by Wei et al. (2022); Paranjapel al. (2021), employ the OPT model to generate explanation along with the paraphrase given an original question and the mentioned SQL tokens related to the category. Table 4 is the prompt prefix of the perturbation col wwwnnnalllll ue. The prompt prefixes for other categories are available at pattps://gitaee
com aws wwwslabs. www.aaaasaaaggssiiiiiceo wwtp///wwaa///.ianoooooooooooooe
As Bandel et al. (2022) discovered, there is trade-off between paraphrase diversity and factuality. We use multiple combinations of hyper-paramett in paraphrase generation to take both into ac- count. For each question, the OPT model times with the hyperparameters top p in (..9.1.0)
14

---

Published as a conference paper at ICLR 2023
and temperature in (0.7,1.0), and paraphrases returned each time (Keskar et al. 2019). We implement the paraphrase generation with Huggingface Wolf et al. (2019). The experiments were done on 8 Nvidia Tesla V100 with 32G memory for about 5 days.
Paraphrase the sentence by implicitly mentioning the type with its entity/number:
(0) Sentence: Show the stadium name and capacity with most number of concerts in year 2014 or after. Mentioned type and entity: 2014 year. Paraphrase: What is the name and capacity of the stadium which held the most concerts in 2014 or later? Explanation: remove year in paraphrase.
(I) Sentence: What are the ids of the students who do not own cats pets? Mentioned type and entity: cat is a pet type. Paraphrase: Find the IDs of students who don't own cats.
Explanation: remove as pets in paraphrase.
(2) Sentence: How many 'United Airlines' flights depart from Airport 'AHD'? Mentioned type and entity: United Airlines airline name; AHD is source airport. Paraphrase: Of the flights out of AHD", how many were "United Airlines'? Explanation: remove Airport in paraphrase.
What are the paragraph texts for the document with the name 'Customer re- views'? Mentioned type and entity: Customer reviews document name. Paraphrase: What is the paragraph text in the document 'Customer reviews"? Explanation: remove name in paraphrase.
(3) Sentence:
(4) Sentence: What is the average expected life expectancy for countries in the region of Cen-
tral Africa? Mentioned type and entity: Central Africa a region. Paraphrase: For coun- tries in central Africa what is the average life expectancy? Explanation: remove region in paraphrase.
(5) Sentence: [new sentence] Mentioned type and entity: [mentioned DB content in SQL] is a [corresponding column]. Paraphrase:
Table 4: Prompt prefix for
A.3 PARAPHRASE FILTERING AND QUALITY CONTROL DETAILS
We use simple and high-recall rule-based classifier to remove out-of-category paraphrases based on the alignment between SQL tokens and their indicators in NLQ. In particular, we use the anno- tations from Lei et al. (2020) build the column and value indicators. And obtain SQL keyword indicators, we collect the common indicators (covering over 80% examples) of each SQL keyword in the NLQs and programmatically label NLQs with those indicators. (5ble(5]containn) the common indicators of SQL keywords in Spider. We filter out the paraphrases where all the indicators for SQL keywords, columns and values remain the same as the original NLQ except category others.
To ensure the data quality in Dr.Spider, we request three annotators with SQL expertise to carefully review paraphrases with gold SQL labels and select the final data. Table contains the randomly sampled NLQ paraphrases in each category and the annotator's decision.
To measure the naturalness of our generated NLQ perturbations, we sample 500 questions from each of the original Spider development set, human paraphrases (we use all 270 factual paraphrases), and our perturbed questions. We hire crowdsourcing annotators on Amazon Mechanical Turk to score the clarity and fluency of sampled questions from 1 to 3. Clarity=1 represents that the annotator couldn't 't understand the question, clarity=2 indicates that the question is ambiguous to the annotator, and clarity=3 means that the question is clear. Fluency=1 represents that the question is full of grammar eтгors or typos, fluency=2 indicates the question has minor errors, and fluency=3 means the question fluent without grammar errors or typos. Figure the interface for labeling the clarity and fluency of questions in the original Spider dataset, collected human paraphrases, and the questions in Dr. Spider.
15

---

Published as a conference paper at ICLR 2023
SQL Keyword
Indicator
more than, larger than, bigger than, higher than, above, after, older than, heavier than
less than, smaller than, lower than, below, before, younger than, lighter than at most, or more at least, or less between
>=
<=
between and count ()
number, amount, count total, amount
sum
avg ()
average, mean
max ()
maximum, most, highest
min( ()
minimum, least, lowest
ASC
ascending, in alphabetical order, in lexicographical order, from youngest to oldest, from young to old, from low to high
ASC LIMIT
least, lowest, smallest, youngest, earliest, shortest, minimum, fewest number, fewest amount
DESC
descending, reverse alphabetical order, in reversed lexicographical order, from the oldest to the youngest, from old to young, from high to low
DESC LIMIT
most, highest, largest, oldest, latest, longest, maximum, greatest number, greatest amount
Table 5: The common NLQ indicator tokens for SQL keywords.
16

---

Published as a conference paper at ICLR 2023
Determine the clarity and fluency of question under context
Gudde: conket exctoetent cumtore belb rebite Read custton casttuly and ly and Fumnyy quetiis.
Context:
Tablo city
dearnyycad
dBuiier
populbsion
lest
1068C0
Table country
sade
name
condieen
regken
surloonamaa
Indeyyarr population
Phorgemaany gre
grpaid
localsamo
headertsare
caphal codo2
ABI
Ando
North
Our booe
135.0
109000
22.4
5050
7050
Arobe
Nonrii-pocion Notherl nds
Bestri
129
AW
Southen
0320000
32720000
391D None
Contral
12460000
FABRO
cond
Abfic
Dembos
Table countrylanguage
eountrycede
Mnngage
percentage
averto
70L7
Question:
*ini mudder sitim. with whaue pepilat an aie live the avnueg pesdlt vilind
Determine the clarity of the above question
what estin ooding about
q masty dies theme
come ambiguet as (Thers ore way arover condion)
The quolice dea and NOT anbiguous
Detarmine the fluency of the above question
g gove inpossii
incomplee
flecgaolicn ownall
platene type shoud sodiing taraaat.
uneticn
from grommor enomonr types
Figure 6: Annotation interface for labeling the clarity and fluency for a given question.
17

---

Published as a conference paper at ICLR 2023
Paraphrase Example
Keywerd-synen synonym
Question: find the code of the country where has the greatest number of players. Paraphrase: Display the code of the country with largest player population. Selected in Dr.Spderr
Keyword-carrier
Question: Count the number dogs that went through a treatment.
Paraphrase: How many dogs have been treated? Selected in Dr.Spider
Column Sain--yynee synonym
Question: What the average weight for each type of pet?
Paraphrase: For each category of pets, what the average weight?
Selected in Dr.pider
Column umn--m--i Column-synonh synonym
Question: What the money rank of the poker player with the highest carnings?
Paraphrase: The poker player at rank made the most income during his or her career.
Not Selected in Dr.Spr
Col colun-ccrrier
Question: Find the name of tourney that has more than 10 matches.
Paraphrase: Which tournament has more than games?
Selected in Dr.Spider
Column Saluunccariicn
Question: What are the song titles and singer names??
Paraphrase: Which songs have titles and singers?
Not selected in Dr.Spieer
Col uenuumn-ttttn ribute
Question: What is the car model with the highest mpg
Paraphrase: what is the car model that is most gas efficient?
Selected in Dr.Spider
Col volume-value
Question: What the phone number of the man with the first name Timmothy and the last name Ward?
Paraphrase: Find the phone number for Timmothy Ward?
Selected in Dr.Spider
Value-synonya
Question:Wher What are the names of conductors whose nationalities not "USA"?
Paraphrase: Names of conductors whose nationalities not US.
Selected in Dr.Spider
Mult type
Question: What are names of countries with the top largest population?
Paraphrase: Which three countries have the highest population?
Selected in Dr.Spider
Mult itype
Question: What are the airline names and abbreviations for airlines in the USA?
Paraphrase: list all the airlines and identify their respective codes in the United States.
selected Dr.Spider
Others
Question: What the average, minimum, and maximum age of all singers from France?
Paraphrase: Show me your data on the average, minimum, and maximum ages of all singers from France. Selected in Dr.Spider
Table 6: Randomly sampled paraphrases and human annotator selection decision.
18

---

Published as a conference paper at ICLR 2023
A.4 SQL PERTURBATION
SQL Perturba-
Pre-perturbation
Post-perturbation
tion
Comparison
SQL: HAVING count (x)
SQL: HAVING count (*)
NLQ: more than one car maker?
NLQ: least one car maker?
Sort-order
SQL: ORDER BY age DESC
SQL: ORDER BY age ASC
NLQ: from the oldest the youngest.
NLQ: from the youngest to the oldest.
NonD3-nunber
SQL: LIMIT 3
ORDER BY winner.age
SQL:
ORDER BY winner.age
LIMIT
NLQ: 3 youngest winners
NLQ: 5 youngest winners
DB-text
SQL: country 'France'
SQL: country 'Netherlands"
NLQ: singers from France
NLQ: singers from Netherlands
DB-number
SQL: horsepower 150
SQL: horsepower > 145
NLQ: horsepower more than 150
NLQ: horsepower more than 145
Table 7: SQL perturbations and examples.
Table example of each SQL perturbation. Since SQL perturbations involve modifications on both SQL. tokens and their indicators in NLQs, we want to minimize the surface-level changes introduced NLQs disentangle SQL perturbations from NLQ perturbations. For compart son and sort- intt-ore order perturbations, we programmatically modify the keywords in SQL and their in- dicators in NLQ following the rules in Tables X - We replace only one indicator with another that contains the same meaning in the context. For example, when replacing > with <, "older than" replaced by "younger than" instead of "lighter than".
above
below
after
beforc
older than younger than
heavier than
lighter than
more than
less than
at most at least
larger/bigger than
smaller than
higher than lower than
or more
or less
Table 8: The common NLQ indicator tokens for comparison operations in Spider.
ORDER BY
ASC
ORDER BY
DESC
ascending
descending
alphahetical order
in reverse alphabetical order reversed lexicographical order from the oldest the youngest from old to young
lexicographical order
from the youngest the oldest from young old
from low to high
from high to low
Table 9: The common NLQ indicator tokens for ascending sort and desending in Spider.
19

---

Published as a conference paper at ICLR 2023
ORDER BY
ASC LIMIT
ORDER BY
DESC LIMIT
least
most
lowest
highest
smallest
largest
youngest
oldest
earliest
latest
shortest
longest
minimum
maximum
fewest number
greatest number
fewest amount
greatest amount
Table 10: The common NLQ indicator tokens for ascending sort and descending sort followed by LIMIT clause in Spider.
20

---

Published as a conference paper at ICLR 2023
B MORE EVALUATION RESULTS
Table 13 shows the EM accuracy of SOTA models on pre-perturbation and post-perturbation data in Dr.Spider. EM accuracy does not evaluate value predictions in SQLs. Therefore, EM is not effective to measure model robustness against NonDB-numbel DB-text, and DB-text. Table 14 [5] contains the relative robustness accuracy EX and EM of SOTA text-to-SQL models. Relative robustness accuracy ignores the examples that the model predicts wrong before perturbations for each model.
Table 11 contains the subcategories and examples in DBcontent -eeettteeeeeuuieeeeeeeee simulate a scenario where a column content represented in different formats by replacing a column with one or multiple semantic-equivalen columns. Table perturbation data. PICARD predicts SQLs correctly for all four examples before perturbations. The first two examples both replace column age with birthyear, however, PICARD defenses against one perturbation example but fails the other. The third example replaces a fix-value text column pettype with two boolean columns is_dog and is.cat. PICARD fails to predict the right value for the boolean columns is dog. The last example replaces compound column name with two columns f name and lastname. PICARD refer both fi firstname and lastname when the NLQ asks for "names". We believe that DBcontent have additional common-sense reasoning to understand the relation between a database column and how it represented in NLQ. This phenomenon common in reality when the actual users not aware of DB schemas.
irrontent-equivalle
12 contains prediction examples of PICARD on
valence requires models to
Category
Column Example
Single text to multiple texts
name => firstname, lasthare
Boolean to text
is.male >> gender
Text to boolean
hand is.right.hadd
Single text to multiple booleans
dogrec.sunmary isbachelor, ismaaseer, 1s_PhD
Number to number
age => birthyear
Table 11: Categories and examples in the DEcontent-eequvvlleeee partiaauaannneeetruurrttttto perturbation.
The effectiveness of entity linking between NLQ and DB content Figure 7 shows the relative robustness accuracy on EX and EM of T5-3B models with and without the en- tity linking feature. Using the entity linking significantly improves model robustness against column-valle value and value-synonym synonym perturbations terms of EX while it gives less robust- ness col umn amn-attribute and
perturbations (both EX and EM).
The EX and EM accuracy on the value value-synonym synonym perturbation are different. The entity linking benefits the SQL value prediction by giving how the value is presented in the DB especially when the value appears a different format in NLQ (value-synonym). However, this feature slightly hurts the SQL prediction in terms of EM accuracy (ignoring values) on value-synonym (66.2 67.2). We believe that because the model overfits the string-matching feature to indicate the mentioned column and does not generalize well when the matching fails due perturbation. Therefore, is beneficial to provide the formats of mentioned values in the DB via the linking. And a better linking mechanism should be adapted for developing robust text-to-SQL models
Robustness Evaluation on in-context learning models Besides the models that are trained on the Spider dataset, we also evaluate the robustness large language model, CODEX Chen et al. (202T), for in-context learning. We use CODEX DAVINCI which finetunes (roww et al. a (2020) on GitHub codes. We choose the Create Table + Select prompt from (Rajkumar et al. 2022), as it achieves the best performance with in-context learning (74.0 execution accuracy on the original Spider development set based on our implementation), which is even comparable with the SOTA finetuned model, PICARD (79.3 execution accuracy). Table 16 shows the comparison between PICARD and CODEX on their pre-perturbation, post -estuerturbation accuracy, and relative robustness accuracy. We have few findings: (1) In-context CODEX is more robust than finetuned PICARD
21

---

Published as a conference paper at ICLR 2023
Example
NLQ: What are the and release years for all the songs of the youngest singer?
Gold SQL: ... ORDER age LIMIT
Column Substitution: age -> birthyear
Gold SQL after perturbation: ORDER BY birthycar DESC LIMIT
Predicted SQL after perturbation: ORDER BY LIMIT 1 Incorrect
NLQ: Find the type and weight of the youngest pet.
Gold SQL: ... ORDER RY pet_age L.MIT
Column Substitution: age -> birthyear
Gold SQL after perturbation: ORRER BY pet bi birthyear D8SC LIMIT 1 Predicted SQL after perturbation: ORDER BY pet birthyear DESC LIMIT Correct
NLQ: How many dog pets are raised by female students?
Gold SQL: ... WHERE Pets. rets.pets pettype 'dog'
Column Substitution: pettype -> is.deg
Gold SQL after perturbation: WHERE Pets. rets.is.dog = True... Predicted SQL after perturbation: WHERE Fets. = 'D'... Incorrect
NLQ: List all singer names concerts in 2014.
Gold SQL: SELECT
FROM
Column Substitution: name first.name, last_name
Gold SQL after perturbation: SELECT first_name, last_name FROM Predicted SQL after perturbation: SELECT first_name, last_name FROM...
Correct
Table 12: Prediction examples of PICARD on col column-equivvlence perturbation data.
to database perturbations though it still suffers performance drop from 72.6 60.7, and (2) in-context CODEX does not show better robustness against NLQ perturbations than finetuned PICARD.
We hypothesize that the robustness of CODEX largely related to its training data. As the training data are obtained from GitHub codes, CODEX has seen a diverse distribution of databases. Specif- ically, we find that CODEX has a limited performance drop to Schema awttttabbbbbvaiiiie bation (from 70.2 to 68.6), compared PICARD (from 74.9 to 64.7). We believe that CODEX is robust to the Schema-abbrewe innniiiiinnnn perturbation, as it has seen a large number of abbreviation schemas its training data.
pertur-
In NLQ perturbations, we find two categories that are particularly challenging to CODEX: Co Column-carries (from 80.8 51.1), and Column-attribut perturbations implicitly mention a column with the question carrier or the attribute of the column. For example, Table the column "name" "Show the name teacher" implied by "Which teachers"" and the column "year" in "worked the greatest number of year" is implied by "worked the longest". CODEX was trained with the goal of generating code from docstrings. The language style of common docstrings is slightly different from the style of spoken language therefore CodeX may not be familiar with such implicit mentions. As the CodeX details are not fully published, the analysis is subjective.
(from 68.9 to 46.2). Those
22

---

Published as a conference paper at ICLR 2023
RATSQL
T5-DASE:
T5-LARGE
Spidkr-dev
7344
$6.0*
65.3
68.8*
75.5
Schema synanym
63.2 41.1
19.4
67.5
48.5 24.3
51.7 37.7-36.0
61.9
63.7 45.2
68.7 54.1
Schema--tovrvviaiion
65.2-400 10.0
60.7 53.3
69.3
49.3-23.1
62.0
64.3 64.3-499 49.9
67.6-50.7
72.7 60.4
DBontteni DBoonten--giivvkeee
81.2-5.8 5.8
85.6 5.2
81.2 34.0
58.4-15.2
71.7-300 30.1
77,0-34.8
82.5 36.4
85.3 39.8
Avergae
69.9-29.0 29.0
74.1-36.0
72.7
52.0-200 20.9
34.8
67.7-11.8
71.3 44.1
75.6 51.4
Xeyword-ynonge
63.4 49.3
70.4
730--58.1
51
63.6-52.5
64.2-52.6
67.9-55.6
70.2-58. 58.7
Keyuord-earier
79.2 76.2
78.2-74.2
7499-72.7
62.0 59.4
72.9-66.7
76-9-71.9
77.0-7559
Column-gyooym
67.9 47.8
71.2 54.4
67.7- 80.1
48.7-323 32.7
55. 44.4
66.6 66.6-50.3
67.9 67.9-54.5
54.5
74.6 59.7
NLQ
Column carrier
14.8 55.3
$1.0 61.5
78.6 78.6-572
57.0 39.4
71.2 56.1
23.6 23.6-55.3
79.6 79.6-54.4
83.6-57.0
Column attribute
51.3 34.5
58.0 48.7
72.3 53.8
42.9 37.8
89.6 18.7
54.6 46.2
55.5 45.4
64.7 52.9
Column-vauue
78.0 55.3
52.6 51.6
52.3-28.3
68.8-43.1
65.8-52.6
78. 62.1
value-ynonym
60.3 38.1
67.4
43.7-33..6
62.1
58.3
Multitpee
67.1 42.0
74.2
30.1-33.5
61.8 44.9
66.1
69.9 48.9
55.4
Others
71.5 63.5
73.1 63.1
74.2 67.6
$6.9 09.7
65.4 60.0
68.4 62.2
72.1 66.1
75.5 70.5
Average
51.3
72.8 728--773
74.1 56.7
51.8-39.5
62.5-50.4
66.4-51.7
69.3
74.3 61.2
Compurison
56.2-33 53.9
64.0
63.5- 57.9
46 18.7
54.5-55.1
58.4-31.77
59.0-55.6
61.2-58.4
23.4 73.4
72.4
SQI
MonDB8-mabber
75.6-72.5
62.6-65.6
77.9-77.99
29.4
81.7-748
DB-aext
63.6 63.6-65.0
67.0-64.2
66.8 62.2
43.5 40.5
54.2-
58.3-54.1
59.3-55.4
66.3 66.5
DR-rumbe
72.9-72.99
$6.1 15.9
83.7 82.2
66 66.8-64.4
754--749
78.8-
83.0-83.4
83.9-81.4
Average
64.7 61.9
74.4 72.5
72.1 68.9
51.7 09.8
63.1 63.0
68.8 66.6
70.7 67.2
23.7 70.9
67.5-1144
73.5
73.3-583
52.7-39.3
62.9-1155
67.3-54.3
70:0-56.8
Table 13: The exact set match (EM) accuracy of SOTA text-to-SQU models on the original Spider development set (Spider-dev) and Dr.Spider. represents the model we trained with official codes as their models are not public. x-y represents the pre-perturbation accuracy and post-perturbation accuracy (i.e. absolute robustness accuracy ). Bold number is most highest absolute robustness accuracy in each category and underscores stand for the second best results. Note that the EM accu- racy does not evaluate value predictions. Therefore, it is not effective to measure model robustness against NonDB-numte -nuubbe,, DB-text, and DB-text.
23

---

Published as a conference paper at ICLR 2023
Perturbation
RATSQL
T5-3B
T5.38
Schema--ynonsm
64.6
73.5
58.5
62.8
67.2
74.6
Schema abbreviation
62.2
79.3
41.4
59.4
72.0
72.7
83.0
Delconeen--qquvvaleee
11.8
42.6
27.1
44.0
44.1
46.1
49.3
Avergae
46.2
65.1
38.2
54.0
59.6
62.0
69.0
syyoonn
73,9
81.5
78.8
81.9
84.8
85.7
88.0
Keyword-canierr
93.0
92.4
Column synonym
NLQ
Column-carrier
67.7
73.2
67.2
74.2
74.5
71.9
73.0
Column altribule
65.6
70.0
67.7
80.7
89.1
83.1
82.9
Column-value
70.1
72.3
50.6
70.8
68.7
78.1
81.7
value symonym
25.4
37.5
49.1
55.2
52.5
63.7
69.2
Multitype
52.5
58.6
59.4
64.3
65.5
69.1
72.8
Otbers
84.6
89.3
82.9
87.5
88.7
94.0
96.1
Average
Comparison
82.1
91.0
60.7
91.2
92.9
Sort-order
88.6
93.1
91.6
89.5
95.8
91.7
92.1
SQL
NonDB-number
73.5
91.1
89.7
93.9
100.0
94.1
92.7
DB-text
79.2
91.1
83.3
90.0
88.6
90.8
94.1
DB-number
98.7
95.5
93.9
98.7
97.6
99.7
98.3
Average
84.4
92.4
83.8
92.7
94.5
93.8
93.3
All
68.1
76.5
76.4
78.5
Table 14: The relative robustness accuracy on EX of SOTA text-to-SQL SQLL models on Dr.Spider. Bold number is most highest absolute robustness accuracy in each category and underscores stand for the second best results.
Robustness Accuracy on
Robustness Accuracy on
Column-synonym
Colume-synorym
carrier
Keyword-corrie
Keyword-camer
60 Keyword-synonym
Keyword- w-wyooyyy
8
Column-value
Colurn-valoe
Others
Others
Value-synorym
allueyyyoormm
Nulitypee
type
(a) Relative Robustness Accuracy on EX
(b) Relative Robustness Accuracy on EM
Figure 7: Relative robustness accuracy on EX and EM of T5-3B models with and without question and DB content linking.
24

---

Published as a conference paper at ICLR 2023
Perturbation
RAYSQL
GRAPPA
SMBOP
T5-BASE
T5-LARGE
T5-3B
T5-3B LK
PICARD
Schema-synonym
59.3
75.9
Schemss-bbbreiianee abbreviation
58.9
73.4
76.2
38.6
59.3
74.0
72.2
80.8
DRconten--epiivaless
6.5
5.5
38.1
22.4
38.0
41.5
43.5
46.6
Avergae
42.2
49.7
61.8
35.7
52.2
59.7
61.2
67.8
Keyword-syooyym
72.2
75.3
77.2
73.5
77.6
78.3
78.4
80.0
Keyword
93.0
93.0
94.0
91.2
91.8
90.4
93.2
95.8
Columm-synonym
77.6
NT.Q
Column carrier
66.3
Column-attribute
65.6
79.7
73.3
78.4
67.8
80.0
66.7
67.5
Column value
66.7
60.2
75.8
54.1
63.9
61.7
76.5
78.6
value-synonym
$7.7
79.0
57.8
68.3
73.2
67.2
66.2
79.2
Mulitype
59.4
68.9
60.5
62.8
67.1
65.1
66.1
72.1
Others
83.5
85.2
86.8
81.6
86.3
86.8
88.8
90.9
Average
70.2
76.5
73.8
70.8
74.8
74.7
75.0
78.7
Compurison
91.4
89.9
Sort-order
93.5
97.2
92.9
93.8
93.9
97.1
92.8
93.8
SQL
NonDB number
96.6
96.0
91.7
94.7
100.0
100.0
92.3
91.6
DB-text
94.5
92.3
90.1
86.9
94.1
91.7
90.2
95.5
DB number
99.3
98.6
98.3
95.3
99.4
98.5
99.1
99.1
Average
94.0
94.5
92.1
85.8
96.0
94.6
93.2
94.0
All
72.2
77.1
77.0
77.1
77.9
77.9
81.2
Table 15: The relative robustness accuracy EM of SOTA text-to-SQL. models Dr.Spider. Bold number is most highest absolute robustness accuracy in each category and underscores stand for the second best results. Note that the EM accuracy does not evaluate value predictions. Therefore, is not effective to measure model robustness against NonDB-number. DB-text and DB-text.
25

---

Published as a conference paper at ICLR 2023
Pre-pertubation The-petubsiiio--PPo-rrrnee Post-perturbation
Relative Robustness
Perturbation
PICARD
CODEX
PICARD
CODEX
Spider-dev
79.3
74.0
synonym
Schemo-atereviation
DBcunient equivalence
88.7
43.7
78.8 51.6
49.3
62.5
Average
78.9
55.0
72.6-600 60.7
69.0
80.0
Keyward synonyy
72.6
66.3
63.8 55.5
88.0
81.7
Keyword--crrier
85
82.7
88.5 85.2
96.2
95.5
Column synonym
71.0
57.2
68.4 54.7
77.7
76.4
NLQ
Column-carrier
86.9
64.9
80.8-51.1
73.0
60.9
Column-
Colamn-value
$2.9-694
86.2 -1.4
81.7
79.8
value-synonyn
72.5 53.0
72. 1 59.9
69.2
78.6
Multitype
74.4 87.1
72.6-533
72.8
70.7
Others
79.6 78.3
76.6 69.7
96.1
87.1
Average
76.0 65.0
75.3-600
82.0
77.3
Comparison
68.0 68.0
73.6 66.9
89.3
84.7
SQI.
onder
NonDR-numher
DB lext
64.7
73.3 72.4
94.1
89.7
Dumnem
86.3 85.1
00.5-79.
98.3
96.1
Average
76.3 74.0
74.6
73.1
93.3
91.6
All
76.6 65.9
64.6
83.0
81.9
Table 16: The execution (EX) accuracy of SOTA finetuned and in-context-leamning text-to-SQL. models on the original Spider development set (Spider-dev) and Dr.Spider. The first two result columns are pre-perturbation and pose-perturbation accuracy of PICARD and CODEX, where x x-y represents the pre-perturbation accuracy and post-perturbation accuracy (i.e. absolute robustness accuracy). The last two result columns the relative robustness accuracy of PICARD and CODEX. We report macro-average scores over multiple perturbations. Bold number is the highest accuracy each category.
26


Reviews:
[{"review_id": "eaTnl2QCHi", "reviewer": "Reviewer_WKcN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work contributes a robustness benchmark built on top of the Spider text-to-SQL benchmark dataset. One novel aspect is the use of crowdsourcing in tandem with language models to generate paraphrases of the natural language queries. The paper presents an evaluation of recent models against this new benchmark along with a corresponding error analysis.", "summary_of_the_review": "This seems like a valuable contribution to this problem area that should spur further research and improvements.", "strength_and_weaknesses": "The paper is clearly written, systematically laying out the kind of data variation they are targeting and how they aim to achieve it. The experimental results show significant problems or gaps with existing models against these variations, but the error analysis also points to potential directions for improvement. The paper seems to reflect deep domain knowledge on this problem area and good contextualization with respect to related works.\n\nI found the paraphrase categories (Table 1) very helpful.\n\nThe Appendix is extensive, containing fairly fine-grained details about the implementation and execution of the experiments, as well as even more details on the results themselves. To be honest I think the Prompt Prefix tables (Appendix C) could be put in the code repo instead but it is a minor suggestion.\n\nThe error analysis and diagnostic insights seem very promising for future work. It's interesting that PICARD got confused by \"8 youngest winners\" but I guess queries in datasets probably often ask for Top 3 vs Top 8.\n\nFleiss Kappa of 0.61 is reasonable but not perfect agreement - were there interesting patterns of disagreement among annotators?\n\nThere may be weaknesses that a deeper expert on this area may have better context on, but I saw no red or yellow flags.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "eB6CaOVuMk", "reviewer": "Reviewer_QnJr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a variation of the SPIDER evaluation benchmark called Dr. Spider. Dr. Spider with the dr standing for Diagnostic Robustness, improves the original SPIDER benchmark by introducing augmentations to the Natural Language Quries and the SQL. While the augmentations do not change the benchmark, the challenge systems' understanding of natural language, and SQL. The augmentations aim to reveal whether models are overfitting to SPIDER's perks and to challenge systems to generalize better. The paper's experiments show that all state-of-the-art systems score lower on Dr. Spider, thus proving the need for an augmented SPIDER.", "summary_of_the_review": "This is a well-written paper with no major weaknesses.\n", "strength_and_weaknesses": "## Strengths\n\n+ The paper is well-written and easy to follow.\n+ It is important that SQL benchmarks test for linguistic variations and challenge systems to do more than lexical matching between the NLQs and the SQL queries\n+ The experiments are convincing and thorough.\n+ Figure 1 and Table 1 are great at helping readers follow along with the particularities of Dr. Spider.\n\n## Weaknesses\n\n- Did not see any.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YxyJGV_2q8u", "reviewer": "Reviewer_MJgi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper created a dataset to evaluate text-to-SQL systems on their robustness. The dataset, DR.SPIDER, is an extension of the openly available SPIDER dataset. It added perturbation to the natural language questions, SQL statements and example database (schema and values), so more robust text-to-SQL systems would have less degradation in accuracy on the perturbed samples. The paper categorized the perturbations into 17 different categories, covering different aspects of possible real-world variations.\n\nTo obtain the perturbations, the paper employed multiple methods. It used synonym sets from previous related works, employed in-house domain experts, used online crowd-source workers (Amazon Mechanical Turks), and leveraged PLMs. The process focused more on getting a balanced and correct dataset than the effectiveness of any single method. \n\nThe paper performed extensive evaluations of state-of-the-art models on DR.SPIDER, showed the ability of robustness measurement of the dataset, and also provided insights on the factors that may affect the robustness of existing algorithms. \n\nThe authors promised to release the dataset and evaluation tools. ", "summary_of_the_review": "The paper provides a good extension to the SPIDER dataset to measure the robustness of text-to-SQL models. It addresses a clearly defined and important aspect of evaluation. The construction of the dataset is well thought, and the effectiveness of it has been demonstrated with experiments involves SOTA models. The paper is also clearly written and easy to follow. The resulted dataset could be a useful resource for the community in the future.\n\nThe only concern is the size of the set (a magnitude smaller than SPIDER, if the Reviewer understood correctly). The final size of the set is not clearly mentioned in the abstract, intro or conclusion (only mentioned \"17 perturbations\"). ", "strength_and_weaknesses": "Strength:\n\n1) A clear and useful goal. The paper addresses a specific and useful problem of evaluating the robustness of text-to-SQL systems, where existing datasets either cannot do, or have gaps in coverage. \n2) Detailed and useful evaluation. Most of STOA algorithms are evaluated, with detailed break-downs and also examples to help reader understand.\n3) Clear writing. The paper is well written, The Reviewer had no trouble following the main idea and technical details.\n4) Helpful to the community. The dataset will be released, which would help the improvement of the robustness of related methods.\n\nWeakness:\n\n1) The necessity and complexity of using PLMs are not clear, as 1) the total amount of data generated is small, in the range of hundreds. 2) significant human review is still needed up and down the pipeline.\n2) The size of DR SPIDER (The Reviewer's impression is ~1k) is much smaller than SPIDER (~10K questions).\n\n\nDetailed suggestions:\n\n1) Section 2 paragraph 1, last sentence: it's better to explain why we need semantic-changing perturbations, e.g. to be used as negative examples? or to test the model's ability to distinguish between closely related but different semantics? \n2) Section 3.2 paragraph 2: better explain what is \"naturally occuring  tables and columns\".\n3) Last paragraph in Section 3.2: Why split the filtered data into chunks, while ensuring each chuck have one annotators, why not just annotate all without chuck-splitting?\n4) Section 4. Upper-case of common model names, \"Bert-large\" -> \"BERT-large\" etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QGckIvXZkDm", "reviewer": "Reviewer_gzxj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a benchmark to test the robustness of text-to-SQL models. Specifically, the benchmark is created to test whether such models work well (i) when names of particular columns in a table are replaced with synonymous strings (DB perturbations), (ii) when natural language questions are replaced by its paraphrases (NLQ perturbation) and (iii) when minor changes are in the natural language query and the SQL query (SQL perturbation). The paper extends the Spider dataset (Yu et al 2018) to cover all the perturbations. They propose a total of 17 perturbations belonging to these 3 categories.\n\nDB perturbation: For DB perturbation the paper proposes replacing column names by synonyms and abbreviations. Moreover, they also propose a novel perturbation where they replace a column by one-or-more multiple equivalent semantic columns (e.g. the “name” column can be replaced by two columns “first_name”, “last_name” etc)\n\nNLQ perturbation: For NLQ perturbation, the paper proposes a scalable approach in which first crowd workers propose 5 paraphrases of 100 questions. This is followed by experts filtering and categorizing the paraphrases into 9 categories. Next, to scale the process, an LLM is prompted with paraphrases from each category to generate paraphrases of new questions. This is followed by another automated filtering stage where an NLI model is used followed by the final round of expert filtering.\n\nSQL perturbation: For SQL perturbation, they replace operations like comparisons, sort-order, etc from both the natural language and SQL query.\n\nThe paper tests a variety of text-to-SQL models on this dataset and finds that the most robust models suffer from a 14% drop in performance with around a 50.7% drop on the most challenging perturbation.", "summary_of_the_review": "I enjoyed reading the paper and I believe this will be an interesting and strong benchmark. I am leaning toward accepting the paper.", "strength_and_weaknesses": "**Strengths**\n\n- Developing robust text-to-SQL models is important and this benchmark can serve an important role to test that\n- The paper covers more comprehensive perturbation for all components of text-to-SQL tasks. Moreover the use of LLMs to generate perturbation makes their process scalable without making the task artificial (e.g. same fluency scores of questions)\n- The paper is clearly written and easy to understand\n\n**Weaknesses**\n\n- I would have liked to see the robustness performance of large LMs such as GPT-3. It would be interesting to see how much incontext learning with large LMs can be robust to the proposed perturbations", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%