Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a benchmark to test the robustness of text-to-SQL models. Specifically, the benchmark is created to test whether such models work well (i) when names of particular columns in a table are replaced with synonymous strings (DB perturbations), (ii) when natural language questions are replaced by its paraphrases (NLQ perturbation) and (iii) when minor changes are in the natural language query and the SQL query (SQL perturbation). The paper extends the Spider dataset (Yu et al 2018) to cover all the perturbations. They propose a total of 17 perturbations belonging to these 3 categories.

DB perturbation: For DB perturbation the paper proposes replacing column names by synonyms and abbreviations. Moreover, they also propose a novel perturbation where they replace a column by one-or-more multiple equivalent semantic columns (e.g. the “name” column can be replaced by two columns “first_name”, “last_name” etc)

NLQ perturbation: For NLQ perturbation, the paper proposes a scalable approach in which first crowd workers propose 5 paraphrases of 100 questions. This is followed by experts filtering and categorizing the paraphrases into 9 categories. Next, to scale the process, an LLM is prompted with paraphrases from each category to generate paraphrases of new questions. This is followed by another automated filtering stage where an NLI model is used followed by the final round of expert filtering.

SQL perturbation: For SQL perturbation, they replace operations like comparisons, sort-order, etc from both the natural language and SQL query.

The paper tests a variety of text-to-SQL models on this dataset and finds that the most robust models suffer from a 14% drop in performance with around a 50.7% drop on the most challenging perturbation.
strength_and_weaknesses: **Strengths**

- Developing robust text-to-SQL models is important and this benchmark can serve an important role to test that
- The paper covers more comprehensive perturbation for all components of text-to-SQL tasks. Moreover the use of LLMs to generate perturbation makes their process scalable without making the task artificial (e.g. same fluency scores of questions)
- The paper is clearly written and easy to understand

**Weaknesses**

- I would have liked to see the robustness performance of large LMs such as GPT-3. It would be interesting to see how much incontext learning with large LMs can be robust to the proposed perturbations
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and easy to follow

Quality: I believe the proposed dataset will be of high quality and would serve as an interesting benchmark to test the robustness of text-to-sql models

Novelty: Even though robustness of text-to-sql models were studied before, the paper does a more extensive and thorough job of creating perturbations. Hence, I believe the paper still makes a novel contribution

Reproducibility: The dataset will be released to the public, so yes.
summary_of_the_review: I enjoyed reading the paper and I believe this will be an interesting and strong benchmark. I am leaning toward accepting the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a variation of the SPIDER evaluation benchmark called Dr. Spider. Dr. Spider with the dr standing for Diagnostic Robustness, improves the original SPIDER benchmark by introducing augmentations to the Natural Language Quries and the SQL. While the augmentations do not change the benchmark, the challenge systems' understanding of natural language, and SQL. The augmentations aim to reveal whether models are overfitting to SPIDER's perks and to challenge systems to generalize better. The paper's experiments show that all state-of-the-art systems score lower on Dr. Spider, thus proving the need for an augmented SPIDER.
strength_and_weaknesses: ## Strengths

+ The paper is well-written and easy to follow.
+ It is important that SQL benchmarks test for linguistic variations and challenge systems to do more than lexical matching between the NLQs and the SQL queries
+ The experiments are convincing and thorough.
+ Figure 1 and Table 1 are great at helping readers follow along with the particularities of Dr. Spider.

## Weaknesses

- Did not see any.

clarity,_quality,_novelty_and_reproducibility: * The paper is well-written and easy to follow.
* Figure 1 and Table 1 are great at helping readers follow along with the particularities of Dr. Spider.
* Augmenting datasets and benchmarks is not a new idea, but, as the experiments show, it is necessary. I think that applying a known approach where necessary is a positive.
* The paper is clear on how the data was collected and the appendices add all necessary details to understand the data collection.

summary_of_the_review: This is a well-written paper with no major weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work contributes a robustness benchmark built on top of the Spider text-to-SQL benchmark dataset. One novel aspect is the use of crowdsourcing in tandem with language models to generate paraphrases of the natural language queries. The paper presents an evaluation of recent models against this new benchmark along with a corresponding error analysis.
strength_and_weaknesses: The paper is clearly written, systematically laying out the kind of data variation they are targeting and how they aim to achieve it. The experimental results show significant problems or gaps with existing models against these variations, but the error analysis also points to potential directions for improvement. The paper seems to reflect deep domain knowledge on this problem area and good contextualization with respect to related works.

I found the paraphrase categories (Table 1) very helpful.

The Appendix is extensive, containing fairly fine-grained details about the implementation and execution of the experiments, as well as even more details on the results themselves. To be honest I think the Prompt Prefix tables (Appendix C) could be put in the code repo instead but it is a minor suggestion.

The error analysis and diagnostic insights seem very promising for future work. It's interesting that PICARD got confused by "8 youngest winners" but I guess queries in datasets probably often ask for Top 3 vs Top 8.

Fleiss Kappa of 0.61 is reasonable but not perfect agreement - were there interesting patterns of disagreement among annotators?

There may be weaknesses that a deeper expert on this area may have better context on, but I saw no red or yellow flags.
clarity,_quality,_novelty_and_reproducibility: I found this work to be straightforward in a good way: they present some novel ideas for generating more meaningful variation in benchmark data which are informed by a good understanding of the problem and existing approaches. They generate this new dataset carefully and systematically, show how existing methods struggle with the new variation, and do some analysis to suggest future directions. There are prior robustness-oriented extensions in this domain, but this work seems to go further in a meaningful way. The work seems very well-executed and clearly described.

Table 3: DB Perturbation "Average" is misspelled as "Avergae"
summary_of_the_review: This seems like a valuable contribution to this problem area that should spur further research and improvements.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper created a dataset to evaluate text-to-SQL systems on their robustness. The dataset, DR.SPIDER, is an extension of the openly available SPIDER dataset. It added perturbation to the natural language questions, SQL statements and example database (schema and values), so more robust text-to-SQL systems would have less degradation in accuracy on the perturbed samples. The paper categorized the perturbations into 17 different categories, covering different aspects of possible real-world variations.

To obtain the perturbations, the paper employed multiple methods. It used synonym sets from previous related works, employed in-house domain experts, used online crowd-source workers (Amazon Mechanical Turks), and leveraged PLMs. The process focused more on getting a balanced and correct dataset than the effectiveness of any single method. 

The paper performed extensive evaluations of state-of-the-art models on DR.SPIDER, showed the ability of robustness measurement of the dataset, and also provided insights on the factors that may affect the robustness of existing algorithms. 

The authors promised to release the dataset and evaluation tools. 
strength_and_weaknesses: Strength:

1) A clear and useful goal. The paper addresses a specific and useful problem of evaluating the robustness of text-to-SQL systems, where existing datasets either cannot do, or have gaps in coverage. 
2) Detailed and useful evaluation. Most of STOA algorithms are evaluated, with detailed break-downs and also examples to help reader understand.
3) Clear writing. The paper is well written, The Reviewer had no trouble following the main idea and technical details.
4) Helpful to the community. The dataset will be released, which would help the improvement of the robustness of related methods.

Weakness:

1) The necessity and complexity of using PLMs are not clear, as 1) the total amount of data generated is small, in the range of hundreds. 2) significant human review is still needed up and down the pipeline.
2) The size of DR SPIDER (The Reviewer's impression is ~1k) is much smaller than SPIDER (~10K questions).


Detailed suggestions:

1) Section 2 paragraph 1, last sentence: it's better to explain why we need semantic-changing perturbations, e.g. to be used as negative examples? or to test the model's ability to distinguish between closely related but different semantics? 
2) Section 3.2 paragraph 2: better explain what is "naturally occuring  tables and columns".
3) Last paragraph in Section 3.2: Why split the filtered data into chunks, while ensuring each chuck have one annotators, why not just annotate all without chuck-splitting?
4) Section 4. Upper-case of common model names, "Bert-large" -> "BERT-large" etc.

clarity,_quality,_novelty_and_reproducibility: Clarity: good. The Reviewer had no trouble following the main idea and details.

Quality: good. The paper addressed a clearly defined problem and the evaluations supported the claim.

Novelty: ok. The methodology employed are mostly known in the field. As pointed out in the strength&weakness, the cost/benefit of using  PLM is not very clear.

Reproducibility: good. The authors promised to release the data and tools. 
summary_of_the_review: The paper provides a good extension to the SPIDER dataset to measure the robustness of text-to-SQL models. It addresses a clearly defined and important aspect of evaluation. The construction of the dataset is well thought, and the effectiveness of it has been demonstrated with experiments involves SOTA models. The paper is also clearly written and easy to follow. The resulted dataset could be a useful resource for the community in the future.

The only concern is the size of the set (a magnitude smaller than SPIDER, if the Reviewer understood correctly). The final size of the set is not clearly mentioned in the abstract, intro or conclusion (only mentioned "17 perturbations").
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new benchmark, Dr.Spider, based on the Spider dataset for evaluating text-to-SQL models. Dr.Spider contains a set of perturbations on databases, natural language questions, and SQL queries to measure the robustness of the models against those perturbations. The paper also argues the necessity of developing the new benchmark and supports its arguments by demonstrating the performance loss of the SOTA models on Dr.Spider due to the perturbations compared to the Spider dataset. The authors also did an excellent job addressing the reviewers' comments in the rebuttal. The new benchmark will certainly support the developments in neural text-to-SQL research.
justification_for_why_not_lower_score: The paper presents a new benchmark and is well-written. The paper received good scores from all reviewers, and the authors did a good job addressing the reviewers' comments.
Decision: Accept: notable-top-5%